This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10362

dirstate-tree: Make Rust DirstateMap bindings go through a trait object
ClosedPublic

Authored by SimonSapin on Apr 12 2021, 8:23 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG787ff5d21bcd: dirstate-tree: Make Rust DirstateMap bindings go through a trait object
rHG3dad278122dd: dirstate-tree: Make Rust DirstateMap bindings go through a trait object
Summary

This changeset starts a series that adds an experiment to make status faster
by changing the dirstate (first only in memory and later also on disk) to
be shaped as a tree matching the directory structure, instead of the current
flat collection of entries. The status algorithm can then traverse this tree
dirstate at the same time as it traverses the filesystem.

We (Octobus) have made prototypes that show promising results but are prone
to bitrot. We would like to start upstreaming some experimental Rust code that
goes in this direction, but to avoid disrupting users it should only be
enabled by some run-time opt-in while keeping the existing dirstate structure
and status algorithm as-is.

The DirstateMap type and status function look like the appropriate
boundary. This adds a new trait that abstracts everything Python bindings need
and makes those bindings go through a dyn trait object. Later we’ll have two
implementations of this trait, and the same bindings can use either.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

SimonSapin created this revision.Apr 12 2021, 8:23 AM
SimonSapin added a comment.Apr 19 2021, 9:47 AM

This introduce an abstraction in order to have a second, opt-in implementation of it where we can experiment without affecting most users. However the abstraction itself may have some cost: this changeset introduces some additional boxing and dynamic dispatch, even in the default case (with Rust enabled).

Fortunately the performance impact turns out to be negligible in this case. Below are the wall clock durations in seconds to run hg status and hg status -mard on various repositories, four times each, with hg changeset d80b2bf99010 (this patch) and 51841b23670b (its parent).

51841b23670b default/ − typing: make minor adjustments to mercurial/util.py to pass pytype checking
d80b2bf99010 default/dirstate-tree − dirstate-tree: Make Rust DirstateMap bindings go through a trait object

hg status

mozilla-clean 51841b23670b 0.565 0.570 0.566 0.568
mozilla-clean d80b2bf99010 0.568 0.568 0.566 0.567
mozilla-dirty 51841b23670b 0.655 0.649 0.654 0.653
mozilla-dirty d80b2bf99010 0.653 0.652 0.653 0.659
mozilla-ignored 51841b23670b 0.567 0.563 0.569 0.564
mozilla-ignored d80b2bf99010 0.567 0.567 0.568 0.570
mozilla-unknowns 51841b23670b 0.687 0.654 0.652 0.654
mozilla-unknowns d80b2bf99010 0.655 0.653 0.652 0.670
netbeans-clean 51841b23670b 0.342 0.342 0.342 0.346
netbeans-clean d80b2bf99010 0.344 0.343 0.341 0.342
netbeans-dirty 51841b23670b 0.671 0.395 0.396 0.396
netbeans-dirty d80b2bf99010 0.395 0.395 0.395 0.394
netbeans-ignored 51841b23670b 0.393 0.351 0.350 0.350
netbeans-ignored d80b2bf99010 0.349 0.349 0.348 0.348
netbeans-unknowns 51841b23670b 0.388 0.387 0.386 0.387
netbeans-unknowns d80b2bf99010 0.389 0.388 0.386 0.388

hg status -mard

mozilla-clean 51841b23670b 0.322 0.322 0.321 0.320
mozilla-clean d80b2bf99010 0.322 0.320 0.321 0.322
mozilla-dirty 51841b23670b 0.321 0.322 0.322 0.323
mozilla-dirty d80b2bf99010 0.322 0.322 0.327 0.322
mozilla-ignored 51841b23670b 0.321 0.320 0.320 0.322
mozilla-ignored d80b2bf99010 0.320 0.322 0.322 0.324
mozilla-unknowns 51841b23670b 0.322 0.323 0.323 0.324
mozilla-unknowns d80b2bf99010 0.324 0.324 0.324 0.323
netbeans-clean 51841b23670b 0.216 0.217 0.218 0.217
netbeans-clean d80b2bf99010 0.216 0.217 0.217 0.218
netbeans-dirty 51841b23670b 0.216 0.217 0.216 0.218
netbeans-dirty d80b2bf99010 0.216 0.216 0.217 0.218
netbeans-ignored 51841b23670b 0.217 0.216 0.216 0.216
netbeans-ignored d80b2bf99010 0.216 0.216 0.215 0.216
netbeans-unknowns 51841b23670b 0.215 0.215 0.216 0.215
netbeans-unknowns d80b2bf99010 0.215 0.216 0.215 0.216
Alphare accepted this revision.Apr 19 2021, 10:56 AM
Alphare added a subscriber: Alphare.

(I should put "wrote get_non_normal_other_parent_entries_panic" on my resume.)

baymax updated this revision to Diff 27046.Apr 19 2021, 11:02 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare accepted this revision.Apr 20 2021, 10:23 AM
SimonSapin added a commit: rHG3dad278122dd: dirstate-tree: Make Rust DirstateMap bindings go through a trait object.May 6 2021, 12:48 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG3dad278122dd: dirstate-tree: Make Rust DirstateMap bindings go through a trait object (authored by SimonSapin). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 27637

rust/hg-core/src/dirstate.rs

// dirstate module // dirstate module
// //
// Copyright 2019 Raphaël Gomès <rgomes@octobus.net> // Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
use crate::errors::HgError; use crate::errors::HgError;
use crate::revlog::Node; use crate::revlog::Node;
use crate::{utils::hg_path::HgPathBuf, FastHashMap}; use crate::{utils::hg_path::HgPathBuf, FastHashMap};
use bytes_cast::{unaligned, BytesCast}; use bytes_cast::{unaligned, BytesCast};
use std::collections::hash_map;
use std::convert::TryFrom; use std::convert::TryFrom;
pub mod dirs_multiset; pub mod dirs_multiset;
pub mod dirstate_map; pub mod dirstate_map;
pub mod parsers; pub mod parsers;
pub mod status; pub mod status;
#[derive(Debug, PartialEq, Clone, BytesCast)] #[derive(Debug, PartialEq, Clone, BytesCast)]
} }
/// A `DirstateEntry` with a size of `-2` means that it was merged from the /// A `DirstateEntry` with a size of `-2` means that it was merged from the
/// other parent. This allows revert to pick the right status back during a /// other parent. This allows revert to pick the right status back during a
/// merge. /// merge.
pub const SIZE_FROM_OTHER_PARENT: i32 = -2; pub const SIZE_FROM_OTHER_PARENT: i32 = -2;
pub type StateMap = FastHashMap<HgPathBuf, DirstateEntry>; pub type StateMap = FastHashMap<HgPathBuf, DirstateEntry>;
pub type StateMapIter<'a> = hash_map::Iter<'a, HgPathBuf, DirstateEntry>; pub type StateMapIter<'a> =
Box<dyn Iterator<Item = (&'a HgPathBuf, &'a DirstateEntry)> + Send + 'a>;
pub type CopyMap = FastHashMap<HgPathBuf, HgPathBuf>; pub type CopyMap = FastHashMap<HgPathBuf, HgPathBuf>;
pub type CopyMapIter<'a> = hash_map::Iter<'a, HgPathBuf, HgPathBuf>; pub type CopyMapIter<'a> =
Box<dyn Iterator<Item = (&'a HgPathBuf, &'a HgPathBuf)> + Send + 'a>;
#[derive(Copy, Clone, Debug, Eq, PartialEq)] #[derive(Copy, Clone, Debug, Eq, PartialEq)]
pub enum EntryState { pub enum EntryState {
Normal, Normal,
Added, Added,
Removed, Removed,
Merged, Merged,
Unknown, Unknown,

rust/hg-core/src/dirstate/dirs_multiset.rs

// dirs_multiset.rs // dirs_multiset.rs
// //
// Copyright 2019 Raphaël Gomès <rgomes@octobus.net> // Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
//! A multiset of directory names. //! A multiset of directory names.
//! //!
//! Used to counts the references to directories in a manifest or dirstate. //! Used to counts the references to directories in a manifest or dirstate.
use crate::{ use crate::{
dirstate::EntryState, dirstate::EntryState,
utils::{ utils::{
files, files,
hg_path::{HgPath, HgPathBuf, HgPathError}, hg_path::{HgPath, HgPathBuf, HgPathError},
}, },
DirstateEntry, DirstateMapError, FastHashMap, StateMap, DirstateEntry, DirstateMapError, FastHashMap,
}; };
use std::collections::{hash_map, hash_map::Entry, HashMap, HashSet}; use std::collections::{hash_map, hash_map::Entry, HashMap, HashSet};
// could be encapsulated if we care API stability more seriously // could be encapsulated if we care API stability more seriously
pub type DirsMultisetIter<'a> = hash_map::Keys<'a, HgPathBuf, u32>; pub type DirsMultisetIter<'a> = hash_map::Keys<'a, HgPathBuf, u32>;
#[derive(PartialEq, Debug)] #[derive(PartialEq, Debug)]
pub struct DirsMultiset { pub struct DirsMultiset {
inner: FastHashMap<HgPathBuf, u32>, inner: FastHashMap<HgPathBuf, u32>,
} }
impl DirsMultiset { impl DirsMultiset {
/// Initializes the multiset from a dirstate. /// Initializes the multiset from a dirstate.
/// ///
/// If `skip_state` is provided, skips dirstate entries with equal state. /// If `skip_state` is provided, skips dirstate entries with equal state.
pub fn from_dirstate( pub fn from_dirstate<'a>(
dirstate: &StateMap, dirstate: impl IntoIterator<Item = (&'a HgPathBuf, &'a DirstateEntry)>,
skip_state: Option<EntryState>, skip_state: Option<EntryState>,
) -> Result<Self, DirstateMapError> { ) -> Result<Self, DirstateMapError> {
let mut multiset = DirsMultiset { let mut multiset = DirsMultiset {
inner: FastHashMap::default(), inner: FastHashMap::default(),
}; };
for (filename, DirstateEntry { state, .. }) in dirstate.iter() { for (filename, DirstateEntry { state, .. }) in dirstate {
// This `if` is optimized out of the loop // This `if` is optimized out of the loop
if let Some(skip) = skip_state { if let Some(skip) = skip_state {
if skip != *state { if skip != *state {
multiset.add_path(filename)?; multiset.add_path(filename)?;
} }
} else { } else {
multiset.add_path(filename)?; multiset.add_path(filename)?;
} }
) -> Option<&HashSet<&'a HgPath>> { ) -> Option<&HashSet<&'a HgPath>> {
self.inner.get(path.as_ref()) self.inner.get(path.as_ref())
} }
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::StateMap;
#[test] #[test]
fn test_delete_path_path_not_found() { fn test_delete_path_path_not_found() {
let manifest: Vec<HgPathBuf> = vec![]; let manifest: Vec<HgPathBuf> = vec![];
let mut map = DirsMultiset::from_manifest(&manifest).unwrap(); let mut map = DirsMultiset::from_manifest(&manifest).unwrap();
let path = HgPathBuf::from_bytes(b"doesnotexist/"); let path = HgPathBuf::from_bytes(b"doesnotexist/");
assert_eq!( assert_eq!(
Err(DirstateMapError::PathNotFound(path.to_owned())), Err(DirstateMapError::PathNotFound(path.to_owned())),
.collect(); .collect();
let new = DirsMultiset::from_manifest(&input_vec).unwrap(); let new = DirsMultiset::from_manifest(&input_vec).unwrap();
let expected = DirsMultiset { let expected = DirsMultiset {
inner: expected_inner, inner: expected_inner,
}; };
assert_eq!(expected, new); assert_eq!(expected, new);
let input_map = ["b/x", "a/c", "a/d/x"] let input_map: HashMap<_, _> = ["b/x", "a/c", "a/d/x"]
.iter() .iter()
.map(|f| { .map(|f| {
( (
HgPathBuf::from_bytes(f.as_bytes()), HgPathBuf::from_bytes(f.as_bytes()),
DirstateEntry { DirstateEntry {
state: EntryState::Normal, state: EntryState::Normal,
mode: 0, mode: 0,
mtime: 0, mtime: 0,
let expected = DirsMultiset { let expected = DirsMultiset {
inner: expected_inner, inner: expected_inner,
}; };
assert_eq!(expected, new); assert_eq!(expected, new);
} }
#[test] #[test]
fn test_dirsmultiset_new_skip() { fn test_dirsmultiset_new_skip() {
let input_map = [ let input_map: HashMap<_, _> = [
("a/", EntryState::Normal), ("a/", EntryState::Normal),
("a/b", EntryState::Normal), ("a/b", EntryState::Normal),
("a/c", EntryState::Removed), ("a/c", EntryState::Removed),
("a/d", EntryState::Merged), ("a/d", EntryState::Merged),
] ]
.iter() .iter()
.map(|(f, state)| { .map(|(f, state)| {
( (

rust/hg-core/src/dirstate/dirstate_map.rs

} }
/// Both of these setters and their uses appear to be the simplest way to /// Both of these setters and their uses appear to be the simplest way to
/// emulate a Python lazy property, but it is ugly and unidiomatic. /// emulate a Python lazy property, but it is ugly and unidiomatic.
/// TODO One day, rewriting this struct using the typestate might be a /// TODO One day, rewriting this struct using the typestate might be a
/// good idea. /// good idea.
pub fn set_all_dirs(&mut self) -> Result<(), DirstateMapError> { pub fn set_all_dirs(&mut self) -> Result<(), DirstateMapError> {
if self.all_dirs.is_none() { if self.all_dirs.is_none() {
self.all_dirs = self.all_dirs = Some(DirsMultiset::from_dirstate(
Some(DirsMultiset::from_dirstate(&self.state_map, None)?); self.state_map.iter(),
None,
)?);
} }
Ok(()) Ok(())
} }
pub fn set_dirs(&mut self) -> Result<(), DirstateMapError> { pub fn set_dirs(&mut self) -> Result<(), DirstateMapError> {
if self.dirs.is_none() { if self.dirs.is_none() {
self.dirs = Some(DirsMultiset::from_dirstate( self.dirs = Some(DirsMultiset::from_dirstate(
&self.state_map, &self.state_map,

rust/hg-core/src/dirstate/status.rs

StatusError::Path(error) => error.fmt(f), StatusError::Path(error) => error.fmt(f),
StatusError::Pattern(error) => error.fmt(f), StatusError::Pattern(error) => error.fmt(f),
} }
} }
} }
/// Gives information about which files are changed in the working directory /// Gives information about which files are changed in the working directory
/// and how, compared to the revision we're based on /// and how, compared to the revision we're based on
pub struct Status<'a, M: Matcher + Sync> { pub struct Status<'a, M: ?Sized + Matcher + Sync> {
dmap: &'a DirstateMap, dmap: &'a DirstateMap,
pub(crate) matcher: &'a M, pub(crate) matcher: &'a M,
root_dir: PathBuf, root_dir: PathBuf,
pub(crate) options: StatusOptions, pub(crate) options: StatusOptions,
ignore_fn: IgnoreFnType<'a>, ignore_fn: IgnoreFnType<'a>,
} }
impl<'a, M> Status<'a, M> impl<'a, M> Status<'a, M>
where where
M: Matcher + Sync, M: ?Sized + Matcher + Sync,
{ {
pub fn new( pub fn new(
dmap: &'a DirstateMap, dmap: &'a DirstateMap,
matcher: &'a M, matcher: &'a M,
root_dir: PathBuf, root_dir: PathBuf,
ignore_files: Vec<PathBuf>, ignore_files: Vec<PathBuf>,
options: StatusOptions, options: StatusOptions,
) -> StatusResult<(Self, Vec<PatternFileWarning>)> { ) -> StatusResult<(Self, Vec<PatternFileWarning>)> {
/// Get the status of files in the working directory. /// Get the status of files in the working directory.
/// ///
/// This is the current entry-point for `hg-core` and is realistically unusable /// This is the current entry-point for `hg-core` and is realistically unusable
/// outside of a Python context because its arguments need to provide a lot of /// outside of a Python context because its arguments need to provide a lot of
/// information that will not be necessary in the future. /// information that will not be necessary in the future.
#[timed] #[timed]
pub fn status<'a>( pub fn status<'a>(
dmap: &'a DirstateMap, dmap: &'a DirstateMap,
matcher: &'a (impl Matcher + Sync), matcher: &'a (dyn Matcher + Sync),
root_dir: PathBuf, root_dir: PathBuf,
ignore_files: Vec<PathBuf>, ignore_files: Vec<PathBuf>,
options: StatusOptions, options: StatusOptions,
) -> StatusResult<( ) -> StatusResult<(
(Vec<HgPathCow<'a>>, DirstateStatus<'a>), (Vec<HgPathCow<'a>>, DirstateStatus<'a>),
Vec<PatternFileWarning>, Vec<PatternFileWarning>,
)> { )> {
let (status, warnings) = let (status, warnings) =
Status::new(dmap, matcher, root_dir, ignore_files, options)?; Status::new(dmap, matcher, root_dir, ignore_files, options)?;
Ok((status.run()?, warnings)) Ok((status.run()?, warnings))
} }

rust/hg-core/src/dirstate_tree.rs

  • This file was added.
pub mod dispatch;

rust/hg-core/src/dirstate_tree/dispatch.rs

  • This file was added.
use std::collections::HashSet;
use std::path::PathBuf;
use std::time::Duration;
use crate::matchers::Matcher;
use crate::utils::hg_path::{HgPath, HgPathBuf};
use crate::CopyMapIter;
use crate::DirstateEntry;
use crate::DirstateError;
use crate::DirstateMap;
use crate::DirstateMapError;
use crate::DirstateParents;
use crate::DirstateStatus;
use crate::EntryState;
use crate::FastHashMap;
use crate::HgPathCow;
use crate::PatternFileWarning;
use crate::StateMapIter;
use crate::StatusError;
use crate::StatusOptions;
pub trait DirstateMapMethods {
fn clear(&mut self);
fn add_file(
&mut self,
filename: &HgPath,
old_state: EntryState,
entry: DirstateEntry,
) -> Result<(), DirstateMapError>;
fn remove_file(
&mut self,
filename: &HgPath,
old_state: EntryState,
size: i32,
) -> Result<(), DirstateMapError>;
fn drop_file(
&mut self,
filename: &HgPath,
old_state: EntryState,
) -> Result<bool, DirstateMapError>;
fn clear_ambiguous_times(&mut self, filenames: Vec<HgPathBuf>, now: i32);
fn non_normal_entries_remove(&mut self, key: &HgPath) -> bool;
fn non_normal_entries_union(
&mut self,
other: HashSet<HgPathBuf>,
) -> Vec<HgPathBuf>;
fn set_non_normal_other_parent_entries(&mut self, force: bool);
fn get_non_normal_other_parent_entries_panic(
&self,
) -> (&HashSet<HgPathBuf>, &HashSet<HgPathBuf>);
fn get_non_normal_other_parent_entries(
&mut self,
) -> (&mut HashSet<HgPathBuf>, &mut HashSet<HgPathBuf>);
fn has_tracked_dir(
&mut self,
directory: &HgPath,
) -> Result<bool, DirstateMapError>;
fn has_dir(
&mut self,
directory: &HgPath,
) -> Result<bool, DirstateMapError>;
fn parents(
&mut self,
file_contents: &[u8],
) -> Result<&DirstateParents, DirstateError>;
fn set_parents(&mut self, parents: &DirstateParents);
fn read<'a>(
&mut self,
file_contents: &'a [u8],
) -> Result<Option<&'a DirstateParents>, DirstateError>;
fn pack(
&mut self,
parents: DirstateParents,
now: Duration,
) -> Result<Vec<u8>, DirstateError>;
fn build_file_fold_map(&mut self) -> &FastHashMap<HgPathBuf, HgPathBuf>;
fn set_all_dirs(&mut self) -> Result<(), DirstateMapError>;
fn set_dirs(&mut self) -> Result<(), DirstateMapError>;
fn status<'a>(
&'a self,
matcher: &'a (dyn Matcher + Sync),
root_dir: PathBuf,
ignore_files: Vec<PathBuf>,
options: StatusOptions,
) -> Result<
(
(Vec<HgPathCow<'a>>, DirstateStatus<'a>),
Vec<PatternFileWarning>,
),
StatusError,
>;
fn copy_map_len(&self) -> usize;
fn copy_map_iter(&self) -> CopyMapIter<'_>;
fn copy_map_contains_key(&self, key: &HgPath) -> bool;
fn copy_map_get(&self, key: &HgPath) -> Option<&HgPathBuf>;
fn copy_map_remove(&mut self, key: &HgPath) -> Option<HgPathBuf>;
fn copy_map_insert(
&mut self,
key: HgPathBuf,
value: HgPathBuf,
) -> Option<HgPathBuf>;
fn len(&self) -> usize;
fn contains_key(&self, key: &HgPath) -> bool;
fn get(&self, key: &HgPath) -> Option<&DirstateEntry>;
fn iter(&self) -> StateMapIter<'_>;
}
impl DirstateMapMethods for DirstateMap {
fn clear(&mut self) {
self.clear()
}
fn add_file(
&mut self,
filename: &HgPath,
old_state: EntryState,
entry: DirstateEntry,
) -> Result<(), DirstateMapError> {
self.add_file(filename, old_state, entry)
}
fn remove_file(
&mut self,
filename: &HgPath,
old_state: EntryState,
size: i32,
) -> Result<(), DirstateMapError> {
self.remove_file(filename, old_state, size)
}
fn drop_file(
&mut self,
filename: &HgPath,
old_state: EntryState,
) -> Result<bool, DirstateMapError> {
self.drop_file(filename, old_state)
}
fn clear_ambiguous_times(&mut self, filenames: Vec<HgPathBuf>, now: i32) {
self.clear_ambiguous_times(filenames, now)
}
fn non_normal_entries_remove(&mut self, key: &HgPath) -> bool {
self.non_normal_entries_remove(key)
}
fn non_normal_entries_union(
&mut self,
other: HashSet<HgPathBuf>,
) -> Vec<HgPathBuf> {
self.non_normal_entries_union(other)
}
fn set_non_normal_other_parent_entries(&mut self, force: bool) {
self.set_non_normal_other_parent_entries(force)
}
fn get_non_normal_other_parent_entries_panic(
&self,
) -> (&HashSet<HgPathBuf>, &HashSet<HgPathBuf>) {
self.get_non_normal_other_parent_entries_panic()
}
fn get_non_normal_other_parent_entries(
&mut self,
) -> (&mut HashSet<HgPathBuf>, &mut HashSet<HgPathBuf>) {
self.get_non_normal_other_parent_entries()
}
fn has_tracked_dir(
&mut self,
directory: &HgPath,
) -> Result<bool, DirstateMapError> {
self.has_tracked_dir(directory)
}
fn has_dir(
&mut self,
directory: &HgPath,
) -> Result<bool, DirstateMapError> {
self.has_dir(directory)
}
fn parents(
&mut self,
file_contents: &[u8],
) -> Result<&DirstateParents, DirstateError> {
self.parents(file_contents)
}
fn set_parents(&mut self, parents: &DirstateParents) {
self.set_parents(parents)
}
fn read<'a>(
&mut self,
file_contents: &'a [u8],
) -> Result<Option<&'a DirstateParents>, DirstateError> {
self.read(file_contents)
}
fn pack(
&mut self,
parents: DirstateParents,
now: Duration,
) -> Result<Vec<u8>, DirstateError> {
self.pack(parents, now)
}
fn build_file_fold_map(&mut self) -> &FastHashMap<HgPathBuf, HgPathBuf> {
self.build_file_fold_map()
}
fn set_all_dirs(&mut self) -> Result<(), DirstateMapError> {
self.set_all_dirs()
}
fn set_dirs(&mut self) -> Result<(), DirstateMapError> {
self.set_dirs()
}
fn status<'a>(
&'a self,
matcher: &'a (dyn Matcher + Sync),
root_dir: PathBuf,
ignore_files: Vec<PathBuf>,
options: StatusOptions,
) -> Result<
(
(Vec<HgPathCow<'a>>, DirstateStatus<'a>),
Vec<PatternFileWarning>,
),
StatusError,
> {
crate::status(self, matcher, root_dir, ignore_files, options)
}
fn copy_map_len(&self) -> usize {
self.copy_map.len()
}
fn copy_map_iter(&self) -> CopyMapIter<'_> {
Box::new(self.copy_map.iter())
}
fn copy_map_contains_key(&self, key: &HgPath) -> bool {
self.copy_map.contains_key(key)
}
fn copy_map_get(&self, key: &HgPath) -> Option<&HgPathBuf> {
self.copy_map.get(key)
}
fn copy_map_remove(&mut self, key: &HgPath) -> Option<HgPathBuf> {
self.copy_map.remove(key)
}
fn copy_map_insert(
&mut self,
key: HgPathBuf,
value: HgPathBuf,
) -> Option<HgPathBuf> {
self.copy_map.insert(key, value)
}
fn len(&self) -> usize {
(&**self).len()
}
fn contains_key(&self, key: &HgPath) -> bool {
(&**self).contains_key(key)
}
fn get(&self, key: &HgPath) -> Option<&DirstateEntry> {
(&**self).get(key)
}
fn iter(&self) -> StateMapIter<'_> {
Box::new((&**self).iter())
}
}

rust/hg-core/src/lib.rs

// Copyright 2018-2020 Georges Racinet <georges.racinet@octobus.net> // Copyright 2018-2020 Georges Racinet <georges.racinet@octobus.net>
// and Mercurial contributors // and Mercurial contributors
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
mod ancestors; mod ancestors;
pub mod dagops; pub mod dagops;
pub mod errors; pub mod errors;
pub use ancestors::{AncestorsIterator, LazyAncestors, MissingAncestors}; pub use ancestors::{AncestorsIterator, LazyAncestors, MissingAncestors};
mod dirstate; mod dirstate;
pub mod dirstate_tree;
pub mod discovery; pub mod discovery;
pub mod requirements; pub mod requirements;
pub mod testing; // unconditionally built, for use from integration tests pub mod testing; // unconditionally built, for use from integration tests
pub use dirstate::{ pub use dirstate::{
dirs_multiset::{DirsMultiset, DirsMultisetIter}, dirs_multiset::{DirsMultiset, DirsMultisetIter},
dirstate_map::DirstateMap, dirstate_map::DirstateMap,
parsers::{pack_dirstate, parse_dirstate, PARENT_SIZE}, parsers::{pack_dirstate, parse_dirstate, PARENT_SIZE},
status::{ status::{

rust/hg-core/src/operations/dirstate_status.rs

// dirstate_status.rs // dirstate_status.rs
// //
// Copyright 2019, Raphaël Gomès <rgomes@octobus.net> // Copyright 2019, Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
use crate::dirstate::status::{build_response, Dispatch, HgPathCow, Status}; use crate::dirstate::status::{build_response, Dispatch, HgPathCow, Status};
use crate::matchers::Matcher; use crate::matchers::Matcher;
use crate::{DirstateStatus, StatusError}; use crate::{DirstateStatus, StatusError};
/// A tuple of the paths that need to be checked in the filelog because it's /// A tuple of the paths that need to be checked in the filelog because it's
/// ambiguous whether they've changed, and the rest of the already dispatched /// ambiguous whether they've changed, and the rest of the already dispatched
/// files. /// files.
pub type LookupAndStatus<'a> = (Vec<HgPathCow<'a>>, DirstateStatus<'a>); pub type LookupAndStatus<'a> = (Vec<HgPathCow<'a>>, DirstateStatus<'a>);
impl<'a, M: Matcher + Sync> Status<'a, M> { impl<'a, M: ?Sized + Matcher + Sync> Status<'a, M> {
pub(crate) fn run(&self) -> Result<LookupAndStatus<'a>, StatusError> { pub(crate) fn run(&self) -> Result<LookupAndStatus<'a>, StatusError> {
let (traversed_sender, traversed_receiver) = let (traversed_sender, traversed_receiver) =
crossbeam_channel::unbounded(); crossbeam_channel::unbounded();
// Step 1: check the files explicitly mentioned by the user // Step 1: check the files explicitly mentioned by the user
let (work, mut results) = self.walk_explicit(traversed_sender.clone()); let (work, mut results) = self.walk_explicit(traversed_sender.clone());
if !work.is_empty() { if !work.is_empty() {

rust/hg-cpython/src/dirstate/dirstate_map.rs

dirstate::copymap::{CopyMap, CopyMapItemsIterator, CopyMapKeysIterator}, dirstate::copymap::{CopyMap, CopyMapItemsIterator, CopyMapKeysIterator},
dirstate::non_normal_entries::{ dirstate::non_normal_entries::{
NonNormalEntries, NonNormalEntriesIterator, NonNormalEntries, NonNormalEntriesIterator,
}, },
dirstate::{dirs_multiset::Dirs, make_dirstate_tuple}, dirstate::{dirs_multiset::Dirs, make_dirstate_tuple},
parsers::dirstate_parents_to_pytuple, parsers::dirstate_parents_to_pytuple,
}; };
use hg::{ use hg::{
dirstate_tree::dispatch::DirstateMapMethods,
errors::HgError, errors::HgError,
revlog::Node, revlog::Node,
utils::hg_path::{HgPath, HgPathBuf}, utils::hg_path::{HgPath, HgPathBuf},
DirsMultiset, DirstateEntry, DirstateMap as RustDirstateMap, DirsMultiset, DirstateEntry, DirstateMap as RustDirstateMap,
DirstateMapError, DirstateParents, EntryState, StateMapIter, DirstateMapError, DirstateParents, EntryState, StateMapIter,
}; };
// TODO // TODO
// This object needs to share references to multiple members of its Rust // This object needs to share references to multiple members of its Rust
// inner struct, namely `copy_map`, `dirs` and `all_dirs`. // inner struct, namely `copy_map`, `dirs` and `all_dirs`.
// Right now `CopyMap` is done, but it needs to have an explicit reference // Right now `CopyMap` is done, but it needs to have an explicit reference
// to `RustDirstateMap` which itself needs to have an encapsulation for // to `RustDirstateMap` which itself needs to have an encapsulation for
// every method in `CopyMap` (copymapcopy, etc.). // every method in `CopyMap` (copymapcopy, etc.).
// This is ugly and hard to maintain. // This is ugly and hard to maintain.
// The same logic applies to `dirs` and `all_dirs`, however the `Dirs` // The same logic applies to `dirs` and `all_dirs`, however the `Dirs`
// `py_class!` is already implemented and does not mention // `py_class!` is already implemented and does not mention
// `RustDirstateMap`, rightfully so. // `RustDirstateMap`, rightfully so.
// All attributes also have to have a separate refcount data attribute for // All attributes also have to have a separate refcount data attribute for
// leaks, with all methods that go along for reference sharing. // leaks, with all methods that go along for reference sharing.
py_class!(pub class DirstateMap |py| { py_class!(pub class DirstateMap |py| {
@shared data inner: RustDirstateMap; @shared data inner: Box<dyn DirstateMapMethods + Send>;
def __new__(_cls, _root: PyObject) -> PyResult<Self> { def __new__(_cls, _root: PyObject) -> PyResult<Self> {
let inner = RustDirstateMap::default(); let inner = Box::new(RustDirstateMap::default());
Self::create_instance(py, inner) Self::create_instance(py, inner)
} }
def clear(&self) -> PyResult<PyObject> { def clear(&self) -> PyResult<PyObject> {
self.inner(py).borrow_mut().clear(); self.inner(py).borrow_mut().clear();
Ok(py.None()) Ok(py.None())
} }
// TODO don't copy, share the reference // TODO don't copy, share the reference
self.inner(py).borrow_mut().set_dirs() self.inner(py).borrow_mut().set_dirs()
.map_err(|e| { .map_err(|e| {
PyErr::new::<exc::ValueError, _>(py, e.to_string()) PyErr::new::<exc::ValueError, _>(py, e.to_string())
})?; })?;
Dirs::from_inner( Dirs::from_inner(
py, py,
DirsMultiset::from_dirstate( DirsMultiset::from_dirstate(
&self.inner(py).borrow(), self.inner(py).borrow().iter(),
Some(EntryState::Removed), Some(EntryState::Removed),
) )
.map_err(|e| { .map_err(|e| {
PyErr::new::<exc::ValueError, _>(py, e.to_string()) PyErr::new::<exc::ValueError, _>(py, e.to_string())
})?, })?,
) )
} }
def getalldirs(&self) -> PyResult<Dirs> { def getalldirs(&self) -> PyResult<Dirs> {
// TODO don't copy, share the reference // TODO don't copy, share the reference
self.inner(py).borrow_mut().set_all_dirs() self.inner(py).borrow_mut().set_all_dirs()
.map_err(|e| { .map_err(|e| {
PyErr::new::<exc::ValueError, _>(py, e.to_string()) PyErr::new::<exc::ValueError, _>(py, e.to_string())
})?; })?;
Dirs::from_inner( Dirs::from_inner(
py, py,
DirsMultiset::from_dirstate( DirsMultiset::from_dirstate(
&self.inner(py).borrow(), self.inner(py).borrow().iter(),
None, None,
).map_err(|e| { ).map_err(|e| {
PyErr::new::<exc::ValueError, _>(py, e.to_string()) PyErr::new::<exc::ValueError, _>(py, e.to_string())
})?, })?,
) )
} }
// TODO all copymap* methods, see docstring above // TODO all copymap* methods, see docstring above
def copymapcopy(&self) -> PyResult<PyDict> { def copymapcopy(&self) -> PyResult<PyDict> {
let dict = PyDict::new(py); let dict = PyDict::new(py);
for (key, value) in self.inner(py).borrow().copy_map.iter() { for (key, value) in self.inner(py).borrow().copy_map_iter() {
dict.set_item( dict.set_item(
py, py,
PyBytes::new(py, key.as_bytes()), PyBytes::new(py, key.as_bytes()),
PyBytes::new(py, value.as_bytes()), PyBytes::new(py, value.as_bytes()),
)?; )?;
} }
Ok(dict) Ok(dict)
} }
def copymapgetitem(&self, key: PyObject) -> PyResult<PyBytes> { def copymapgetitem(&self, key: PyObject) -> PyResult<PyBytes> {
let key = key.extract::<PyBytes>(py)?; let key = key.extract::<PyBytes>(py)?;
match self.inner(py).borrow().copy_map.get(HgPath::new(key.data(py))) { match self.inner(py).borrow().copy_map_get(HgPath::new(key.data(py))) {
Some(copy) => Ok(PyBytes::new(py, copy.as_bytes())), Some(copy) => Ok(PyBytes::new(py, copy.as_bytes())),
None => Err(PyErr::new::<exc::KeyError, _>( None => Err(PyErr::new::<exc::KeyError, _>(
py, py,
String::from_utf8_lossy(key.data(py)), String::from_utf8_lossy(key.data(py)),
)), )),
} }
} }
def copymap(&self) -> PyResult<CopyMap> { def copymap(&self) -> PyResult<CopyMap> {
CopyMap::from_inner(py, self.clone_ref(py)) CopyMap::from_inner(py, self.clone_ref(py))
} }
def copymaplen(&self) -> PyResult<usize> { def copymaplen(&self) -> PyResult<usize> {
Ok(self.inner(py).borrow().copy_map.len()) Ok(self.inner(py).borrow().copy_map_len())
} }
def copymapcontains(&self, key: PyObject) -> PyResult<bool> { def copymapcontains(&self, key: PyObject) -> PyResult<bool> {
let key = key.extract::<PyBytes>(py)?; let key = key.extract::<PyBytes>(py)?;
Ok(self Ok(self
.inner(py) .inner(py)
.borrow() .borrow()
.copy_map .copy_map_contains_key(HgPath::new(key.data(py))))
.contains_key(HgPath::new(key.data(py))))
} }
def copymapget( def copymapget(
&self, &self,
key: PyObject, key: PyObject,
default: Option<PyObject> default: Option<PyObject>
) -> PyResult<Option<PyObject>> { ) -> PyResult<Option<PyObject>> {
let key = key.extract::<PyBytes>(py)?; let key = key.extract::<PyBytes>(py)?;
match self match self
.inner(py) .inner(py)
.borrow() .borrow()
.copy_map .copy_map_get(HgPath::new(key.data(py)))
.get(HgPath::new(key.data(py)))
{ {
Some(copy) => Ok(Some( Some(copy) => Ok(Some(
PyBytes::new(py, copy.as_bytes()).into_object(), PyBytes::new(py, copy.as_bytes()).into_object(),
)), )),
None => Ok(default), None => Ok(default),
} }
} }
def copymapsetitem( def copymapsetitem(
&self, &self,
key: PyObject, key: PyObject,
value: PyObject value: PyObject
) -> PyResult<PyObject> { ) -> PyResult<PyObject> {
let key = key.extract::<PyBytes>(py)?; let key = key.extract::<PyBytes>(py)?;
let value = value.extract::<PyBytes>(py)?; let value = value.extract::<PyBytes>(py)?;
self.inner(py).borrow_mut().copy_map.insert( self.inner(py).borrow_mut().copy_map_insert(
HgPathBuf::from_bytes(key.data(py)), HgPathBuf::from_bytes(key.data(py)),
HgPathBuf::from_bytes(value.data(py)), HgPathBuf::from_bytes(value.data(py)),
); );
Ok(py.None()) Ok(py.None())
} }
def copymappop( def copymappop(
&self, &self,
key: PyObject, key: PyObject,
default: Option<PyObject> default: Option<PyObject>
) -> PyResult<Option<PyObject>> { ) -> PyResult<Option<PyObject>> {
let key = key.extract::<PyBytes>(py)?; let key = key.extract::<PyBytes>(py)?;
match self match self
.inner(py) .inner(py)
.borrow_mut() .borrow_mut()
.copy_map .copy_map_remove(HgPath::new(key.data(py)))
.remove(HgPath::new(key.data(py)))
{ {
Some(_) => Ok(None), Some(_) => Ok(None),
None => Ok(default), None => Ok(default),
} }
} }
def copymapiter(&self) -> PyResult<CopyMapKeysIterator> { def copymapiter(&self) -> PyResult<CopyMapKeysIterator> {
let leaked_ref = self.inner(py).leak_immutable(); let leaked_ref = self.inner(py).leak_immutable();
CopyMapKeysIterator::from_inner( CopyMapKeysIterator::from_inner(
py, py,
unsafe { leaked_ref.map(py, |o| o.copy_map.iter()) }, unsafe { leaked_ref.map(py, |o| o.copy_map_iter()) },
) )
} }
def copymapitemsiter(&self) -> PyResult<CopyMapItemsIterator> { def copymapitemsiter(&self) -> PyResult<CopyMapItemsIterator> {
let leaked_ref = self.inner(py).leak_immutable(); let leaked_ref = self.inner(py).leak_immutable();
CopyMapItemsIterator::from_inner( CopyMapItemsIterator::from_inner(
py, py,
unsafe { leaked_ref.map(py, |o| o.copy_map.iter()) }, unsafe { leaked_ref.map(py, |o| o.copy_map_iter()) },
) )
} }
}); });
impl DirstateMap { impl DirstateMap {
pub fn get_inner<'a>( pub fn get_inner<'a>(
&'a self, &'a self,
py: Python<'a>, py: Python<'a>,
) -> Ref<'a, RustDirstateMap> { ) -> Ref<'a, Box<dyn DirstateMapMethods + Send>> {
self.inner(py).borrow() self.inner(py).borrow()
} }
fn translate_key( fn translate_key(
py: Python, py: Python,
res: (&HgPathBuf, &DirstateEntry), res: (&HgPathBuf, &DirstateEntry),
) -> PyResult<Option<PyBytes>> { ) -> PyResult<Option<PyBytes>> {
Ok(Some(PyBytes::new(py, res.0.as_bytes()))) Ok(Some(PyBytes::new(py, res.0.as_bytes())))
} }

rust/hg-cpython/src/dirstate/status.rs

use crate::{dirstate::DirstateMap, exceptions::FallbackError}; use crate::{dirstate::DirstateMap, exceptions::FallbackError};
use cpython::exc::OSError; use cpython::exc::OSError;
use cpython::{ use cpython::{
exc::ValueError, ObjectProtocol, PyBytes, PyErr, PyList, PyObject, exc::ValueError, ObjectProtocol, PyBytes, PyErr, PyList, PyObject,
PyResult, PyTuple, Python, PythonObject, ToPyObject, PyResult, PyTuple, Python, PythonObject, ToPyObject,
}; };
use hg::{ use hg::{
matchers::{AlwaysMatcher, FileMatcher, IncludeMatcher}, matchers::{AlwaysMatcher, FileMatcher, IncludeMatcher},
parse_pattern_syntax, status, parse_pattern_syntax,
utils::{ utils::{
files::{get_bytes_from_path, get_path_from_bytes}, files::{get_bytes_from_path, get_path_from_bytes},
hg_path::{HgPath, HgPathBuf}, hg_path::{HgPath, HgPathBuf},
}, },
BadMatch, DirstateStatus, IgnorePattern, PatternFileWarning, StatusError, BadMatch, DirstateStatus, IgnorePattern, PatternFileWarning, StatusError,
StatusOptions, StatusOptions,
}; };
use std::borrow::{Borrow, Cow}; use std::borrow::{Borrow, Cow};
Ok(get_path_from_bytes(file.data(py)).to_owned()) Ok(get_path_from_bytes(file.data(py)).to_owned())
}) })
.collect(); .collect();
let ignore_files = ignore_files?; let ignore_files = ignore_files?;
match matcher.get_type(py).name(py).borrow() { match matcher.get_type(py).name(py).borrow() {
"alwaysmatcher" => { "alwaysmatcher" => {
let matcher = AlwaysMatcher; let matcher = AlwaysMatcher;
let ((lookup, status_res), warnings) = status( let ((lookup, status_res), warnings) = dmap
&dmap, .status(
&matcher, &matcher,
root_dir.to_path_buf(), root_dir.to_path_buf(),
ignore_files, ignore_files,
StatusOptions { StatusOptions {
check_exec, check_exec,
last_normal_time, last_normal_time,
list_clean, list_clean,
list_ignored, list_ignored,
list_unknown, list_unknown,
collect_traversed_dirs, collect_traversed_dirs,
}, },
) )
.map_err(|e| handle_fallback(py, e))?; .map_err(|e| handle_fallback(py, e))?;
build_response(py, lookup, status_res, warnings) build_response(py, lookup, status_res, warnings)
} }
"exactmatcher" => { "exactmatcher" => {
let files = matcher.call_method( let files = matcher.call_method(
py, py,
"files", "files",
PyTuple::new(py, &[]), PyTuple::new(py, &[]),
None, None,
)?; )?;
let files: PyList = files.cast_into(py)?; let files: PyList = files.cast_into(py)?;
let files: PyResult<Vec<HgPathBuf>> = files let files: PyResult<Vec<HgPathBuf>> = files
.iter(py) .iter(py)
.map(|f| { .map(|f| {
Ok(HgPathBuf::from_bytes( Ok(HgPathBuf::from_bytes(
f.extract::<PyBytes>(py)?.data(py), f.extract::<PyBytes>(py)?.data(py),
)) ))
}) })
.collect(); .collect();
let files = files?; let files = files?;
let matcher = FileMatcher::new(files.as_ref()) let matcher = FileMatcher::new(files.as_ref())
.map_err(|e| PyErr::new::<ValueError, _>(py, e.to_string()))?; .map_err(|e| PyErr::new::<ValueError, _>(py, e.to_string()))?;
let ((lookup, status_res), warnings) = status( let ((lookup, status_res), warnings) = dmap
&dmap, .status(
&matcher, &matcher,
root_dir.to_path_buf(), root_dir.to_path_buf(),
ignore_files, ignore_files,
StatusOptions { StatusOptions {
check_exec, check_exec,
last_normal_time, last_normal_time,
list_clean, list_clean,
list_ignored, list_ignored,
list_unknown, list_unknown,
collect_traversed_dirs, collect_traversed_dirs,
}, },
) )
.map_err(|e| handle_fallback(py, e))?; .map_err(|e| handle_fallback(py, e))?;
build_response(py, lookup, status_res, warnings) build_response(py, lookup, status_res, warnings)
} }
"includematcher" => { "includematcher" => {
// Get the patterns from Python even though most of them are // Get the patterns from Python even though most of them are
// redundant with those we will parse later on, as they include // redundant with those we will parse later on, as they include
// those passed from the command line. // those passed from the command line.
let ignore_patterns: PyResult<Vec<_>> = matcher let ignore_patterns: PyResult<Vec<_>> = matcher
.getattr(py, "_kindpats")? .getattr(py, "_kindpats")?
let ignore_patterns = ignore_patterns?; let ignore_patterns = ignore_patterns?;
let mut all_warnings = vec![]; let mut all_warnings = vec![];
let (matcher, warnings) = let (matcher, warnings) =
IncludeMatcher::new(ignore_patterns, &root_dir) IncludeMatcher::new(ignore_patterns, &root_dir)
.map_err(|e| handle_fallback(py, e.into()))?; .map_err(|e| handle_fallback(py, e.into()))?;
all_warnings.extend(warnings); all_warnings.extend(warnings);
let ((lookup, status_res), warnings) = status( let ((lookup, status_res), warnings) = dmap
&dmap, .status(
&matcher, &matcher,
root_dir.to_path_buf(), root_dir.to_path_buf(),
ignore_files, ignore_files,
StatusOptions { StatusOptions {
check_exec, check_exec,
last_normal_time, last_normal_time,
list_clean, list_clean,
list_ignored, list_ignored,
list_unknown, list_unknown,
collect_traversed_dirs, collect_traversed_dirs,
}, },
) )
.map_err(|e| handle_fallback(py, e))?; .map_err(|e| handle_fallback(py, e))?;
all_warnings.extend(warnings); all_warnings.extend(warnings);
build_response(py, lookup, status_res, all_warnings) build_response(py, lookup, status_res, all_warnings)
} }
e => Err(PyErr::new::<ValueError, _>( e => Err(PyErr::new::<ValueError, _>(
py, py,
format!("Unsupported matcher {}", e), format!("Unsupported matcher {}", e),