( )⚙ D11516 dirstate: Remove the flat Rust DirstateMap implementation

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11516

dirstate: Remove the flat Rust DirstateMap implementation
ClosedPublic

Authored by SimonSapin on Sep 29 2021, 10:56 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGbf8837e3d7ce: dirstate: Remove the flat Rust DirstateMap implementation
Summary

Before this changeset we had two Rust implementations of DirstateMap.
This removes the "flat" DirstateMap so that the "tree" DirstateMap is always
used when Rust enabled. This simplifies the code a lot, and will enable
(in the next changeset) further removal of a trait abstraction.

This is a performance regression when:

  • Rust is enabled, and
  • The repository uses the legacy dirstate-v1 file format, and
  • For hg status, unknown files are not listed (such as with -mard)

The regression is about 100 milliseconds for hg status -mard on a
semi-large repository (mozilla-central), from ~320ms to ~420ms.
We deem this to be small enough to be worth it.

The new dirstate-v2 is still experimental at this point, but we aim to
stabilize it (though not yet enable it by default for new repositories)
in Mercurial 6.0. Eventually, upgrating repositories to dirsate-v2 will
eliminate this regression (and enable other performance improvements).

Background

The flat DirstateMap was introduced with the first Rust implementation of the
status algorithm. It works similarly to the previous Python + C one, with a
single HashMap that associates file paths to a DirstateEntry (where Python
has a dict).

We later added the tree DirstateMap where the root of the tree contains nodes
for files and directories that are directly at the root of the repository,
and nodes for directories can contain child nodes representing the files and
directly that *they* contain directly. The shape of this tree mirrors that of
the working directory in the filesystem. This enables the status algorithm to
traverse this tree in tandem with traversing the filesystem tree, which in
turns enables a more efficient algorithm.

Furthermore, the new dirstate-v2 file format is also based on a tree of the
same shape. The tree DirstateMap can access a dirstate-v2 file without parsing
it: binary data in a single large (possibly memory-mapped) bytes buffer is
traversed on demand. This allows DirstateMap creation to take O(1) time.
(Mutation works by creating new in-memory nodes with copy-on-write semantics,
and serialization is append-mostly.)

The tradeoff is that for "legacy" repositories that use the dirstate-v1 file
format, parsing that file into a tree DirstateMap takes more time. Profiling
shows that this time is dominated by HashMap. For a dirstate containing F
files with an average D directory depth, the flat DirstateMap does parsing
in O(F) number of HashMap operations but the tree DirstateMap in O(F × D)
operations, since each node has its own HashMap containing its child nodes.
This slower costs ~140ms on an old snapshot of mozilla-central, and ~80ms
on an old snapshot of the Netbeans repository.

The status algorithm is faster, but with -mard (when not listing unknown
files) it is typically not faster *enough* to compensate the slower parsing.

Both Rust implementations are always faster than the Python + C implementation

Benchmark results

All benchmarks are run on changeset 98c0408324e6, with repositories that use
the dirstate-v1 file format, on a server with 4 CPU cores and 4 CPU threads
(no HyperThreading).

hg status benchmarks show wall clock times of the entire command as the
average and standard deviation of serveral runs, collected by
https://github.com/sharkdp/hyperfine and reformated.

Parsing benchmarks are wall clock time of the Rust function that converts a
bytes buffer of the dirstate file into the DirstateMap data structure as
used by the status algorithm. A single run each, collected by running
hg status this environment variable:

RUST_LOG=hg::dirstate::dirstate_map=trace,hg::dirstate_tree::dirstate_map=trace

Benchmark 1: Rust flat DirstateMap → Rust tree DirstateMap

hg status

mozilla-clean    	562.3 ms ± 2.0 ms	→	462.5 ms ± 0.6 ms	 1.22 ± 0.00 times faster
mozilla-dirty    	859.6 ms ± 2.2 ms	→	719.5 ms ± 3.2 ms	 1.19 ± 0.01 times faster
mozilla-ignored  	558.2 ms ± 3.0 ms	→	457.9 ms ± 2.9 ms	 1.22 ± 0.01 times faster
mozilla-unknowns 	859.4 ms ± 5.7 ms	→	716.0 ms ± 4.7 ms	 1.20 ± 0.01 times faster
netbeans-clean   	336.5 ms ± 0.9 ms	→	339.5 ms ± 0.4 ms	 0.99 ± 0.00 times faster
netbeans-dirty   	491.4 ms ± 1.6 ms	→	475.1 ms ± 1.2 ms	 1.03 ± 0.00 times faster
netbeans-ignored 	343.7 ms ± 1.0 ms	→	347.8 ms ± 0.4 ms	 0.99 ± 0.00 times faster
netbeans-unknowns	484.3 ms ± 1.0 ms	→	466.0 ms ± 1.2 ms	 1.04 ± 0.00 times faster

hg status -mard

mozilla-clean    	317.3 ms ± 0.6 ms	→	422.5 ms ± 1.2 ms	 0.75 ± 0.00 times faster
mozilla-dirty    	315.4 ms ± 0.6 ms	→	417.7 ms ± 1.1 ms	 0.76 ± 0.00 times faster
mozilla-ignored  	314.6 ms ± 0.6 ms	→	417.4 ms ± 1.0 ms	 0.75 ± 0.00 times faster
mozilla-unknowns 	312.9 ms ± 0.9 ms	→	417.3 ms ± 1.6 ms	 0.75 ± 0.00 times faster
netbeans-clean   	212.0 ms ± 0.6 ms	→	283.6 ms ± 0.8 ms	 0.75 ± 0.00 times faster
netbeans-dirty   	211.4 ms ± 1.0 ms	→	283.4 ms ± 1.6 ms	 0.75 ± 0.01 times faster
netbeans-ignored 	211.4 ms ± 0.9 ms	→	283.9 ms ± 0.8 ms	 0.74 ± 0.01 times faster
netbeans-unknowns	211.1 ms ± 0.6 ms	→	283.4 ms ± 1.0 ms	 0.74 ± 0.00 times faster

Parsing

mozilla-clean       38.4ms → 177.6ms
mozilla-dirty       38.8ms → 177.0ms
mozilla-ignored     38.8ms → 178.0ms
mozilla-unknowns    38.7ms → 176.9ms
netbeans-clean      16.5ms → 97.3ms
netbeans-dirty      16.5ms → 98.4ms
netbeans-ignored    16.9ms → 97.4ms
netbeans-unknowns   16.9ms → 96.3ms

Benchmark 2: Python + C dirstatemap → Rust tree DirstateMap

hg status

mozilla-clean    	1261.0 ms ± 3.6 ms	→	461.1 ms ± 0.5 ms	 2.73 ± 0.00 times faster
mozilla-dirty    	2293.4 ms ± 9.1 ms	→	719.6 ms ± 3.6 ms	 3.19 ± 0.01 times faster
mozilla-ignored  	1240.4 ms ± 2.3 ms	→	457.7 ms ± 1.9 ms	 2.71 ± 0.00 times faster
mozilla-unknowns 	2283.3 ms ± 9.0 ms	→	719.7 ms ± 3.8 ms	 3.17 ± 0.01 times faster
netbeans-clean   	879.7 ms ± 3.5 ms	→	339.9 ms ± 0.5 ms	 2.59 ± 0.00 times faster
netbeans-dirty   	1257.3 ms ± 4.7 ms	→	474.6 ms ± 1.6 ms	 2.65 ± 0.01 times faster
netbeans-ignored 	943.9 ms ± 1.9 ms	→	347.3 ms ± 1.1 ms	 2.72 ± 0.00 times faster
netbeans-unknowns	1188.1 ms ± 5.0 ms	→	465.2 ms ± 2.3 ms	 2.55 ± 0.01 times faster

hg status -mard

mozilla-clean    	903.2 ms ± 3.6 ms	→	423.4 ms ± 2.2 ms	 2.13 ± 0.01 times faster
mozilla-dirty    	884.6 ms ± 4.5 ms	→	417.3 ms ± 1.4 ms	 2.12 ± 0.01 times faster
mozilla-ignored  	881.9 ms ± 1.3 ms	→	417.3 ms ± 0.8 ms	 2.11 ± 0.00 times faster
mozilla-unknowns 	878.5 ms ± 1.9 ms	→	416.4 ms ± 0.9 ms	 2.11 ± 0.00 times faster
netbeans-clean   	434.9 ms ± 1.8 ms	→	284.0 ms ± 0.8 ms	 1.53 ± 0.01 times faster
netbeans-dirty   	434.1 ms ± 0.8 ms	→	283.1 ms ± 0.8 ms	 1.53 ± 0.00 times faster
netbeans-ignored 	431.7 ms ± 1.1 ms	→	283.6 ms ± 1.8 ms	 1.52 ± 0.01 times faster
netbeans-unknowns	433.0 ms ± 1.3 ms	→	283.5 ms ± 0.7 ms	 1.53 ± 0.00 times faster

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

SimonSapin created this revision.Sep 29 2021, 10:56 AM
Alphare accepted this revision.Sep 29 2021, 12:31 PM
Alphare added a subscriber: Alphare.

Removing this much code makes me happy, this has come a long way. It is worth noting that we had one or two extra optimization ideas if the regression *really* becomes an issue, but dirstate-v2 should really be the focus.

This revision is now accepted and ready to land.Sep 29 2021, 12:31 PM

Revision Contents
Changeset List

Diff 30442

mercurial/configitems.py

) )
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'directaccess.revnums', b'directaccess.revnums',
default=False, default=False,
) )
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'dirstate-tree.in-memory',
default=False,
)
coreconfigitem(
b'experimental',
b'editortmpinhg', b'editortmpinhg',
default=False, default=False,
) )
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'evolution', b'evolution',
default=list, default=list,
) )

mercurial/dirstatemap.py

self._nodelen = 20 # Also update Rust code when changing this! self._nodelen = 20 # Also update Rust code when changing this!
self._parents = None self._parents = None
self._dirtyparents = False self._dirtyparents = False
self._docket = None self._docket = None
# for consistent view between _pl() and _read() invocations # for consistent view between _pl() and _read() invocations
self._pendingmode = None self._pendingmode = None
self._use_dirstate_tree = self._ui.configbool(
b"experimental",
b"dirstate-tree.in-memory",
False,
)
def addfile( def addfile(
self, self,
f, f,
mode=0, mode=0,
size=None, size=None,
mtime=None, mtime=None,
added=False, added=False,
merged=False, merged=False,
else: else:
data = b'' data = b''
self._rustmap = rustmod.DirstateMap.new_v2( self._rustmap = rustmod.DirstateMap.new_v2(
data, self.docket.data_size, self.docket.tree_metadata data, self.docket.data_size, self.docket.tree_metadata
) )
parents = self.docket.parents parents = self.docket.parents
else: else:
self._rustmap, parents = rustmod.DirstateMap.new_v1( self._rustmap, parents = rustmod.DirstateMap.new_v1(
self._use_dirstate_tree, self._readdirstatefile() self._readdirstatefile()
) )
if parents and not self._dirtyparents: if parents and not self._dirtyparents:
self.setparents(*parents) self.setparents(*parents)
self.__contains__ = self._rustmap.__contains__ self.__contains__ = self._rustmap.__contains__
self.__getitem__ = self._rustmap.__getitem__ self.__getitem__ = self._rustmap.__getitem__
self.get = self._rustmap.get self.get = self._rustmap.get

mercurial/upgrade_utils/engine.py

util.copyfile( util.copyfile(
srcrepo.vfs.join(b'requires'), backupvfs.join(b'requires') srcrepo.vfs.join(b'requires'), backupvfs.join(b'requires')
) )
util.copyfile( util.copyfile(
srcrepo.vfs.join(b'dirstate'), backupvfs.join(b'dirstate') srcrepo.vfs.join(b'dirstate'), backupvfs.join(b'dirstate')
) )
assert srcrepo.dirstate._use_dirstate_v2 == (old == b'v2') assert srcrepo.dirstate._use_dirstate_v2 == (old == b'v2')
srcrepo.dirstate._map._use_dirstate_tree = True
srcrepo.dirstate._map.preload() srcrepo.dirstate._map.preload()
srcrepo.dirstate._use_dirstate_v2 = new == b'v2' srcrepo.dirstate._use_dirstate_v2 = new == b'v2'
srcrepo.dirstate._map._use_dirstate_v2 = srcrepo.dirstate._use_dirstate_v2 srcrepo.dirstate._map._use_dirstate_v2 = srcrepo.dirstate._use_dirstate_v2
srcrepo.dirstate._dirty = True srcrepo.dirstate._dirty = True
srcrepo.vfs.unlink(b'dirstate') srcrepo.vfs.unlink(b'dirstate')
srcrepo.dirstate.write(None) srcrepo.dirstate.write(None)
scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements) scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements)

rust/hg-core/src/dirstate.rs

// dirstate module // dirstate module
// //
// Copyright 2019 Raphaël Gomès <rgomes@octobus.net> // Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
use crate::dirstate_tree::on_disk::DirstateV2ParseError; use crate::dirstate_tree::on_disk::DirstateV2ParseError;
use crate::revlog::node::NULL_NODE; use crate::revlog::node::NULL_NODE;
use crate::revlog::Node; use crate::revlog::Node;
use crate::utils::hg_path::{HgPath, HgPathBuf}; use crate::utils::hg_path::HgPath;
use crate::FastHashMap;
use bytes_cast::BytesCast; use bytes_cast::BytesCast;
pub mod dirs_multiset; pub mod dirs_multiset;
pub mod dirstate_map;
pub mod entry; pub mod entry;
pub mod parsers; pub mod parsers;
pub mod status; pub mod status;
pub use self::entry::*; pub use self::entry::*;
#[derive(Debug, PartialEq, Copy, Clone, BytesCast)] #[derive(Debug, PartialEq, Copy, Clone, BytesCast)]
#[repr(C)] #[repr(C)]
pub struct DirstateParents { pub struct DirstateParents {
pub p1: Node, pub p1: Node,
pub p2: Node, pub p2: Node,
} }
impl DirstateParents { impl DirstateParents {
pub const NULL: Self = Self { pub const NULL: Self = Self {
p1: NULL_NODE, p1: NULL_NODE,
p2: NULL_NODE, p2: NULL_NODE,
}; };
} }
pub type StateMap = FastHashMap<HgPathBuf, DirstateEntry>;
pub type StateMapIter<'a> = Box< pub type StateMapIter<'a> = Box<
dyn Iterator< dyn Iterator<
Item = Result<(&'a HgPath, DirstateEntry), DirstateV2ParseError>, Item = Result<(&'a HgPath, DirstateEntry), DirstateV2ParseError>,
> + Send > + Send
+ 'a, + 'a,
>; >;
pub type CopyMap = FastHashMap<HgPathBuf, HgPathBuf>;
pub type CopyMapIter<'a> = Box< pub type CopyMapIter<'a> = Box<
dyn Iterator<Item = Result<(&'a HgPath, &'a HgPath), DirstateV2ParseError>> dyn Iterator<Item = Result<(&'a HgPath, &'a HgPath), DirstateV2ParseError>>
+ Send + Send
+ 'a, + 'a,
>; >;

rust/hg-core/src/dirstate/dirs_multiset.rs

) -> Option<&HashSet<&'a HgPath>> { ) -> Option<&HashSet<&'a HgPath>> {
self.inner.get(path.as_ref()) self.inner.get(path.as_ref())
} }
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::StateMap;
#[test] #[test]
fn test_delete_path_path_not_found() { fn test_delete_path_path_not_found() {
let manifest: Vec<HgPathBuf> = vec![]; let manifest: Vec<HgPathBuf> = vec![];
let mut map = DirsMultiset::from_manifest(&manifest).unwrap(); let mut map = DirsMultiset::from_manifest(&manifest).unwrap();
let path = HgPathBuf::from_bytes(b"doesnotexist/"); let path = HgPathBuf::from_bytes(b"doesnotexist/");
assert_eq!( assert_eq!(
Err(DirstateMapError::PathNotFound(path.to_owned())), Err(DirstateMapError::PathNotFound(path.to_owned())),
fn test_dirsmultiset_new_empty() { fn test_dirsmultiset_new_empty() {
let manifest: Vec<HgPathBuf> = vec![]; let manifest: Vec<HgPathBuf> = vec![];
let new = DirsMultiset::from_manifest(&manifest).unwrap(); let new = DirsMultiset::from_manifest(&manifest).unwrap();
let expected = DirsMultiset { let expected = DirsMultiset {
inner: FastHashMap::default(), inner: FastHashMap::default(),
}; };
assert_eq!(expected, new); assert_eq!(expected, new);
let new = DirsMultiset::from_dirstate( let new = DirsMultiset::from_dirstate::<_, HgPathBuf>(
StateMap::default().into_iter().map(Ok), std::iter::empty(),
false, false,
) )
.unwrap(); .unwrap();
let expected = DirsMultiset { let expected = DirsMultiset {
inner: FastHashMap::default(), inner: FastHashMap::default(),
}; };
assert_eq!(expected, new); assert_eq!(expected, new);
} }

rust/hg-core/src/dirstate/dirstate_map.rs

  • This file was deleted.
// dirstate_map.rs
//
// Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
//
// This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version.
use crate::dirstate::parsers::Timestamp;
use crate::errors::HgError;
use crate::{
dirstate::EntryState,
dirstate::SIZE_FROM_OTHER_PARENT,
dirstate::SIZE_NON_NORMAL,
pack_dirstate, parse_dirstate,
utils::hg_path::{HgPath, HgPathBuf},
CopyMap, DirsMultiset, DirstateEntry, DirstateError, DirstateParents,
StateMap,
};
use micro_timer::timed;
use std::iter::FromIterator;
use std::ops::Deref;
#[derive(Default)]
pub struct DirstateMap {
state_map: StateMap,
pub copy_map: CopyMap,
pub dirs: Option<DirsMultiset>,
pub all_dirs: Option<DirsMultiset>,
}
/// Should only really be used in python interface code, for clarity
impl Deref for DirstateMap {
type Target = StateMap;
fn deref(&self) -> &Self::Target {
&self.state_map
}
}
impl FromIterator<(HgPathBuf, DirstateEntry)> for DirstateMap {
fn from_iter<I: IntoIterator<Item = (HgPathBuf, DirstateEntry)>>(
iter: I,
) -> Self {
Self {
state_map: iter.into_iter().collect(),
..Self::default()
}
}
}
impl DirstateMap {
pub fn new() -> Self {
Self::default()
}
pub fn clear(&mut self) {
self.state_map = StateMap::default();
self.copy_map.clear();
}
pub fn set_entry(&mut self, filename: &HgPath, entry: DirstateEntry) {
self.state_map.insert(filename.to_owned(), entry);
}
/// Add a tracked file to the dirstate
pub fn add_file(
&mut self,
filename: &HgPath,
entry: DirstateEntry,
) -> Result<(), DirstateError> {
let old_state = self.get(filename).map(|e| e.state());
if old_state.is_none() || old_state == Some(EntryState::Removed) {
if let Some(ref mut dirs) = self.dirs {
dirs.add_path(filename)?;
}
}
if old_state.is_none() {
if let Some(ref mut all_dirs) = self.all_dirs {
all_dirs.add_path(filename)?;
}
}
self.state_map.insert(filename.to_owned(), entry.to_owned());
Ok(())
}
/// Mark a file as removed in the dirstate.
///
/// The `size` parameter is used to store sentinel values that indicate
/// the file's previous state. In the future, we should refactor this
/// to be more explicit about what that state is.
pub fn remove_file(
&mut self,
filename: &HgPath,
in_merge: bool,
) -> Result<(), DirstateError> {
let old_entry_opt = self.get(filename);
let old_state = old_entry_opt.map(|e| e.state());
let mut size = 0;
if in_merge {
// XXX we should not be able to have 'm' state and 'FROM_P2' if not
// during a merge. So I (marmoute) am not sure we need the
// conditionnal at all. Adding double checking this with assert
// would be nice.
if let Some(old_entry) = old_entry_opt {
// backup the previous state
if old_entry.state() == EntryState::Merged {
size = SIZE_NON_NORMAL;
} else if old_entry.state() == EntryState::Normal
&& old_entry.size() == SIZE_FROM_OTHER_PARENT
{
// other parent
size = SIZE_FROM_OTHER_PARENT;
}
}
}
if old_state.is_some() && old_state != Some(EntryState::Removed) {
if let Some(ref mut dirs) = self.dirs {
dirs.delete_path(filename)?;
}
}
if old_state.is_none() {
if let Some(ref mut all_dirs) = self.all_dirs {
all_dirs.add_path(filename)?;
}
}
if size == 0 {
self.copy_map.remove(filename);
}
self.state_map
.insert(filename.to_owned(), DirstateEntry::new_removed(size));
Ok(())
}
/// Remove a file from the dirstate.
/// Returns `true` if the file was previously recorded.
pub fn drop_entry_and_copy_source(
&mut self,
filename: &HgPath,
) -> Result<(), DirstateError> {
let old_state = self.get(filename).map(|e| e.state());
let exists = self.state_map.remove(filename).is_some();
if exists {
if old_state != Some(EntryState::Removed) {
if let Some(ref mut dirs) = self.dirs {
dirs.delete_path(filename)?;
}
}
if let Some(ref mut all_dirs) = self.all_dirs {
all_dirs.delete_path(filename)?;
}
}
self.copy_map.remove(filename);
Ok(())
}
/// Both of these setters and their uses appear to be the simplest way to
/// emulate a Python lazy property, but it is ugly and unidiomatic.
/// TODO One day, rewriting this struct using the typestate might be a
/// good idea.
pub fn set_all_dirs(&mut self) -> Result<(), DirstateError> {
if self.all_dirs.is_none() {
self.all_dirs = Some(DirsMultiset::from_dirstate(
self.state_map.iter().map(|(k, v)| Ok((k, *v))),
false,
)?);
}
Ok(())
}
pub fn set_dirs(&mut self) -> Result<(), DirstateError> {
if self.dirs.is_none() {
self.dirs = Some(DirsMultiset::from_dirstate(
self.state_map.iter().map(|(k, v)| Ok((k, *v))),
true,
)?);
}
Ok(())
}
pub fn has_tracked_dir(
&mut self,
directory: &HgPath,
) -> Result<bool, DirstateError> {
self.set_dirs()?;
Ok(self.dirs.as_ref().unwrap().contains(directory))
}
pub fn has_dir(
&mut self,
directory: &HgPath,
) -> Result<bool, DirstateError> {
self.set_all_dirs()?;
Ok(self.all_dirs.as_ref().unwrap().contains(directory))
}
#[timed]
pub fn read(
&mut self,
file_contents: &[u8],
) -> Result<Option<DirstateParents>, DirstateError> {
if file_contents.is_empty() {
return Ok(None);
}
let (parents, entries, copies) = parse_dirstate(file_contents)?;
self.state_map.extend(
entries
.into_iter()
.map(|(path, entry)| (path.to_owned(), entry)),
);
self.copy_map.extend(
copies
.into_iter()
.map(|(path, copy)| (path.to_owned(), copy.to_owned())),
);
Ok(Some(parents.clone()))
}
pub fn pack(
&mut self,
parents: DirstateParents,
now: Timestamp,
) -> Result<Vec<u8>, HgError> {
pack_dirstate(&mut self.state_map, &self.copy_map, parents, now)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_dirs_multiset() {
let mut map = DirstateMap::new();
assert!(map.dirs.is_none());
assert!(map.all_dirs.is_none());
assert_eq!(map.has_dir(HgPath::new(b"nope")).unwrap(), false);
assert!(map.all_dirs.is_some());
assert!(map.dirs.is_none());
assert_eq!(map.has_tracked_dir(HgPath::new(b"nope")).unwrap(), false);
assert!(map.dirs.is_some());
}
#[test]
fn test_add_file() {
let mut map = DirstateMap::new();
assert_eq!(0, map.len());
map.add_file(
HgPath::new(b"meh"),
DirstateEntry::from_v1_data(EntryState::Normal, 1337, 1337, 1337),
)
.unwrap();
assert_eq!(1, map.len());
}
}

rust/hg-core/src/dirstate/parsers.rs

// Copyright 2019 Raphaël Gomès <rgomes@octobus.net> // Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
use crate::errors::HgError; use crate::errors::HgError;
use crate::utils::hg_path::HgPath; use crate::utils::hg_path::HgPath;
use crate::{ use crate::{dirstate::EntryState, DirstateEntry, DirstateParents};
dirstate::{CopyMap, EntryState, StateMap},
DirstateEntry, DirstateParents,
};
use byteorder::{BigEndian, WriteBytesExt}; use byteorder::{BigEndian, WriteBytesExt};
use bytes_cast::{unaligned, BytesCast}; use bytes_cast::{unaligned, BytesCast};
use micro_timer::timed; use micro_timer::timed;
use std::convert::{TryFrom, TryInto}; use std::convert::TryFrom;
/// Parents are stored in the dirstate as byte hashes. /// Parents are stored in the dirstate as byte hashes.
pub const PARENT_SIZE: usize = 20; pub const PARENT_SIZE: usize = 20;
/// Dirstate entries have a static part of 8 + 32 + 32 + 32 + 32 bits. /// Dirstate entries have a static part of 8 + 32 + 32 + 32 + 32 bits.
const MIN_ENTRY_SIZE: usize = 17; const MIN_ENTRY_SIZE: usize = 17;
type ParseResult<'a> = ( type ParseResult<'a> = (
&'a DirstateParents, &'a DirstateParents,
if let Some(source) = copy_source { if let Some(source) = copy_source {
packed.push(b'\0'); packed.push(b'\0');
packed.extend(source.as_bytes()); packed.extend(source.as_bytes());
} }
} }
/// Seconds since the Unix epoch /// Seconds since the Unix epoch
pub struct Timestamp(pub i64); pub struct Timestamp(pub i64);
pub fn pack_dirstate(
state_map: &mut StateMap,
copy_map: &CopyMap,
parents: DirstateParents,
now: Timestamp,
) -> Result<Vec<u8>, HgError> {
// TODO move away from i32 before 2038.
let now: i32 = now.0.try_into().expect("time overflow");
let expected_size: usize = state_map
.iter()
.map(|(filename, _)| {
packed_entry_size(filename, copy_map.get(filename).map(|p| &**p))
})
.sum();
let expected_size = expected_size + PARENT_SIZE * 2;
let mut packed = Vec::with_capacity(expected_size);
packed.extend(parents.p1.as_bytes());
packed.extend(parents.p2.as_bytes());
for (filename, entry) in state_map.iter_mut() {
entry.clear_ambiguous_mtime(now);
pack_entry(
filename,
entry,
copy_map.get(filename).map(|p| &**p),
&mut packed,
)
}
if packed.len() != expected_size {
return Err(HgError::CorruptedRepository(format!(
"bad dirstate size: {} != {}",
expected_size,
packed.len()
)));
}
Ok(packed)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{utils::hg_path::HgPathBuf, FastHashMap};
use pretty_assertions::assert_eq;
#[test]
fn test_pack_dirstate_empty() {
let mut state_map = StateMap::default();
let copymap = FastHashMap::default();
let parents = DirstateParents {
p1: b"12345678910111213141".into(),
p2: b"00000000000000000000".into(),
};
let now = Timestamp(15000000);
let expected = b"1234567891011121314100000000000000000000".to_vec();
assert_eq!(
expected,
pack_dirstate(&mut state_map, &copymap, parents, now).unwrap()
);
assert!(state_map.is_empty())
}
#[test]
fn test_pack_dirstate_one_entry() {
let expected_state_map: StateMap = [(
HgPathBuf::from_bytes(b"f1"),
DirstateEntry::from_v1_data(
EntryState::Normal,
0o644,
0,
791231220,
),
)]
.iter()
.cloned()
.collect();
let mut state_map = expected_state_map.clone();
let copymap = FastHashMap::default();
let parents = DirstateParents {
p1: b"12345678910111213141".into(),
p2: b"00000000000000000000".into(),
};
let now = Timestamp(15000000);
let expected = [
49, 50, 51, 52, 53, 54, 55, 56, 57, 49, 48, 49, 49, 49, 50, 49,
51, 49, 52, 49, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48,
48, 48, 48, 48, 48, 48, 48, 48, 110, 0, 0, 1, 164, 0, 0, 0, 0, 47,
41, 58, 244, 0, 0, 0, 2, 102, 49,
]
.to_vec();
assert_eq!(
expected,
pack_dirstate(&mut state_map, &copymap, parents, now).unwrap()
);
assert_eq!(expected_state_map, state_map);
}
#[test]
fn test_pack_dirstate_one_entry_with_copy() {
let expected_state_map: StateMap = [(
HgPathBuf::from_bytes(b"f1"),
DirstateEntry::from_v1_data(
EntryState::Normal,
0o644,
0,
791231220,
),
)]
.iter()
.cloned()
.collect();
let mut state_map = expected_state_map.clone();
let mut copymap = FastHashMap::default();
copymap.insert(
HgPathBuf::from_bytes(b"f1"),
HgPathBuf::from_bytes(b"copyname"),
);
let parents = DirstateParents {
p1: b"12345678910111213141".into(),
p2: b"00000000000000000000".into(),
};
let now = Timestamp(15000000);
let expected = [
49, 50, 51, 52, 53, 54, 55, 56, 57, 49, 48, 49, 49, 49, 50, 49,
51, 49, 52, 49, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48,
48, 48, 48, 48, 48, 48, 48, 48, 110, 0, 0, 1, 164, 0, 0, 0, 0, 47,
41, 58, 244, 0, 0, 0, 11, 102, 49, 0, 99, 111, 112, 121, 110, 97,
109, 101,
]
.to_vec();
assert_eq!(
expected,
pack_dirstate(&mut state_map, &copymap, parents, now).unwrap()
);
assert_eq!(expected_state_map, state_map);
}
#[test]
fn test_parse_pack_one_entry_with_copy() {
let mut state_map: StateMap = [(
HgPathBuf::from_bytes(b"f1"),
DirstateEntry::from_v1_data(
EntryState::Normal,
0o644,
0,
791231220,
),
)]
.iter()
.cloned()
.collect();
let mut copymap = FastHashMap::default();
copymap.insert(
HgPathBuf::from_bytes(b"f1"),
HgPathBuf::from_bytes(b"copyname"),
);
let parents = DirstateParents {
p1: b"12345678910111213141".into(),
p2: b"00000000000000000000".into(),
};
let now = Timestamp(15000000);
let result =
pack_dirstate(&mut state_map, &copymap, parents.clone(), now)
.unwrap();
let (new_parents, entries, copies) =
parse_dirstate(result.as_slice()).unwrap();
let new_state_map: StateMap = entries
.into_iter()
.map(|(path, entry)| (path.to_owned(), entry))
.collect();
let new_copy_map: CopyMap = copies
.into_iter()
.map(|(path, copy)| (path.to_owned(), copy.to_owned()))
.collect();
assert_eq!(
(&parents, state_map, copymap),
(new_parents, new_state_map, new_copy_map)
)
}
#[test]
fn test_parse_pack_multiple_entries_with_copy() {
let mut state_map: StateMap = [
(
HgPathBuf::from_bytes(b"f1"),
DirstateEntry::from_v1_data(
EntryState::Normal,
0o644,
0,
791231220,
),
),
(
HgPathBuf::from_bytes(b"f2"),
DirstateEntry::from_v1_data(
EntryState::Merged,
0o777,
1000,
791231220,
),
),
(
HgPathBuf::from_bytes(b"f3"),
DirstateEntry::from_v1_data(
EntryState::Removed,
0o644,
234553,
791231220,
),
),
(
HgPathBuf::from_bytes(b"f4\xF6"),
DirstateEntry::from_v1_data(EntryState::Added, 0o644, -1, -1),
),
]
.iter()
.cloned()
.collect();
let mut copymap = FastHashMap::default();
copymap.insert(
HgPathBuf::from_bytes(b"f1"),
HgPathBuf::from_bytes(b"copyname"),
);
copymap.insert(
HgPathBuf::from_bytes(b"f4\xF6"),
HgPathBuf::from_bytes(b"copyname2"),
);
let parents = DirstateParents {
p1: b"12345678910111213141".into(),
p2: b"00000000000000000000".into(),
};
let now = Timestamp(15000000);
let result =
pack_dirstate(&mut state_map, &copymap, parents.clone(), now)
.unwrap();
let (new_parents, entries, copies) =
parse_dirstate(result.as_slice()).unwrap();
let new_state_map: StateMap = entries
.into_iter()
.map(|(path, entry)| (path.to_owned(), entry))
.collect();
let new_copy_map: CopyMap = copies
.into_iter()
.map(|(path, copy)| (path.to_owned(), copy.to_owned()))
.collect();
assert_eq!(
(&parents, state_map, copymap),
(new_parents, new_state_map, new_copy_map)
)
}
#[test]
/// https://www.mercurial-scm.org/repo/hg/rev/af3f26b6bba4
fn test_parse_pack_one_entry_with_copy_and_time_conflict() {
let mut state_map: StateMap = [(
HgPathBuf::from_bytes(b"f1"),
DirstateEntry::from_v1_data(
EntryState::Normal,
0o644,
0,
15000000,
),
)]
.iter()
.cloned()
.collect();
let mut copymap = FastHashMap::default();
copymap.insert(
HgPathBuf::from_bytes(b"f1"),
HgPathBuf::from_bytes(b"copyname"),
);
let parents = DirstateParents {
p1: b"12345678910111213141".into(),
p2: b"00000000000000000000".into(),
};
let now = Timestamp(15000000);
let result =
pack_dirstate(&mut state_map, &copymap, parents.clone(), now)
.unwrap();
let (new_parents, entries, copies) =
parse_dirstate(result.as_slice()).unwrap();
let new_state_map: StateMap = entries
.into_iter()
.map(|(path, entry)| (path.to_owned(), entry))
.collect();
let new_copy_map: CopyMap = copies
.into_iter()
.map(|(path, copy)| (path.to_owned(), copy.to_owned()))
.collect();
assert_eq!(
(
&parents,
[(
HgPathBuf::from_bytes(b"f1"),
DirstateEntry::from_v1_data(
EntryState::Normal,
0o644,
0,
-1
)
)]
.iter()
.cloned()
.collect::<StateMap>(),
copymap,
),
(new_parents, new_state_map, new_copy_map)
)
}
}

rust/hg-core/src/dirstate/status.rs

// status.rs // status.rs
// //
// Copyright 2019 Raphaël Gomès <rgomes@octobus.net> // Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
//! Rust implementation of dirstate.status (dirstate.py). //! Rust implementation of dirstate.status (dirstate.py).
//! It is currently missing a lot of functionality compared to the Python one //! It is currently missing a lot of functionality compared to the Python one
//! and will only be triggered in narrow cases. //! and will only be triggered in narrow cases.
use crate::dirstate_tree::on_disk::DirstateV2ParseError; use crate::dirstate_tree::on_disk::DirstateV2ParseError;
use crate::utils::path_auditor::PathAuditor;
use crate::{ use crate::{
dirstate::SIZE_FROM_OTHER_PARENT, utils::hg_path::{HgPath, HgPathError},
filepatterns::PatternFileWarning,
matchers::{get_ignore_function, Matcher, VisitChildrenSet},
utils::{
files::{find_dirs, HgMetadata},
hg_path::{
hg_path_to_path_buf, os_string_to_hg_path_buf, HgPath, HgPathBuf,
HgPathError,
},
},
CopyMap, DirstateEntry, DirstateMap, EntryState, FastHashMap,
PatternError, PatternError,
}; };
use lazy_static::lazy_static;
use micro_timer::timed; use std::{borrow::Cow, fmt};
use rayon::prelude::*;
use std::{
borrow::Cow,
collections::HashSet,
fmt,
fs::{read_dir, DirEntry},
io::ErrorKind,
ops::Deref,
path::{Path, PathBuf},
};
/// Wrong type of file from a `BadMatch` /// Wrong type of file from a `BadMatch`
/// Note: a lot of those don't exist on all platforms. /// Note: a lot of those don't exist on all platforms.
#[derive(Debug, Copy, Clone)] #[derive(Debug, Copy, Clone)]
pub enum BadType { pub enum BadType {
CharacterDevice, CharacterDevice,
BlockDevice, BlockDevice,
FIFO, FIFO,
/// Was explicitly matched but cannot be found/accessed /// Was explicitly matched but cannot be found/accessed
#[derive(Debug, Copy, Clone)] #[derive(Debug, Copy, Clone)]
pub enum BadMatch { pub enum BadMatch {
OsError(i32), OsError(i32),
BadType(BadType), BadType(BadType),
} }
/// Enum used to dispatch new status entries into the right collections.
/// Is similar to `crate::EntryState`, but represents the transient state of
/// entries during the lifetime of a command.
#[derive(Debug, Copy, Clone)]
pub enum Dispatch {
Unsure,
Modified,
Added,
Removed,
Deleted,
Clean,
Unknown,
Ignored,
/// Empty dispatch, the file is not worth listing
None,
/// Was explicitly matched but cannot be found/accessed
Bad(BadMatch),
Directory {
/// True if the directory used to be a file in the dmap so we can say
/// that it's been removed.
was_file: bool,
},
}
type IoResult<T> = std::io::Result<T>;
/// `Box<dyn Trait>` is syntactic sugar for `Box<dyn Trait + 'static>`, so add /// `Box<dyn Trait>` is syntactic sugar for `Box<dyn Trait + 'static>`, so add
/// an explicit lifetime here to not fight `'static` bounds "out of nowhere". /// an explicit lifetime here to not fight `'static` bounds "out of nowhere".
pub type IgnoreFnType<'a> = pub type IgnoreFnType<'a> =
Box<dyn for<'r> Fn(&'r HgPath) -> bool + Sync + 'a>; Box<dyn for<'r> Fn(&'r HgPath) -> bool + Sync + 'a>;
/// We have a good mix of owned (from directory traversal) and borrowed (from /// We have a good mix of owned (from directory traversal) and borrowed (from
/// the dirstate/explicit) paths, this comes up a lot. /// the dirstate/explicit) paths, this comes up a lot.
pub type HgPathCow<'a> = Cow<'a, HgPath>; pub type HgPathCow<'a> = Cow<'a, HgPath>;
/// A path with its computed ``Dispatch`` information
type DispatchedPath<'a> = (HgPathCow<'a>, Dispatch);
/// The conversion from `HgPath` to a real fs path failed.
/// `22` is the error code for "Invalid argument"
const INVALID_PATH_DISPATCH: Dispatch = Dispatch::Bad(BadMatch::OsError(22));
/// Dates and times that are outside the 31-bit signed range are compared
/// modulo 2^31. This should prevent hg from behaving badly with very large
/// files or corrupt dates while still having a high probability of detecting
/// changes. (issue2608)
/// TODO I haven't found a way of having `b` be `Into<i32>`, since `From<u64>`
/// is not defined for `i32`, and there is no `As` trait. This forces the
/// caller to cast `b` as `i32`.
fn mod_compare(a: i32, b: i32) -> bool {
a & i32::max_value() != b & i32::max_value()
}
/// Return a sorted list containing information about the entries
/// in the directory.
///
/// * `skip_dot_hg` - Return an empty vec if `path` contains a `.hg` directory
fn list_directory(
path: impl AsRef<Path>,
skip_dot_hg: bool,
) -> std::io::Result<Vec<(HgPathBuf, DirEntry)>> {
let mut results = vec![];
let entries = read_dir(path.as_ref())?;
for entry in entries {
let entry = entry?;
let filename = os_string_to_hg_path_buf(entry.file_name())?;
let file_type = entry.file_type()?;
if skip_dot_hg && filename.as_bytes() == b".hg" && file_type.is_dir() {
return Ok(vec![]);
} else {
results.push((filename, entry))
}
}
results.sort_unstable_by_key(|e| e.0.clone());
Ok(results)
}
/// The file corresponding to the dirstate entry was found on the filesystem.
fn dispatch_found(
filename: impl AsRef<HgPath>,
entry: DirstateEntry,
metadata: HgMetadata,
copy_map: &CopyMap,
options: StatusOptions,
) -> Dispatch {
match entry.state() {
EntryState::Normal => {
let mode = entry.mode();
let size = entry.size();
let mtime = entry.mtime();
let HgMetadata {
st_mode,
st_size,
st_mtime,
..
} = metadata;
let size_changed = mod_compare(size, st_size as i32);
let mode_changed =
(mode ^ st_mode as i32) & 0o100 != 0o000 && options.check_exec;
let metadata_changed = size >= 0 && (size_changed || mode_changed);
let other_parent = size == SIZE_FROM_OTHER_PARENT;
if metadata_changed
|| other_parent
|| copy_map.contains_key(filename.as_ref())
{
if metadata.is_symlink() && size_changed {
// issue6456: Size returned may be longer due to encryption
// on EXT-4 fscrypt. TODO maybe only do it on EXT4?
Dispatch::Unsure
} else {
Dispatch::Modified
}
} else if mod_compare(mtime, st_mtime as i32)
|| st_mtime == options.last_normal_time
{
// the file may have just been marked as normal and
// it may have changed in the same second without
// changing its size. This can happen if we quickly
// do multiple commits. Force lookup, so we don't
// miss such a racy file change.
Dispatch::Unsure
} else if options.list_clean {
Dispatch::Clean
} else {
Dispatch::None
}
}
EntryState::Merged => Dispatch::Modified,
EntryState::Added => Dispatch::Added,
EntryState::Removed => Dispatch::Removed,
}
}
/// The file corresponding to this Dirstate entry is missing.
fn dispatch_missing(state: EntryState) -> Dispatch {
match state {
// File was removed from the filesystem during commands
EntryState::Normal | EntryState::Merged | EntryState::Added => {
Dispatch::Deleted
}
// File was removed, everything is normal
EntryState::Removed => Dispatch::Removed,
}
}
fn dispatch_os_error(e: &std::io::Error) -> Dispatch {
Dispatch::Bad(BadMatch::OsError(
e.raw_os_error().expect("expected real OS error"),
))
}
lazy_static! {
static ref DEFAULT_WORK: HashSet<&'static HgPath> = {
let mut h = HashSet::new();
h.insert(HgPath::new(b""));
h
};
}
#[derive(Debug, Copy, Clone)] #[derive(Debug, Copy, Clone)]
pub struct StatusOptions { pub struct StatusOptions {
/// Remember the most recent modification timeslot for status, to make /// Remember the most recent modification timeslot for status, to make
/// sure we won't miss future size-preserving file content modifications /// sure we won't miss future size-preserving file content modifications
/// that happen within the same timeslot. /// that happen within the same timeslot.
pub last_normal_time: i64, pub last_normal_time: i64,
/// Whether we are on a filesystem with UNIX-like exec flags /// Whether we are on a filesystem with UNIX-like exec flags
pub check_exec: bool, pub check_exec: bool,
StatusError::Path(error) => error.fmt(f), StatusError::Path(error) => error.fmt(f),
StatusError::Pattern(error) => error.fmt(f), StatusError::Pattern(error) => error.fmt(f),
StatusError::DirstateV2ParseError(_) => { StatusError::DirstateV2ParseError(_) => {
f.write_str("dirstate-v2 parse error") f.write_str("dirstate-v2 parse error")
} }
} }
} }
} }
/// Gives information about which files are changed in the working directory
/// and how, compared to the revision we're based on
pub struct Status<'a, M: ?Sized + Matcher + Sync> {
dmap: &'a DirstateMap,
pub(crate) matcher: &'a M,
root_dir: PathBuf,
pub(crate) options: StatusOptions,
ignore_fn: IgnoreFnType<'a>,
}
impl<'a, M> Status<'a, M>
where
M: ?Sized + Matcher + Sync,
{
pub fn new(
dmap: &'a DirstateMap,
matcher: &'a M,
root_dir: PathBuf,
ignore_files: Vec<PathBuf>,
options: StatusOptions,
) -> StatusResult<(Self, Vec<PatternFileWarning>)> {
// Needs to outlive `dir_ignore_fn` since it's captured.
let (ignore_fn, warnings): (IgnoreFnType, _) =
if options.list_ignored || options.list_unknown {
get_ignore_function(ignore_files, &root_dir, &mut |_| {})?
} else {
(Box::new(|&_| true), vec![])
};
Ok((
Self {
dmap,
matcher,
root_dir,
options,
ignore_fn,
},
warnings,
))
}
/// Is the path ignored?
pub fn is_ignored(&self, path: impl AsRef<HgPath>) -> bool {
(self.ignore_fn)(path.as_ref())
}
/// Is the path or one of its ancestors ignored?
pub fn dir_ignore(&self, dir: impl AsRef<HgPath>) -> bool {
// Only involve ignore mechanism if we're listing unknowns or ignored.
if self.options.list_ignored || self.options.list_unknown {
if self.is_ignored(&dir) {
true
} else {
for p in find_dirs(dir.as_ref()) {
if self.is_ignored(p) {
return true;
}
}
false
}
} else {
true
}
}
/// Get stat data about the files explicitly specified by the matcher.
/// Returns a tuple of the directories that need to be traversed and the
/// files with their corresponding `Dispatch`.
/// TODO subrepos
#[timed]
pub fn walk_explicit(
&self,
traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
) -> (Vec<DispatchedPath<'a>>, Vec<DispatchedPath<'a>>) {
self.matcher
.file_set()
.unwrap_or(&DEFAULT_WORK)
.par_iter()
.flat_map(|&filename| -> Option<_> {
// TODO normalization
let normalized = filename;
let buf = match hg_path_to_path_buf(normalized) {
Ok(x) => x,
Err(_) => {
return Some((
Cow::Borrowed(normalized),
INVALID_PATH_DISPATCH,
))
}
};
let target = self.root_dir.join(buf);
let st = target.symlink_metadata();
let in_dmap = self.dmap.get(normalized);
match st {
Ok(meta) => {
let file_type = meta.file_type();
return if file_type.is_file() || file_type.is_symlink()
{
if let Some(entry) = in_dmap {
return Some((
Cow::Borrowed(normalized),
dispatch_found(
&normalized,
*entry,
HgMetadata::from_metadata(meta),
&self.dmap.copy_map,
self.options,
),
));
}
Some((
Cow::Borrowed(normalized),
Dispatch::Unknown,
))
} else if file_type.is_dir() {
if self.options.collect_traversed_dirs {
traversed_sender
.send(normalized.to_owned())
.expect("receiver should outlive sender");
}
Some((
Cow::Borrowed(normalized),
Dispatch::Directory {
was_file: in_dmap.is_some(),
},
))
} else {
Some((
Cow::Borrowed(normalized),
Dispatch::Bad(BadMatch::BadType(
// TODO do more than unknown
// Support for all `BadType` variant
// varies greatly between platforms.
// So far, no tests check the type and
// this should be good enough for most
// users.
BadType::Unknown,
)),
))
};
}
Err(_) => {
if let Some(entry) = in_dmap {
return Some((
Cow::Borrowed(normalized),
dispatch_missing(entry.state()),
));
}
}
};
None
})
.partition(|(_, dispatch)| match dispatch {
Dispatch::Directory { .. } => true,
_ => false,
})
}
/// Walk the working directory recursively to look for changes compared to
/// the current `DirstateMap`.
///
/// This takes a mutable reference to the results to account for the
/// `extend` in timings
#[timed]
pub fn traverse(
&self,
path: impl AsRef<HgPath>,
old_results: &FastHashMap<HgPathCow<'a>, Dispatch>,
results: &mut Vec<DispatchedPath<'a>>,
traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
) {
// The traversal is done in parallel, so use a channel to gather
// entries. `crossbeam_channel::Sender` is `Sync`, while `mpsc::Sender`
// is not.
let (files_transmitter, files_receiver) =
crossbeam_channel::unbounded();
self.traverse_dir(
&files_transmitter,
path,
&old_results,
traversed_sender,
);
// Disconnect the channel so the receiver stops waiting
drop(files_transmitter);
let new_results = files_receiver
.into_iter()
.par_bridge()
.map(|(f, d)| (Cow::Owned(f), d));
results.par_extend(new_results);
}
/// Dispatch a single entry (file, folder, symlink...) found during
/// `traverse`. If the entry is a folder that needs to be traversed, it
/// will be handled in a separate thread.
fn handle_traversed_entry<'b>(
&'a self,
scope: &rayon::Scope<'b>,
files_sender: &'b crossbeam_channel::Sender<(HgPathBuf, Dispatch)>,
old_results: &'a FastHashMap<Cow<HgPath>, Dispatch>,
filename: HgPathBuf,
dir_entry: DirEntry,
traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
) -> IoResult<()>
where
'a: 'b,
{
let file_type = dir_entry.file_type()?;
let entry_option = self.dmap.get(&filename);
if filename.as_bytes() == b".hg" {
// Could be a directory or a symlink
return Ok(());
}
if file_type.is_dir() {
self.handle_traversed_dir(
scope,
files_sender,
old_results,
entry_option,
filename,
traversed_sender,
);
} else if file_type.is_file() || file_type.is_symlink() {
if let Some(entry) = entry_option {
if self.matcher.matches_everything()
|| self.matcher.matches(&filename)
{
let metadata = dir_entry.metadata()?;
files_sender
.send((
filename.to_owned(),
dispatch_found(
&filename,
*entry,
HgMetadata::from_metadata(metadata),
&self.dmap.copy_map,
self.options,
),
))
.unwrap();
}
} else if (self.matcher.matches_everything()
|| self.matcher.matches(&filename))
&& !self.is_ignored(&filename)
{
if (self.options.list_ignored
|| self.matcher.exact_match(&filename))
&& self.dir_ignore(&filename)
{
if self.options.list_ignored {
files_sender
.send((filename.to_owned(), Dispatch::Ignored))
.unwrap();
}
} else if self.options.list_unknown {
files_sender
.send((filename.to_owned(), Dispatch::Unknown))
.unwrap();
}
} else if self.is_ignored(&filename) && self.options.list_ignored {
if self.matcher.matches(&filename) {
files_sender
.send((filename.to_owned(), Dispatch::Ignored))
.unwrap();
}
}
} else if let Some(entry) = entry_option {
// Used to be a file or a folder, now something else.
if self.matcher.matches_everything()
|| self.matcher.matches(&filename)
{
files_sender
.send((
filename.to_owned(),
dispatch_missing(entry.state()),
))
.unwrap();
}
}
Ok(())
}
/// A directory was found in the filesystem and needs to be traversed
fn handle_traversed_dir<'b>(
&'a self,
scope: &rayon::Scope<'b>,
files_sender: &'b crossbeam_channel::Sender<(HgPathBuf, Dispatch)>,
old_results: &'a FastHashMap<Cow<HgPath>, Dispatch>,
entry_option: Option<&'a DirstateEntry>,
directory: HgPathBuf,
traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
) where
'a: 'b,
{
scope.spawn(move |_| {
// Nested `if` until `rust-lang/rust#53668` is stable
if let Some(entry) = entry_option {
// Used to be a file, is now a folder
if self.matcher.matches_everything()
|| self.matcher.matches(&directory)
{
files_sender
.send((
directory.to_owned(),
dispatch_missing(entry.state()),
))
.unwrap();
}
}
// Do we need to traverse it?
if !self.is_ignored(&directory) || self.options.list_ignored {
self.traverse_dir(
files_sender,
directory,
&old_results,
traversed_sender,
)
}
});
}
/// Decides whether the directory needs to be listed, and if so handles the
/// entries in a separate thread.
fn traverse_dir(
&self,
files_sender: &crossbeam_channel::Sender<(HgPathBuf, Dispatch)>,
directory: impl AsRef<HgPath>,
old_results: &FastHashMap<Cow<HgPath>, Dispatch>,
traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
) {
let directory = directory.as_ref();
if self.options.collect_traversed_dirs {
traversed_sender
.send(directory.to_owned())
.expect("receiver should outlive sender");
}
let visit_entries = match self.matcher.visit_children_set(directory) {
VisitChildrenSet::Empty => return,
VisitChildrenSet::This | VisitChildrenSet::Recursive => None,
VisitChildrenSet::Set(set) => Some(set),
};
let buf = match hg_path_to_path_buf(directory) {
Ok(b) => b,
Err(_) => {
files_sender
.send((directory.to_owned(), INVALID_PATH_DISPATCH))
.expect("receiver should outlive sender");
return;
}
};
let dir_path = self.root_dir.join(buf);
let skip_dot_hg = !directory.as_bytes().is_empty();
let entries = match list_directory(dir_path, skip_dot_hg) {
Err(e) => {
files_sender
.send((directory.to_owned(), dispatch_os_error(&e)))
.expect("receiver should outlive sender");
return;
}
Ok(entries) => entries,
};
rayon::scope(|scope| {
for (filename, dir_entry) in entries {
if let Some(ref set) = visit_entries {
if !set.contains(filename.deref()) {
continue;
}
}
// TODO normalize
let filename = if directory.is_empty() {
filename.to_owned()
} else {
directory.join(&filename)
};
if !old_results.contains_key(filename.deref()) {
match self.handle_traversed_entry(
scope,
files_sender,
old_results,
filename,
dir_entry,
traversed_sender.clone(),
) {
Err(e) => {
files_sender
.send((
directory.to_owned(),
dispatch_os_error(&e),
))
.expect("receiver should outlive sender");
}
Ok(_) => {}
}
}
}
})
}
/// Add the files in the dirstate to the results.
///
/// This takes a mutable reference to the results to account for the
/// `extend` in timings
#[timed]
pub fn extend_from_dmap(&self, results: &mut Vec<DispatchedPath<'a>>) {
results.par_extend(
self.dmap
.par_iter()
.filter(|(path, _)| self.matcher.matches(path))
.map(move |(filename, entry)| {
let filename: &HgPath = filename;
let filename_as_path = match hg_path_to_path_buf(filename)
{
Ok(f) => f,
Err(_) => {
return (
Cow::Borrowed(filename),
INVALID_PATH_DISPATCH,
)
}
};
let meta = self
.root_dir
.join(filename_as_path)
.symlink_metadata();
match meta {
Ok(m)
if !(m.file_type().is_file()
|| m.file_type().is_symlink()) =>
{
(
Cow::Borrowed(filename),
dispatch_missing(entry.state()),
)
}
Ok(m) => (
Cow::Borrowed(filename),
dispatch_found(
filename,
*entry,
HgMetadata::from_metadata(m),
&self.dmap.copy_map,
self.options,
),
),
Err(e)
if e.kind() == ErrorKind::NotFound
|| e.raw_os_error() == Some(20) =>
{
// Rust does not yet have an `ErrorKind` for
// `NotADirectory` (errno 20)
// It happens if the dirstate contains `foo/bar`
// and foo is not a
// directory
(
Cow::Borrowed(filename),
dispatch_missing(entry.state()),
)
}
Err(e) => {
(Cow::Borrowed(filename), dispatch_os_error(&e))
}
}
}),
);
}
/// Checks all files that are in the dirstate but were not found during the
/// working directory traversal. This means that the rest must
/// be either ignored, under a symlink or under a new nested repo.
///
/// This takes a mutable reference to the results to account for the
/// `extend` in timings
#[timed]
pub fn handle_unknowns(&self, results: &mut Vec<DispatchedPath<'a>>) {
let to_visit: Vec<(&HgPath, &DirstateEntry)> =
if results.is_empty() && self.matcher.matches_everything() {
self.dmap.iter().map(|(f, e)| (f.deref(), e)).collect()
} else {
// Only convert to a hashmap if needed.
let old_results: FastHashMap<_, _> =
results.iter().cloned().collect();
self.dmap
.iter()
.filter_map(move |(f, e)| {
if !old_results.contains_key(f.deref())
&& self.matcher.matches(f)
{
Some((f.deref(), e))
} else {
None
}
})
.collect()
};
let path_auditor = PathAuditor::new(&self.root_dir);
let new_results = to_visit.into_par_iter().filter_map(
|(filename, entry)| -> Option<_> {
// Report ignored items in the dmap as long as they are not
// under a symlink directory.
if path_auditor.check(filename) {
// TODO normalize for case-insensitive filesystems
let buf = match hg_path_to_path_buf(filename) {
Ok(x) => x,
Err(_) => {
return Some((
Cow::Owned(filename.to_owned()),
INVALID_PATH_DISPATCH,
));
}
};
Some((
Cow::Owned(filename.to_owned()),
match self.root_dir.join(&buf).symlink_metadata() {
// File was just ignored, no links, and exists
Ok(meta) => {
let metadata = HgMetadata::from_metadata(meta);
dispatch_found(
filename,
*entry,
metadata,
&self.dmap.copy_map,
self.options,
)
}
// File doesn't exist
Err(_) => dispatch_missing(entry.state()),
},
))
} else {
// It's either missing or under a symlink directory which
// we, in this case, report as missing.
Some((
Cow::Owned(filename.to_owned()),
dispatch_missing(entry.state()),
))
}
},
);
results.par_extend(new_results);
}
}
#[timed]
pub fn build_response<'a>(
results: impl IntoIterator<Item = DispatchedPath<'a>>,
traversed: Vec<HgPathCow<'a>>,
) -> DirstateStatus<'a> {
let mut unsure = vec![];
let mut modified = vec![];
let mut added = vec![];
let mut removed = vec![];
let mut deleted = vec![];
let mut clean = vec![];
let mut ignored = vec![];
let mut unknown = vec![];
let mut bad = vec![];
for (filename, dispatch) in results.into_iter() {
match dispatch {
Dispatch::Unknown => unknown.push(filename),
Dispatch::Unsure => unsure.push(filename),
Dispatch::Modified => modified.push(filename),
Dispatch::Added => added.push(filename),
Dispatch::Removed => removed.push(filename),
Dispatch::Deleted => deleted.push(filename),
Dispatch::Clean => clean.push(filename),
Dispatch::Ignored => ignored.push(filename),
Dispatch::None => {}
Dispatch::Bad(reason) => bad.push((filename, reason)),
Dispatch::Directory { .. } => {}
}
}
DirstateStatus {
modified,
added,
removed,
deleted,
clean,
ignored,
unknown,
bad,
unsure,
traversed,
dirty: false,
}
}
/// Get the status of files in the working directory.
///
/// This is the current entry-point for `hg-core` and is realistically unusable
/// outside of a Python context because its arguments need to provide a lot of
/// information that will not be necessary in the future.
#[timed]
pub fn status<'a>(
dmap: &'a DirstateMap,
matcher: &'a (dyn Matcher + Sync),
root_dir: PathBuf,
ignore_files: Vec<PathBuf>,
options: StatusOptions,
) -> StatusResult<(DirstateStatus<'a>, Vec<PatternFileWarning>)> {
let (status, warnings) =
Status::new(dmap, matcher, root_dir, ignore_files, options)?;
Ok((status.run()?, warnings))
}

rust/hg-core/src/dirstate_tree/dirstate_map.rs

use bytes_cast::BytesCast; use bytes_cast::BytesCast;
use micro_timer::timed; use micro_timer::timed;
use std::borrow::Cow; use std::borrow::Cow;
use std::convert::TryInto; use std::convert::TryInto;
use std::path::PathBuf; use std::path::PathBuf;
use super::on_disk; use super::on_disk;
use super::on_disk::DirstateV2ParseError; use super::on_disk::DirstateV2ParseError;
use super::path_with_basename::WithBasename; use super::path_with_basename::WithBasename;
use crate::dirstate::parsers::pack_entry; use crate::dirstate::parsers::pack_entry;
use crate::dirstate::parsers::packed_entry_size; use crate::dirstate::parsers::packed_entry_size;
use crate::dirstate::parsers::parse_dirstate_entries; use crate::dirstate::parsers::parse_dirstate_entries;
use crate::dirstate::parsers::Timestamp; use crate::dirstate::parsers::Timestamp;
use crate::dirstate::CopyMapIter;
use crate::dirstate::StateMapIter;
use crate::dirstate::SIZE_FROM_OTHER_PARENT; use crate::dirstate::SIZE_FROM_OTHER_PARENT;
use crate::dirstate::SIZE_NON_NORMAL; use crate::dirstate::SIZE_NON_NORMAL;
use crate::matchers::Matcher; use crate::matchers::Matcher;
use crate::utils::hg_path::{HgPath, HgPathBuf}; use crate::utils::hg_path::{HgPath, HgPathBuf};
use crate::CopyMapIter;
use crate::DirstateEntry; use crate::DirstateEntry;
use crate::DirstateError; use crate::DirstateError;
use crate::DirstateParents; use crate::DirstateParents;
use crate::DirstateStatus; use crate::DirstateStatus;
use crate::EntryState; use crate::EntryState;
use crate::FastHashMap; use crate::FastHashMap;
use crate::PatternFileWarning; use crate::PatternFileWarning;
use crate::StateMapIter;
use crate::StatusError; use crate::StatusError;
use crate::StatusOptions; use crate::StatusOptions;
/// Append to an existing data file if the amount of unreachable data (not used /// Append to an existing data file if the amount of unreachable data (not used
/// anymore) is less than this fraction of the total amount of existing data. /// anymore) is less than this fraction of the total amount of existing data.
const ACCEPTABLE_UNREACHABLE_BYTES_RATIO: f32 = 0.5; const ACCEPTABLE_UNREACHABLE_BYTES_RATIO: f32 = 0.5;
pub struct DirstateMap<'on_disk> { pub struct DirstateMap<'on_disk> {

rust/hg-core/src/dirstate_tree/dispatch.rs

use std::path::PathBuf; use std::path::PathBuf;
use crate::dirstate::parsers::Timestamp; use crate::dirstate::parsers::Timestamp;
use crate::dirstate::CopyMapIter;
use crate::dirstate::StateMapIter;
use crate::dirstate_tree::on_disk::DirstateV2ParseError; use crate::dirstate_tree::on_disk::DirstateV2ParseError;
use crate::matchers::Matcher; use crate::matchers::Matcher;
use crate::utils::hg_path::{HgPath, HgPathBuf}; use crate::utils::hg_path::{HgPath, HgPathBuf};
use crate::CopyMapIter;
use crate::DirstateEntry; use crate::DirstateEntry;
use crate::DirstateError; use crate::DirstateError;
use crate::DirstateMap;
use crate::DirstateParents; use crate::DirstateParents;
use crate::DirstateStatus; use crate::DirstateStatus;
use crate::PatternFileWarning; use crate::PatternFileWarning;
use crate::StateMapIter;
use crate::StatusError; use crate::StatusError;
use crate::StatusOptions; use crate::StatusOptions;
/// `rust/hg-cpython/src/dirstate/dirstate_map.rs` implements in Rust a /// `rust/hg-cpython/src/dirstate/dirstate_map.rs` implements in Rust a
/// `DirstateMap` Python class that wraps `Box<dyn DirstateMapMethods + Send>`, /// `DirstateMap` Python class that wraps `Box<dyn DirstateMapMethods + Send>`,
/// a trait object of this trait. Except for constructors, this trait defines /// a trait object of this trait. Except for constructors, this trait defines
/// all APIs that the class needs to interact with its inner dirstate map. /// all APIs that the class needs to interact with its inner dirstate map.
/// ///
Item = Result< Item = Result<
(&HgPath, (u8, i32, i32, i32)), (&HgPath, (u8, i32, i32, i32)),
DirstateV2ParseError, DirstateV2ParseError,
>, >,
> + Send > + Send
+ '_, + '_,
>; >;
} }
impl DirstateMapMethods for DirstateMap {
fn clear(&mut self) {
self.clear()
}
/// Used to set a value directory.
///
/// XXX Is temporary during a refactor of V1 dirstate and will disappear
/// shortly.
fn set_entry(
&mut self,
filename: &HgPath,
entry: DirstateEntry,
) -> Result<(), DirstateV2ParseError> {
self.set_entry(&filename, entry);
Ok(())
}
fn add_file(
&mut self,
filename: &HgPath,
entry: DirstateEntry,
) -> Result<(), DirstateError> {
self.add_file(filename, entry)
}
fn remove_file(
&mut self,
filename: &HgPath,
in_merge: bool,
) -> Result<(), DirstateError> {
self.remove_file(filename, in_merge)
}
fn drop_entry_and_copy_source(
&mut self,
filename: &HgPath,
) -> Result<(), DirstateError> {
self.drop_entry_and_copy_source(filename)
}
fn has_tracked_dir(
&mut self,
directory: &HgPath,
) -> Result<bool, DirstateError> {
self.has_tracked_dir(directory)
}
fn has_dir(&mut self, directory: &HgPath) -> Result<bool, DirstateError> {
self.has_dir(directory)
}
fn pack_v1(
&mut self,
parents: DirstateParents,
now: Timestamp,
) -> Result<Vec<u8>, DirstateError> {
Ok(self.pack(parents, now)?)
}
fn pack_v2(
&mut self,
_now: Timestamp,
_can_append: bool,
) -> Result<(Vec<u8>, Vec<u8>, bool), DirstateError> {
panic!(
"should have used dirstate_tree::DirstateMap to use the v2 format"
)
}
fn status<'a>(
&'a mut self,
matcher: &'a (dyn Matcher + Sync),
root_dir: PathBuf,
ignore_files: Vec<PathBuf>,
options: StatusOptions,
) -> Result<(DirstateStatus<'a>, Vec<PatternFileWarning>), StatusError>
{
crate::status(self, matcher, root_dir, ignore_files, options)
}
fn copy_map_len(&self) -> usize {
self.copy_map.len()
}
fn copy_map_iter(&self) -> CopyMapIter<'_> {
Box::new(
self.copy_map
.iter()
.map(|(key, value)| Ok((&**key, &**value))),
)
}
fn copy_map_contains_key(
&self,
key: &HgPath,
) -> Result<bool, DirstateV2ParseError> {
Ok(self.copy_map.contains_key(key))
}
fn copy_map_get(
&self,
key: &HgPath,
) -> Result<Option<&HgPath>, DirstateV2ParseError> {
Ok(self.copy_map.get(key).map(|p| &**p))
}
fn copy_map_remove(
&mut self,
key: &HgPath,
) -> Result<Option<HgPathBuf>, DirstateV2ParseError> {
Ok(self.copy_map.remove(key))
}
fn copy_map_insert(
&mut self,
key: HgPathBuf,
value: HgPathBuf,
) -> Result<Option<HgPathBuf>, DirstateV2ParseError> {
Ok(self.copy_map.insert(key, value))
}
fn len(&self) -> usize {
(&**self).len()
}
fn contains_key(
&self,
key: &HgPath,
) -> Result<bool, DirstateV2ParseError> {
Ok((&**self).contains_key(key))
}
fn get(
&self,
key: &HgPath,
) -> Result<Option<DirstateEntry>, DirstateV2ParseError> {
Ok((&**self).get(key).cloned())
}
fn iter(&self) -> StateMapIter<'_> {
Box::new((&**self).iter().map(|(key, value)| Ok((&**key, *value))))
}
fn iter_tracked_dirs(
&mut self,
) -> Result<
Box<
dyn Iterator<Item = Result<&HgPath, DirstateV2ParseError>>
+ Send
+ '_,
>,
DirstateError,
> {
self.set_all_dirs()?;
Ok(Box::new(
self.all_dirs
.as_ref()
.unwrap()
.iter()
.map(|path| Ok(&**path)),
))
}
fn debug_iter(
&self,
all: bool,
) -> Box<
dyn Iterator<
Item = Result<
(&HgPath, (u8, i32, i32, i32)),
DirstateV2ParseError,
>,
> + Send
+ '_,
> {
// Not used for the flat (not tree-based) DirstateMap
let _ = all;
Box::new(
(&**self)
.iter()
.map(|(path, entry)| Ok((&**path, entry.debug_tuple()))),
)
}
}

rust/hg-core/src/dirstate_tree/owning_dispatch.rs

use crate::dirstate::parsers::Timestamp; use crate::dirstate::parsers::Timestamp;
use crate::dirstate::CopyMapIter;
use crate::dirstate::StateMapIter;
use crate::dirstate_tree::dispatch::DirstateMapMethods; use crate::dirstate_tree::dispatch::DirstateMapMethods;
use crate::dirstate_tree::on_disk::DirstateV2ParseError; use crate::dirstate_tree::on_disk::DirstateV2ParseError;
use crate::dirstate_tree::owning::OwningDirstateMap; use crate::dirstate_tree::owning::OwningDirstateMap;
use crate::matchers::Matcher; use crate::matchers::Matcher;
use crate::utils::hg_path::{HgPath, HgPathBuf}; use crate::utils::hg_path::{HgPath, HgPathBuf};
use crate::CopyMapIter;
use crate::DirstateEntry; use crate::DirstateEntry;
use crate::DirstateError; use crate::DirstateError;
use crate::DirstateParents; use crate::DirstateParents;
use crate::DirstateStatus; use crate::DirstateStatus;
use crate::PatternFileWarning; use crate::PatternFileWarning;
use crate::StateMapIter;
use crate::StatusError; use crate::StatusError;
use crate::StatusOptions; use crate::StatusOptions;
use std::path::PathBuf; use std::path::PathBuf;
impl DirstateMapMethods for OwningDirstateMap { impl DirstateMapMethods for OwningDirstateMap {
fn clear(&mut self) { fn clear(&mut self) {
self.get_mut().clear() self.get_mut().clear()
} }

rust/hg-core/src/lib.rs

pub mod dirstate; pub mod dirstate;
pub mod dirstate_tree; pub mod dirstate_tree;
pub mod discovery; pub mod discovery;
pub mod exit_codes; pub mod exit_codes;
pub mod requirements; pub mod requirements;
pub mod testing; // unconditionally built, for use from integration tests pub mod testing; // unconditionally built, for use from integration tests
pub use dirstate::{ pub use dirstate::{
dirs_multiset::{DirsMultiset, DirsMultisetIter}, dirs_multiset::{DirsMultiset, DirsMultisetIter},
dirstate_map::DirstateMap,
parsers::{pack_dirstate, parse_dirstate, PARENT_SIZE},
status::{ status::{
status, BadMatch, BadType, DirstateStatus, HgPathCow, StatusError, BadMatch, BadType, DirstateStatus, HgPathCow, StatusError,
StatusOptions, StatusOptions,
}, },
CopyMap, CopyMapIter, DirstateEntry, DirstateParents, EntryState, DirstateEntry, DirstateParents, EntryState,
StateMap, StateMapIter,
}; };
pub mod copy_tracing; pub mod copy_tracing;
mod filepatterns; mod filepatterns;
pub mod matchers; pub mod matchers;
pub mod repo; pub mod repo;
pub mod revlog; pub mod revlog;
pub use revlog::*; pub use revlog::*;
pub mod config; pub mod config;

rust/hg-core/src/operations/dirstate_status.rs

  • This file was deleted.
// dirstate_status.rs
//
// Copyright 2019, Raphaël Gomès <rgomes@octobus.net>
//
// This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version.
use crate::dirstate::status::{build_response, Dispatch, Status};
use crate::matchers::Matcher;
use crate::{DirstateStatus, StatusError};
impl<'a, M: ?Sized + Matcher + Sync> Status<'a, M> {
pub(crate) fn run(&self) -> Result<DirstateStatus<'a>, StatusError> {
let (traversed_sender, traversed_receiver) =
crossbeam_channel::unbounded();
// Step 1: check the files explicitly mentioned by the user
let (work, mut results) = self.walk_explicit(traversed_sender.clone());
if !work.is_empty() {
// Hashmaps are quite a bit slower to build than vecs, so only
// build it if needed.
let old_results = results.iter().cloned().collect();
// Step 2: recursively check the working directory for changes if
// needed
for (dir, dispatch) in work {
match dispatch {
Dispatch::Directory { was_file } => {
if was_file {
results.push((dir.to_owned(), Dispatch::Removed));
}
if self.options.list_ignored
|| self.options.list_unknown
&& !self.dir_ignore(&dir)
{
self.traverse(
&dir,
&old_results,
&mut results,
traversed_sender.clone(),
);
}
}
_ => {
unreachable!("There can only be directories in `work`")
}
}
}
}
if !self.matcher.is_exact() {
if self.options.list_unknown {
self.handle_unknowns(&mut results);
} else {
// TODO this is incorrect, see issue6335
// This requires a fix in both Python and Rust that can happen
// with other pending changes to `status`.
self.extend_from_dmap(&mut results);
}
}
drop(traversed_sender);
let traversed = traversed_receiver
.into_iter()
.map(std::borrow::Cow::Owned)
.collect();
Ok(build_response(results, traversed))
}
}

rust/hg-core/src/operations/mod.rs

//! A distinction is made between operations and commands. //! A distinction is made between operations and commands.
//! An operation is what can be done whereas a command is what is exposed by //! An operation is what can be done whereas a command is what is exposed by
//! the cli. A single command can use several operations to achieve its goal. //! the cli. A single command can use several operations to achieve its goal.
mod cat; mod cat;
mod debugdata; mod debugdata;
mod dirstate_status;
mod list_tracked_files; mod list_tracked_files;
pub use cat::{cat, CatOutput}; pub use cat::{cat, CatOutput};
pub use debugdata::{debug_data, DebugDataKind}; pub use debugdata::{debug_data, DebugDataKind};
pub use list_tracked_files::Dirstate; pub use list_tracked_files::Dirstate;
pub use list_tracked_files::{list_rev_tracked_files, FilesForRev}; pub use list_tracked_files::{list_rev_tracked_files, FilesForRev};

rust/hg-cpython/src/dirstate/copymap.rs

use cpython::{ use cpython::{
PyBytes, PyClone, PyDict, PyObject, PyResult, Python, UnsafePyLeaked, PyBytes, PyClone, PyDict, PyObject, PyResult, Python, UnsafePyLeaked,
}; };
use std::cell::RefCell; use std::cell::RefCell;
use crate::dirstate::dirstate_map::v2_error; use crate::dirstate::dirstate_map::v2_error;
use crate::dirstate::dirstate_map::DirstateMap; use crate::dirstate::dirstate_map::DirstateMap;
use hg::dirstate::CopyMapIter;
use hg::dirstate_tree::on_disk::DirstateV2ParseError; use hg::dirstate_tree::on_disk::DirstateV2ParseError;
use hg::utils::hg_path::HgPath; use hg::utils::hg_path::HgPath;
use hg::CopyMapIter;
py_class!(pub class CopyMap |py| { py_class!(pub class CopyMap |py| {
data dirstate_map: DirstateMap; data dirstate_map: DirstateMap;
def __getitem__(&self, key: PyObject) -> PyResult<PyBytes> { def __getitem__(&self, key: PyObject) -> PyResult<PyBytes> {
(*self.dirstate_map(py)).copymapgetitem(py, key) (*self.dirstate_map(py)).copymapgetitem(py, key)
} }

rust/hg-cpython/src/dirstate/dirstate_map.rs

use crate::{ use crate::{
dirstate::copymap::{CopyMap, CopyMapItemsIterator, CopyMapKeysIterator}, dirstate::copymap::{CopyMap, CopyMapItemsIterator, CopyMapKeysIterator},
dirstate::item::DirstateItem, dirstate::item::DirstateItem,
pybytes_deref::PyBytesDeref, pybytes_deref::PyBytesDeref,
}; };
use hg::{ use hg::{
dirstate::parsers::Timestamp, dirstate::parsers::Timestamp,
dirstate::StateMapIter,
dirstate_tree::dirstate_map::DirstateMap as TreeDirstateMap, dirstate_tree::dirstate_map::DirstateMap as TreeDirstateMap,
dirstate_tree::dispatch::DirstateMapMethods, dirstate_tree::dispatch::DirstateMapMethods,
dirstate_tree::on_disk::DirstateV2ParseError, dirstate_tree::on_disk::DirstateV2ParseError,
dirstate_tree::owning::OwningDirstateMap, dirstate_tree::owning::OwningDirstateMap,
revlog::Node, revlog::Node,
utils::files::normalize_case, utils::files::normalize_case,
utils::hg_path::{HgPath, HgPathBuf}, utils::hg_path::{HgPath, HgPathBuf},
DirstateEntry, DirstateError, DirstateMap as RustDirstateMap, DirstateEntry, DirstateError, DirstateParents, EntryState,
DirstateParents, EntryState, StateMapIter,
}; };
// TODO // TODO
// This object needs to share references to multiple members of its Rust // This object needs to share references to multiple members of its Rust
// inner struct, namely `copy_map`, `dirs` and `all_dirs`. // inner struct, namely `copy_map`, `dirs` and `all_dirs`.
// Right now `CopyMap` is done, but it needs to have an explicit reference // Right now `CopyMap` is done, but it needs to have an explicit reference
// to `RustDirstateMap` which itself needs to have an encapsulation for // to `RustDirstateMap` which itself needs to have an encapsulation for
// every method in `CopyMap` (copymapcopy, etc.). // every method in `CopyMap` (copymapcopy, etc.).
// This is ugly and hard to maintain. // This is ugly and hard to maintain.
// The same logic applies to `dirs` and `all_dirs`, however the `Dirs` // The same logic applies to `dirs` and `all_dirs`, however the `Dirs`
// `py_class!` is already implemented and does not mention // `py_class!` is already implemented and does not mention
// `RustDirstateMap`, rightfully so. // `RustDirstateMap`, rightfully so.
// All attributes also have to have a separate refcount data attribute for // All attributes also have to have a separate refcount data attribute for
// leaks, with all methods that go along for reference sharing. // leaks, with all methods that go along for reference sharing.
py_class!(pub class DirstateMap |py| { py_class!(pub class DirstateMap |py| {
@shared data inner: Box<dyn DirstateMapMethods + Send>; @shared data inner: Box<dyn DirstateMapMethods + Send>;
/// Returns a `(dirstate_map, parents)` tuple /// Returns a `(dirstate_map, parents)` tuple
@staticmethod @staticmethod
def new_v1( def new_v1(
use_dirstate_tree: bool,
on_disk: PyBytes, on_disk: PyBytes,
) -> PyResult<PyObject> { ) -> PyResult<PyObject> {
let (inner, parents) = if use_dirstate_tree {
let on_disk = PyBytesDeref::new(py, on_disk); let on_disk = PyBytesDeref::new(py, on_disk);
let mut map = OwningDirstateMap::new_empty(on_disk); let mut map = OwningDirstateMap::new_empty(on_disk);
let (on_disk, map_placeholder) = map.get_mut_pair(); let (on_disk, map_placeholder) = map.get_mut_pair();
let (actual_map, parents) = TreeDirstateMap::new_v1(on_disk) let (actual_map, parents) = TreeDirstateMap::new_v1(on_disk)
.map_err(|e| dirstate_error(py, e))?; .map_err(|e| dirstate_error(py, e))?;
*map_placeholder = actual_map; *map_placeholder = actual_map;
(Box::new(map) as _, parents) let map = Self::create_instance(py, Box::new(map))?;
} else {
let bytes = on_disk.data(py);
let mut map = RustDirstateMap::default();
let parents = map.read(bytes).map_err(|e| dirstate_error(py, e))?;
(Box::new(map) as _, parents)
};
let map = Self::create_instance(py, inner)?;
let parents = parents.map(|p| { let parents = parents.map(|p| {
let p1 = PyBytes::new(py, p.p1.as_bytes()); let p1 = PyBytes::new(py, p.p1.as_bytes());
let p2 = PyBytes::new(py, p.p2.as_bytes()); let p2 = PyBytes::new(py, p.p2.as_bytes());
(p1, p2) (p1, p2)
}); });
Ok((map, parents).to_py_object(py).into_object()) Ok((map, parents).to_py_object(py).into_object())
} }

tests/test-dirstate-race.t

#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
$ hg init repo $ hg init repo

tests/test-dirstate-race2.t

#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
Checking the size/permissions/file-type of files stored in the Checking the size/permissions/file-type of files stored in the

tests/test-dirstate.t

#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
------ Test dirstate._dirs refcounting ------ Test dirstate._dirs refcounting

tests/test-hgignore.t

#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
$ hg init ignorerepo $ hg init ignorerepo

tests/test-permissions.t

#require unix-permissions no-root reporevlogstore #require unix-permissions no-root reporevlogstore
#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
$ hg init t $ hg init t

tests/test-purge.t

#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
init init

tests/test-status.t

#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if no-rust #if no-rust
$ hg init repo0 --config format.exp-dirstate-v2=1 $ hg init repo0 --config format.exp-dirstate-v2=1
abort: dirstate v2 format requested by config but not supported (requires Rust extensions) abort: dirstate v2 format requested by config but not supported (requires Rust extensions)
[255] [255]
#endif #endif
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
$ hg init repo1 $ hg init repo1
$ cd repo1 $ cd repo1

tests/test-symlinks.t

#require symlink #require symlink
#testcases dirstate-v1 dirstate-v1-tree dirstate-v2 #testcases dirstate-v1 dirstate-v2
#if dirstate-v1-tree
#require rust
$ echo '[experimental]' >> $HGRCPATH
$ echo 'dirstate-tree.in-memory=1' >> $HGRCPATH
#endif
#if dirstate-v2 #if dirstate-v2
#require rust #require rust
$ echo '[format]' >> $HGRCPATH $ echo '[format]' >> $HGRCPATH
$ echo 'exp-dirstate-v2=1' >> $HGRCPATH $ echo 'exp-dirstate-v2=1' >> $HGRCPATH
#endif #endif
== tests added in 0.7 == == tests added in 0.7 ==
rust/hg-core/src/lib.rs