( )⚙ D11097 dirstate-v2: Add heuristic for when to create a new data file

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11097

dirstate-v2: Add heuristic for when to create a new data file
ClosedPublic

Authored by SimonSapin on Jul 15 2021, 11:28 AM.

Details

Reviewers
marmoute
Group Reviewers
hg-reviewers
Commits
rHGd94118365ec5: dirstate-v2: Add heuristic for when to create a new data file
Summary

… instead of appending to the existing one. This is based on keeping track
of how much of the existing data is not used anymore.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

SimonSapin created this revision.Jul 15 2021, 11:28 AM
SimonSapin retitled this revision from dirstate-v2: Add heuristic for when to a new data file to dirstate-v2: Add heuristic for when to create a new data file.Jul 15 2021, 1:11 PM
SimonSapin updated this revision to Diff 29292.
baymax updated this revision to Diff 29322.Jul 16 2021, 5:39 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

marmoute requested changes to this revision.Jul 16 2021, 6:19 AM
marmoute added a subscriber: marmoute.
marmoute added inline comments.
rust/hg-core/src/dirstate_tree/dirstate_map.rs
52–53

Maybe point out in the commentthat this is an estimate, not an exact value ?

131

We should document this, (and point out this is an estimate)

488

We should put this ratio in some constant (at least) to make its value clearer and easier to configure.

rust/hg-core/src/dirstate_tree/on_disk.rs
76

same feedback about being clear this is an estimate.

This revision now requires changes to proceed.Jul 16 2021, 6:19 AM
SimonSapin added a comment.Jul 16 2021, 8:21 AM

At the moment I *think* unreachable_bytes should be exact. This might change if we change the serializer to reuse paths in more than one node each (within the same "version" of the tree): then when removing a node we wouldn’t know if that path is still reachable from somewhere else. (Unless we add some kind of reference-counting for paths, but I think it’s not worth it.)

SimonSapin updated this revision to Diff 29337.Jul 16 2021, 8:37 AM
Alphare added a subscriber: Alphare.Jul 16 2021, 9:10 AM
In D11097#170228, @SimonSapin wrote:

At the moment I *think* unreachable_bytes should be exact. This might change if we change the serializer to reuse paths in more than one node each (within the same "version" of the tree): then when removing a node we wouldn’t know if that path is still reachable from somewhere else. (Unless we add some kind of reference-counting for paths, but I think it’s not worth it.)

Probably let's just say that it might become an estimate later and that it's not that important anyway. The change looks good to me otherwise

baymax updated this revision to Diff 29354.Jul 16 2021, 5:45 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 29613.Jul 20 2021, 3:42 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

SimonSapin added a commit: rHGd94118365ec5: dirstate-v2: Add heuristic for when to create a new data file.Jul 20 2021, 3:50 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGd94118365ec5: dirstate-v2: Add heuristic for when to create a new data file (authored by SimonSapin). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 29613

rust/hg-core/src/dirstate_tree/dirstate_map.rs

use crate::DirstateStatus; use crate::DirstateStatus;
use crate::EntryState; use crate::EntryState;
use crate::FastHashMap; use crate::FastHashMap;
use crate::PatternFileWarning; use crate::PatternFileWarning;
use crate::StateMapIter; use crate::StateMapIter;
use crate::StatusError; use crate::StatusError;
use crate::StatusOptions; use crate::StatusOptions;
/// Append to an existing data file if the amount of unreachable data (not used
/// anymore) is less than this fraction of the total amount of existing data.
const ACCEPTABLE_UNREACHABLE_BYTES_RATIO: f32 = 0.5;
pub struct DirstateMap<'on_disk> { pub struct DirstateMap<'on_disk> {
/// Contents of the `.hg/dirstate` file /// Contents of the `.hg/dirstate` file
pub(super) on_disk: &'on_disk [u8], pub(super) on_disk: &'on_disk [u8],
pub(super) root: ChildNodes<'on_disk>, pub(super) root: ChildNodes<'on_disk>,
/// Number of nodes anywhere in the tree that have `.entry.is_some()`. /// Number of nodes anywhere in the tree that have `.entry.is_some()`.
pub(super) nodes_with_entry_count: u32, pub(super) nodes_with_entry_count: u32,
/// Number of nodes anywhere in the tree that have /// Number of nodes anywhere in the tree that have
/// `.copy_source.is_some()`. /// `.copy_source.is_some()`.
pub(super) nodes_with_copy_source_count: u32, pub(super) nodes_with_copy_source_count: u32,
/// See on_disk::Header /// See on_disk::Header
pub(super) ignore_patterns_hash: on_disk::IgnorePatternsHash, pub(super) ignore_patterns_hash: on_disk::IgnorePatternsHash,
/// How many bytes of `on_disk` are not used anymore
pub(super) unreachable_bytes: u32,
marmouteUnsubmitted
Not Done

Maybe point out in the commentthat this is an estimate, not an exact value ?

marmoute: Maybe point out in the commentthat this is an estimate, not an exact value ?
} }
/// Using a plain `HgPathBuf` of the full path from the repository root as a /// Using a plain `HgPathBuf` of the full path from the repository root as a
/// map key would also work: all paths in a given map have the same parent /// map key would also work: all paths in a given map have the same parent
/// path, so comparing full paths gives the same result as comparing base /// path, so comparing full paths gives the same result as comparing base
/// names. However `HashMap` would waste time always re-hashing the same /// names. However `HashMap` would waste time always re-hashing the same
/// string prefix. /// string prefix.
pub(super) type NodeKey<'on_disk> = WithBasename<Cow<'on_disk, HgPath>>; pub(super) type NodeKey<'on_disk> = WithBasename<Cow<'on_disk, HgPath>>;
pub(super) fn is_empty(&self) -> bool { pub(super) fn is_empty(&self) -> bool {
match self { match self {
ChildNodes::InMemory(nodes) => nodes.is_empty(), ChildNodes::InMemory(nodes) => nodes.is_empty(),
ChildNodes::OnDisk(nodes) => nodes.is_empty(), ChildNodes::OnDisk(nodes) => nodes.is_empty(),
} }
} }
pub(super) fn make_mut( fn make_mut(
&mut self, &mut self,
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
unreachable_bytes: &mut u32,
marmouteUnsubmitted
Not Done

We should document this, (and point out this is an estimate)

marmoute: We should document this, (and point out this is an estimate)
) -> Result< ) -> Result<
&mut FastHashMap<NodeKey<'on_disk>, Node<'on_disk>>, &mut FastHashMap<NodeKey<'on_disk>, Node<'on_disk>>,
DirstateV2ParseError, DirstateV2ParseError,
> { > {
match self { match self {
ChildNodes::InMemory(nodes) => Ok(nodes), ChildNodes::InMemory(nodes) => Ok(nodes),
ChildNodes::OnDisk(nodes) => { ChildNodes::OnDisk(nodes) => {
*unreachable_bytes +=
std::mem::size_of_val::<[on_disk::Node]>(nodes) as u32;
let nodes = nodes let nodes = nodes
.iter() .iter()
.map(|node| { .map(|node| {
Ok(( Ok((
node.path(on_disk)?, node.path(on_disk)?,
node.to_in_memory_node(on_disk)?, node.to_in_memory_node(on_disk)?,
)) ))
}) })
impl<'on_disk> DirstateMap<'on_disk> { impl<'on_disk> DirstateMap<'on_disk> {
pub(super) fn empty(on_disk: &'on_disk [u8]) -> Self { pub(super) fn empty(on_disk: &'on_disk [u8]) -> Self {
Self { Self {
on_disk, on_disk,
root: ChildNodes::default(), root: ChildNodes::default(),
nodes_with_entry_count: 0, nodes_with_entry_count: 0,
nodes_with_copy_source_count: 0, nodes_with_copy_source_count: 0,
ignore_patterns_hash: [0; on_disk::IGNORE_PATTERNS_HASH_LEN], ignore_patterns_hash: [0; on_disk::IGNORE_PATTERNS_HASH_LEN],
unreachable_bytes: 0,
} }
} }
#[timed] #[timed]
pub fn new_v2( pub fn new_v2(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
data_size: usize, data_size: usize,
) -> Result<Self, DirstateError> { ) -> Result<Self, DirstateError> {
} }
let parents = parse_dirstate_entries( let parents = parse_dirstate_entries(
map.on_disk, map.on_disk,
|path, entry, copy_source| { |path, entry, copy_source| {
let tracked = entry.state.is_tracked(); let tracked = entry.state.is_tracked();
let node = Self::get_or_insert_node( let node = Self::get_or_insert_node(
map.on_disk, map.on_disk,
&mut map.unreachable_bytes,
&mut map.root, &mut map.root,
path, path,
WithBasename::to_cow_borrowed, WithBasename::to_cow_borrowed,
|ancestor| { |ancestor| {
if tracked { if tracked {
ancestor.tracked_descendants_count += 1 ancestor.tracked_descendants_count += 1
} }
ancestor.descendants_with_entry_count += 1 ancestor.descendants_with_entry_count += 1
Ok((map, parents)) Ok((map, parents))
} }
/// Assuming dirstate-v2 format, returns whether the next write should /// Assuming dirstate-v2 format, returns whether the next write should
/// append to the existing data file that contains `self.on_disk` (true), /// append to the existing data file that contains `self.on_disk` (true),
/// or create a new data file from scratch (false). /// or create a new data file from scratch (false).
pub(super) fn write_should_append(&self) -> bool { pub(super) fn write_should_append(&self) -> bool {
// Soon this will be a heuristic based on the amount of unreachable let ratio = self.unreachable_bytes as f32 / self.on_disk.len() as f32;
// data. For now it’s pseudo-random in order to make tests exercise ratio < ACCEPTABLE_UNREACHABLE_BYTES_RATIO
marmouteUnsubmitted
Not Done

We should put this ratio in some constant (at least) to make its value clearer and easier to configure.

marmoute: We should put this ratio in some constant (at least) to make its value clearer and easier to…
// both code paths.
fn bad_rng() -> u32 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.subsec_millis()
}
bad_rng() % 2 == 0
} }
fn get_node<'tree>( fn get_node<'tree>(
&'tree self, &'tree self,
path: &HgPath, path: &HgPath,
) -> Result<Option<NodeRef<'tree, 'on_disk>>, DirstateV2ParseError> { ) -> Result<Option<NodeRef<'tree, 'on_disk>>, DirstateV2ParseError> {
let mut children = self.root.as_ref(); let mut children = self.root.as_ref();
let mut components = path.components(); let mut components = path.components();
} }
/// Returns a mutable reference to the node at `path` if it exists /// Returns a mutable reference to the node at `path` if it exists
/// ///
/// This takes `root` instead of `&mut self` so that callers can mutate /// This takes `root` instead of `&mut self` so that callers can mutate
/// other fields while the returned borrow is still valid /// other fields while the returned borrow is still valid
fn get_node_mut<'tree>( fn get_node_mut<'tree>(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
unreachable_bytes: &mut u32,
root: &'tree mut ChildNodes<'on_disk>, root: &'tree mut ChildNodes<'on_disk>,
path: &HgPath, path: &HgPath,
) -> Result<Option<&'tree mut Node<'on_disk>>, DirstateV2ParseError> { ) -> Result<Option<&'tree mut Node<'on_disk>>, DirstateV2ParseError> {
let mut children = root; let mut children = root;
let mut components = path.components(); let mut components = path.components();
let mut component = let mut component =
components.next().expect("expected at least one components"); components.next().expect("expected at least one components");
loop { loop {
if let Some(child) = children.make_mut(on_disk)?.get_mut(component) if let Some(child) = children
.make_mut(on_disk, unreachable_bytes)?
.get_mut(component)
{ {
if let Some(next_component) = components.next() { if let Some(next_component) = components.next() {
component = next_component; component = next_component;
children = &mut child.children; children = &mut child.children;
} else { } else {
return Ok(Some(child)); return Ok(Some(child));
} }
} else { } else {
return Ok(None); return Ok(None);
} }
} }
} }
pub(super) fn get_or_insert<'tree, 'path>( pub(super) fn get_or_insert<'tree, 'path>(
&'tree mut self, &'tree mut self,
path: &HgPath, path: &HgPath,
) -> Result<&'tree mut Node<'on_disk>, DirstateV2ParseError> { ) -> Result<&'tree mut Node<'on_disk>, DirstateV2ParseError> {
Self::get_or_insert_node( Self::get_or_insert_node(
self.on_disk, self.on_disk,
&mut self.unreachable_bytes,
&mut self.root, &mut self.root,
path, path,
WithBasename::to_cow_owned, WithBasename::to_cow_owned,
|_| {}, |_| {},
) )
} }
pub(super) fn get_or_insert_node<'tree, 'path>( fn get_or_insert_node<'tree, 'path>(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
unreachable_bytes: &mut u32,
root: &'tree mut ChildNodes<'on_disk>, root: &'tree mut ChildNodes<'on_disk>,
path: &'path HgPath, path: &'path HgPath,
to_cow: impl Fn( to_cow: impl Fn(
WithBasename<&'path HgPath>, WithBasename<&'path HgPath>,
) -> WithBasename<Cow<'on_disk, HgPath>>, ) -> WithBasename<Cow<'on_disk, HgPath>>,
mut each_ancestor: impl FnMut(&mut Node), mut each_ancestor: impl FnMut(&mut Node),
) -> Result<&'tree mut Node<'on_disk>, DirstateV2ParseError> { ) -> Result<&'tree mut Node<'on_disk>, DirstateV2ParseError> {
let mut child_nodes = root; let mut child_nodes = root;
let mut inclusive_ancestor_paths = let mut inclusive_ancestor_paths =
WithBasename::inclusive_ancestors_of(path); WithBasename::inclusive_ancestors_of(path);
let mut ancestor_path = inclusive_ancestor_paths let mut ancestor_path = inclusive_ancestor_paths
.next() .next()
.expect("expected at least one inclusive ancestor"); .expect("expected at least one inclusive ancestor");
loop { loop {
// TODO: can we avoid allocating an owned key in cases where the // TODO: can we avoid allocating an owned key in cases where the
// map already contains that key, without introducing double // map already contains that key, without introducing double
// lookup? // lookup?
let child_node = child_nodes let child_node = child_nodes
.make_mut(on_disk)? .make_mut(on_disk, unreachable_bytes)?
.entry(to_cow(ancestor_path)) .entry(to_cow(ancestor_path))
.or_default(); .or_default();
if let Some(next) = inclusive_ancestor_paths.next() { if let Some(next) = inclusive_ancestor_paths.next() {
each_ancestor(child_node); each_ancestor(child_node);
ancestor_path = next; ancestor_path = next;
child_nodes = &mut child_node.children; child_nodes = &mut child_node.children;
} else { } else {
return Ok(child_node); return Ok(child_node);
match (old_state.is_tracked(), new_entry.state.is_tracked()) { match (old_state.is_tracked(), new_entry.state.is_tracked()) {
(false, true) => 1, (false, true) => 1,
(true, false) => -1, (true, false) => -1,
_ => 0, _ => 0,
}; };
let node = Self::get_or_insert_node( let node = Self::get_or_insert_node(
self.on_disk, self.on_disk,
&mut self.unreachable_bytes,
&mut self.root, &mut self.root,
path, path,
WithBasename::to_cow_owned, WithBasename::to_cow_owned,
|ancestor| { |ancestor| {
if !had_entry { if !had_entry {
ancestor.descendants_with_entry_count += 1; ancestor.descendants_with_entry_count += 1;
} }
fn clear_known_ambiguous_mtimes( fn clear_known_ambiguous_mtimes(
&mut self, &mut self,
paths: &[impl AsRef<HgPath>], paths: &[impl AsRef<HgPath>],
) -> Result<(), DirstateV2ParseError> { ) -> Result<(), DirstateV2ParseError> {
for path in paths { for path in paths {
if let Some(node) = Self::get_node_mut( if let Some(node) = Self::get_node_mut(
self.on_disk, self.on_disk,
&mut self.unreachable_bytes,
&mut self.root, &mut self.root,
path.as_ref(), path.as_ref(),
)? { )? {
if let NodeData::Entry(entry) = &mut node.data { if let NodeData::Entry(entry) = &mut node.data {
entry.clear_mtime(); entry.clear_mtime();
} }
} }
} }
if let Some(entry) = node.entry()? { if let Some(entry) = node.entry()? {
if predicate(&entry) { if predicate(&entry) {
return Ok(Some(node.full_path(self.on_disk)?)); return Ok(Some(node.full_path(self.on_disk)?));
} }
} }
Ok(None) Ok(None)
}) })
} }
fn count_dropped_path(unreachable_bytes: &mut u32, path: &Cow<HgPath>) {
if let Cow::Borrowed(path) = path {
*unreachable_bytes += path.len() as u32
}
}
} }
/// Like `Iterator::filter_map`, but over a fallible iterator of `Result`s. /// Like `Iterator::filter_map`, but over a fallible iterator of `Result`s.
/// ///
/// The callback is only called for incoming `Ok` values. Errors are passed /// The callback is only called for incoming `Ok` values. Errors are passed
/// through as-is. In order to let it use the `?` operator the callback is /// through as-is. In order to let it use the `?` operator the callback is
/// expected to return a `Result` of `Option`, instead of an `Option` of /// expected to return a `Result` of `Option`, instead of an `Option` of
/// `Result`. /// `Result`.
/// If this returns `Ok(Some((dropped, removed)))`, then /// If this returns `Ok(Some((dropped, removed)))`, then
/// ///
/// * `dropped` is about the leaf node that was at `filename` /// * `dropped` is about the leaf node that was at `filename`
/// * `removed` is whether this particular level of recursion just /// * `removed` is whether this particular level of recursion just
/// removed a node in `nodes`. /// removed a node in `nodes`.
fn recur<'on_disk>( fn recur<'on_disk>(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
unreachable_bytes: &mut u32,
nodes: &mut ChildNodes<'on_disk>, nodes: &mut ChildNodes<'on_disk>,
path: &HgPath, path: &HgPath,
) -> Result<Option<(Dropped, bool)>, DirstateV2ParseError> { ) -> Result<Option<(Dropped, bool)>, DirstateV2ParseError> {
let (first_path_component, rest_of_path) = let (first_path_component, rest_of_path) =
path.split_first_component(); path.split_first_component();
let node = if let Some(node) = let nodes = nodes.make_mut(on_disk, unreachable_bytes)?;
nodes.make_mut(on_disk)?.get_mut(first_path_component) let node = if let Some(node) = nodes.get_mut(first_path_component)
{ {
node node
} else { } else {
return Ok(None); return Ok(None);
}; };
let dropped; let dropped;
if let Some(rest) = rest_of_path { if let Some(rest) = rest_of_path {
if let Some((d, removed)) = if let Some((d, removed)) = recur(
recur(on_disk, &mut node.children, rest)? on_disk,
{ unreachable_bytes,
&mut node.children,
rest,
)? {
dropped = d; dropped = d;
if dropped.had_entry { if dropped.had_entry {
node.descendants_with_entry_count -= 1; node.descendants_with_entry_count -= 1;
} }
if dropped.was_tracked { if dropped.was_tracked {
node.tracked_descendants_count -= 1; node.tracked_descendants_count -= 1;
} }
// Directory caches must be invalidated when removing a // Directory caches must be invalidated when removing a
// child node // child node
if removed { if removed {
if let NodeData::CachedDirectory { .. } = &node.data { if let NodeData::CachedDirectory { .. } = &node.data {
node.data = NodeData::None node.data = NodeData::None
} }
} }
} else { } else {
return Ok(None); return Ok(None);
} }
} else { } else {
let had_entry = node.data.has_entry(); let had_entry = node.data.has_entry();
if had_entry { if had_entry {
node.data = NodeData::None node.data = NodeData::None
} }
if let Some(source) = &node.copy_source {
DirstateMap::count_dropped_path(unreachable_bytes, source)
}
dropped = Dropped { dropped = Dropped {
was_tracked: node was_tracked: node
.data .data
.as_entry() .as_entry()
.map_or(false, |entry| entry.state.is_tracked()), .map_or(false, |entry| entry.state.is_tracked()),
had_entry, had_entry,
had_copy_source: node.copy_source.take().is_some(), had_copy_source: node.copy_source.take().is_some(),
}; };
} }
// After recursion, for both leaf (rest_of_path is None) nodes and // After recursion, for both leaf (rest_of_path is None) nodes and
// parent nodes, remove a node if it just became empty. // parent nodes, remove a node if it just became empty.
let remove = !node.data.has_entry() let remove = !node.data.has_entry()
&& node.copy_source.is_none() && node.copy_source.is_none()
&& node.children.is_empty(); && node.children.is_empty();
if remove { if remove {
nodes.make_mut(on_disk)?.remove(first_path_component); let (key, _) =
nodes.remove_entry(first_path_component).unwrap();
DirstateMap::count_dropped_path(
unreachable_bytes,
key.full_path(),
)
} }
Ok(Some((dropped, remove))) Ok(Some((dropped, remove)))
} }
if let Some((dropped, _removed)) = if let Some((dropped, _removed)) = recur(
recur(self.on_disk, &mut self.root, filename)? self.on_disk,
{ &mut self.unreachable_bytes,
&mut self.root,
filename,
)? {
if dropped.had_entry { if dropped.had_entry {
self.nodes_with_entry_count -= 1 self.nodes_with_entry_count -= 1
} }
if dropped.had_copy_source { if dropped.had_copy_source {
self.nodes_with_copy_source_count -= 1 self.nodes_with_copy_source_count -= 1
} }
Ok(dropped.had_entry) Ok(dropped.had_entry)
} else { } else {
debug_assert!(!old_state.is_tracked()); debug_assert!(!old_state.is_tracked());
Ok(false) Ok(false)
} }
} }
fn clear_ambiguous_times( fn clear_ambiguous_times(
&mut self, &mut self,
filenames: Vec<HgPathBuf>, filenames: Vec<HgPathBuf>,
now: i32, now: i32,
) -> Result<(), DirstateV2ParseError> { ) -> Result<(), DirstateV2ParseError> {
for filename in filenames { for filename in filenames {
if let Some(node) = if let Some(node) = Self::get_node_mut(
Self::get_node_mut(self.on_disk, &mut self.root, &filename)? self.on_disk,
{ &mut self.unreachable_bytes,
&mut self.root,
&filename,
)? {
if let NodeData::Entry(entry) = &mut node.data { if let NodeData::Entry(entry) = &mut node.data {
entry.clear_ambiguous_mtime(now); entry.clear_ambiguous_mtime(now);
} }
} }
} }
Ok(()) Ok(())
} }
Ok(None) Ok(None)
} }
fn copy_map_remove( fn copy_map_remove(
&mut self, &mut self,
key: &HgPath, key: &HgPath,
) -> Result<Option<HgPathBuf>, DirstateV2ParseError> { ) -> Result<Option<HgPathBuf>, DirstateV2ParseError> {
let count = &mut self.nodes_with_copy_source_count; let count = &mut self.nodes_with_copy_source_count;
Ok( let unreachable_bytes = &mut self.unreachable_bytes;
Self::get_node_mut(self.on_disk, &mut self.root, key)?.and_then( Ok(Self::get_node_mut(
|node| { self.on_disk,
if node.copy_source.is_some() { unreachable_bytes,
*count -= 1 &mut self.root,
key,
)?
.and_then(|node| {
if let Some(source) = &node.copy_source {
*count -= 1;
Self::count_dropped_path(unreachable_bytes, source);
} }
node.copy_source.take().map(Cow::into_owned) node.copy_source.take().map(Cow::into_owned)
}, }))
),
)
} }
fn copy_map_insert( fn copy_map_insert(
&mut self, &mut self,
key: HgPathBuf, key: HgPathBuf,
value: HgPathBuf, value: HgPathBuf,
) -> Result<Option<HgPathBuf>, DirstateV2ParseError> { ) -> Result<Option<HgPathBuf>, DirstateV2ParseError> {
let node = Self::get_or_insert_node( let node = Self::get_or_insert_node(
self.on_disk, self.on_disk,
&mut self.unreachable_bytes,
&mut self.root, &mut self.root,
&key, &key,
WithBasename::to_cow_owned, WithBasename::to_cow_owned,
|_ancestor| {}, |_ancestor| {},
)?; )?;
if node.copy_source.is_none() { if node.copy_source.is_none() {
self.nodes_with_copy_source_count += 1 self.nodes_with_copy_source_count += 1
} }

rust/hg-core/src/dirstate_tree/on_disk.rs

#[derive(BytesCast)] #[derive(BytesCast)]
#[repr(C)] #[repr(C)]
struct Root { struct Root {
root_nodes: ChildNodes, root_nodes: ChildNodes,
nodes_with_entry_count: Size, nodes_with_entry_count: Size,
nodes_with_copy_source_count: Size, nodes_with_copy_source_count: Size,
/// How many bytes of this data file are not used anymore
marmouteUnsubmitted
Not Done

same feedback about being clear this is an estimate.

marmoute: same feedback about being clear this is an estimate.
unreachable_bytes: Size,
/// If non-zero, a hash of ignore files that were used for some previous /// If non-zero, a hash of ignore files that were used for some previous
/// run of the `status` algorithm. /// run of the `status` algorithm.
/// ///
/// We define: /// We define:
/// ///
/// * "Root" ignore files are `.hgignore` at the root of the repository if /// * "Root" ignore files are `.hgignore` at the root of the repository if
/// it exists, and files from `ui.ignore.*` config. This set of files is /// it exists, and files from `ui.ignore.*` config. This set of files is
/// then sorted by the string representation of their path. /// then sorted by the string representation of their path.
} }
/// Either nothing if `start == 0`, or a `HgPath` of `len` bytes /// Either nothing if `start == 0`, or a `HgPath` of `len` bytes
type OptPathSlice = PathSlice; type OptPathSlice = PathSlice;
/// Make sure that size-affecting changes are made knowingly /// Make sure that size-affecting changes are made knowingly
fn _static_assert_size_of() { fn _static_assert_size_of() {
let _ = std::mem::transmute::<DocketHeader, [u8; 81]>; let _ = std::mem::transmute::<DocketHeader, [u8; 81]>;
let _ = std::mem::transmute::<Root, [u8; 36]>; let _ = std::mem::transmute::<Root, [u8; 40]>;
let _ = std::mem::transmute::<Node, [u8; 43]>; let _ = std::mem::transmute::<Node, [u8; 43]>;
} }
/// Unexpected file format found in `.hg/dirstate` with the "v2" format. /// Unexpected file format found in `.hg/dirstate` with the "v2" format.
/// ///
/// This should only happen if Mercurial is buggy or a repository is corrupted. /// This should only happen if Mercurial is buggy or a repository is corrupted.
#[derive(Debug)] #[derive(Debug)]
pub struct DirstateV2ParseError; pub struct DirstateV2ParseError;
pub(super) fn read<'on_disk>( pub(super) fn read<'on_disk>(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
) -> Result<DirstateMap<'on_disk>, DirstateV2ParseError> { ) -> Result<DirstateMap<'on_disk>, DirstateV2ParseError> {
if on_disk.is_empty() { if on_disk.is_empty() {
return Ok(DirstateMap::empty(on_disk)); return Ok(DirstateMap::empty(on_disk));
} }
let root = read_root(on_disk)?; let root = read_root(on_disk)?;
let mut unreachable_bytes = root.unreachable_bytes.get();
// Each append writes a new `Root`, so it’s never reused
unreachable_bytes += std::mem::size_of::<Root>() as u32;
let dirstate_map = DirstateMap { let dirstate_map = DirstateMap {
on_disk, on_disk,
root: dirstate_map::ChildNodes::OnDisk(read_nodes( root: dirstate_map::ChildNodes::OnDisk(read_nodes(
on_disk, on_disk,
root.root_nodes, root.root_nodes,
)?), )?),
nodes_with_entry_count: root.nodes_with_entry_count.get(), nodes_with_entry_count: root.nodes_with_entry_count.get(),
nodes_with_copy_source_count: root.nodes_with_copy_source_count.get(), nodes_with_copy_source_count: root.nodes_with_copy_source_count.get(),
ignore_patterns_hash: root.ignore_patterns_hash, ignore_patterns_hash: root.ignore_patterns_hash,
unreachable_bytes,
}; };
Ok(dirstate_map) Ok(dirstate_map)
} }
impl Node { impl Node {
pub(super) fn full_path<'on_disk>( pub(super) fn full_path<'on_disk>(
&self, &self,
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
let root_nodes = writer.write_nodes(dirstate_map.root.as_ref())?; let root_nodes = writer.write_nodes(dirstate_map.root.as_ref())?;
let root = Root { let root = Root {
root_nodes, root_nodes,
nodes_with_entry_count: dirstate_map.nodes_with_entry_count.into(), nodes_with_entry_count: dirstate_map.nodes_with_entry_count.into(),
nodes_with_copy_source_count: dirstate_map nodes_with_copy_source_count: dirstate_map
.nodes_with_copy_source_count .nodes_with_copy_source_count
.into(), .into(),
unreachable_bytes: dirstate_map.unreachable_bytes.into(),
ignore_patterns_hash: dirstate_map.ignore_patterns_hash, ignore_patterns_hash: dirstate_map.ignore_patterns_hash,
}; };
writer.out.extend(root.as_bytes()); writer.out.extend(root.as_bytes());
Ok((writer.out, append)) Ok((writer.out, append))
} }
struct Writer<'dmap, 'on_disk> { struct Writer<'dmap, 'on_disk> {
dirstate_map: &'dmap DirstateMap<'on_disk>, dirstate_map: &'dmap DirstateMap<'on_disk>,