This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11090

dirstate-v2: Rename Header to Root, move it to the end of the data file
ClosedPublic

Authored by SimonSapin on Jul 12 2021, 5:10 PM.

Details

Reviewers
Alphare
marmoute
Group Reviewers
hg-reviewers
Commits
rHG096ee2e260a3: dirstate-v2: Rename Header to Root, move it to the end of the data file
Summary

Now that they don’t have to be at the start, a given data file may contain
multiple "roots". A docket only points to one of them, and previous ones
are left unused to allow allow append-only in-place writing to an existing
data file.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

SimonSapin created this revision.Jul 12 2021, 5:10 PM
baymax updated this revision to Diff 29220.Jul 12 2021, 6:39 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare accepted this revision.Jul 13 2021, 11:10 AM
This revision is now accepted and ready to land.Jul 13 2021, 11:10 AM
marmoute added a subscriber: marmoute.Jul 16 2021, 5:54 AM
marmoute added inline comments.
rust/hg-core/src/dirstate_tree/on_disk.rs
5–15

strange char detected.

marmoute accepted this revision.Jul 16 2021, 5:55 AM
SimonSapin added inline comments.Jul 16 2021, 8:15 AM
rust/hg-core/src/dirstate_tree/on_disk.rs
5–15

I find it sad that Phabricator gets so upset over a perfectly normal non-breaking space in a comment, but having it here was not intentional so I’ll change it.

SimonSapin updated this revision to Diff 29332.Jul 16 2021, 8:37 AM
baymax updated this revision to Diff 29608.Jul 20 2021, 3:41 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Revision Contents
Changeset List

Diff 29619

mercurial/debugcommands.py

[], [],
_(b''), _(b''),
) )
def debugdirstateignorepatternshash(ui, repo, **opts): def debugdirstateignorepatternshash(ui, repo, **opts):
"""show the hash of ignore patterns stored in dirstate if v2, """show the hash of ignore patterns stored in dirstate if v2,
or nothing for dirstate-v2 or nothing for dirstate-v2
""" """
if repo.dirstate._use_dirstate_v2: if repo.dirstate._use_dirstate_v2:
hash_offset = 16 # Four 32-bit integers before this field docket = repo.dirstate._map.docket
hash_len = 20 # 160 bits for SHA-1 hash_len = 20 # 160 bits for SHA-1
data_filename = repo.dirstate._map.docket.data_filename() hash_offset = docket.data_size - hash_len # hash is at the end
data_filename = docket.data_filename()
with repo.vfs(data_filename) as f: with repo.vfs(data_filename) as f:
hash_bytes = f.read(hash_offset + hash_len)[-hash_len:] f.seek(hash_offset)
hash_bytes = f.read(hash_len)
ui.write(binascii.hexlify(hash_bytes) + b'\n') ui.write(binascii.hexlify(hash_bytes) + b'\n')
@command( @command(
b'debugdiscovery', b'debugdiscovery',
[ [
(b'', b'old', None, _(b'use old-style discovery')), (b'', b'old', None, _(b'use old-style discovery')),
( (

rust/hg-core/src/dirstate_tree/on_disk.rs

//! The "version 2" disk representation of the dirstate //! The "version 2" disk representation of the dirstate
//! //!
//! # File format //! # File format
//! //!
//! The file starts with a fixed-sized header, whose layout is defined by the //! In dirstate-v2 format, the `.hg/dirstate` file is a "docket that starts
//! `Header` struct. Its `root` field contains the slice (offset and length) to //! with a fixed-sized header whose layout is defined by the `DocketHeader`
//! struct, followed by the data file identifier.
//!
//! A separate `.hg/dirstate.{uuid}.d` file contains most of the data. That
//! file may be longer than the size given in the docket, but not shorter. Only
//! the start of the data file up to the given size is considered. The
//! fixed-size "root" of the dirstate tree whose layout is defined by the
//! `Root` struct is found at the end of that slice of data.
//!
//! Its `root_nodes` field contains the slice (offset and length) to
marmouteUnsubmitted
Not Done

strange char detected.

marmoute: strange char detected.
SimonSapinAuthorUnsubmitted
Done

I find it sad that Phabricator gets so upset over a perfectly normal non-breaking space in a comment, but having it here was not intentional so I’ll change it.

SimonSapin: I find it sad that Phabricator gets so upset over a perfectly normal non-breaking space in a…
//! the nodes representing the files and directories at the root of the //! the nodes representing the files and directories at the root of the
//! repository. Each node is also fixed-size, defined by the `Node` struct. //! repository. Each node is also fixed-size, defined by the `Node` struct.
//! Nodes in turn contain slices to variable-size paths, and to their own child //! Nodes in turn contain slices to variable-size paths, and to their own child
//! nodes (if any) for nested files and directories. //! nodes (if any) for nested files and directories.
use crate::dirstate_tree::dirstate_map::{self, DirstateMap, NodeRef}; use crate::dirstate_tree::dirstate_map::{self, DirstateMap, NodeRef};
use crate::dirstate_tree::path_with_basename::WithBasename; use crate::dirstate_tree::path_with_basename::WithBasename;
use crate::errors::HgError; use crate::errors::HgError;
pub struct Docket<'on_disk> { pub struct Docket<'on_disk> {
header: &'on_disk DocketHeader, header: &'on_disk DocketHeader,
uuid: &'on_disk [u8], uuid: &'on_disk [u8],
} }
#[derive(BytesCast)] #[derive(BytesCast)]
#[repr(C)] #[repr(C)]
struct Header { struct Root {
root: ChildNodes, root_nodes: ChildNodes,
nodes_with_entry_count: Size, nodes_with_entry_count: Size,
nodes_with_copy_source_count: Size, nodes_with_copy_source_count: Size,
/// If non-zero, a hash of ignore files that were used for some previous /// If non-zero, a hash of ignore files that were used for some previous
/// run of the `status` algorithm. /// run of the `status` algorithm.
/// ///
/// We define: /// We define:
/// ///
/// - That timestamp was already strictly in the past when observed, /// - That timestamp was already strictly in the past when observed,
/// meaning that later changes cannot happen in the same clock tick /// meaning that later changes cannot happen in the same clock tick
/// and must cause a different modification time (unless the system /// and must cause a different modification time (unless the system
/// clock jumps back and we get unlucky, which is not impossible but /// clock jumps back and we get unlucky, which is not impossible but
/// but deemed unlikely enough). /// but deemed unlikely enough).
/// - All direct children of this directory (as returned by /// - All direct children of this directory (as returned by
/// `std::fs::read_dir`) either have a corresponding dirstate node, or /// `std::fs::read_dir`) either have a corresponding dirstate node, or
/// are ignored by ignore patterns whose hash is in /// are ignored by ignore patterns whose hash is in
/// `Header::ignore_patterns_hash`. /// `Root::ignore_patterns_hash`.
/// ///
/// This means that if `std::fs::symlink_metadata` later reports the /// This means that if `std::fs::symlink_metadata` later reports the
/// same modification time and ignored patterns haven’t changed, a run /// same modification time and ignored patterns haven’t changed, a run
/// of status that is not listing ignored files can skip calling /// of status that is not listing ignored files can skip calling
/// `std::fs::read_dir` again for this directory, iterate child /// `std::fs::read_dir` again for this directory, iterate child
/// dirstate nodes instead. /// dirstate nodes instead.
state: u8, state: u8,
data: Entry, data: Entry,
type PathSlice = Slice; type PathSlice = Slice;
/// Either nothing if `start == 0`, or a `HgPath` of `len` bytes /// Either nothing if `start == 0`, or a `HgPath` of `len` bytes
type OptPathSlice = Slice; type OptPathSlice = Slice;
/// Make sure that size-affecting changes are made knowingly /// Make sure that size-affecting changes are made knowingly
fn _static_assert_size_of() { fn _static_assert_size_of() {
let _ = std::mem::transmute::<DocketHeader, [u8; 81]>; let _ = std::mem::transmute::<DocketHeader, [u8; 81]>;
let _ = std::mem::transmute::<Header, [u8; 36]>; let _ = std::mem::transmute::<Root, [u8; 36]>;
let _ = std::mem::transmute::<Node, [u8; 49]>; let _ = std::mem::transmute::<Node, [u8; 49]>;
} }
/// Unexpected file format found in `.hg/dirstate` with the "v2" format. /// Unexpected file format found in `.hg/dirstate` with the "v2" format.
/// ///
/// This should only happen if Mercurial is buggy or a repository is corrupted. /// This should only happen if Mercurial is buggy or a repository is corrupted.
#[derive(Debug)] #[derive(Debug)]
pub struct DirstateV2ParseError; pub struct DirstateV2ParseError;
let uuid_size = header.uuid_size as usize; let uuid_size = header.uuid_size as usize;
if header.marker == *V2_FORMAT_MARKER && uuid.len() == uuid_size { if header.marker == *V2_FORMAT_MARKER && uuid.len() == uuid_size {
Ok(Docket { header, uuid }) Ok(Docket { header, uuid })
} else { } else {
Err(DirstateV2ParseError) Err(DirstateV2ParseError)
} }
} }
fn read_root<'on_disk>(
on_disk: &'on_disk [u8],
) -> Result<&'on_disk Root, DirstateV2ParseError> {
// Find the `Root` at the end of the given slice
let root_offset = on_disk
.len()
.checked_sub(std::mem::size_of::<Root>())
// A non-empty slice too short is an error
.ok_or(DirstateV2ParseError)?;
let (root, _) = Root::from_bytes(&on_disk[root_offset..])
.map_err(|_| DirstateV2ParseError)?;
Ok(root)
}
pub(super) fn read<'on_disk>( pub(super) fn read<'on_disk>(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
) -> Result<DirstateMap<'on_disk>, DirstateV2ParseError> { ) -> Result<DirstateMap<'on_disk>, DirstateV2ParseError> {
if on_disk.is_empty() { if on_disk.is_empty() {
return Ok(DirstateMap::empty(on_disk)); return Ok(DirstateMap::empty(on_disk));
} }
let (header, _) = let root = read_root(on_disk)?;
Header::from_bytes(on_disk).map_err(|_| DirstateV2ParseError)?;
let dirstate_map = DirstateMap { let dirstate_map = DirstateMap {
on_disk, on_disk,
root: dirstate_map::ChildNodes::OnDisk(read_slice::<Node>( root: dirstate_map::ChildNodes::OnDisk(read_slice::<Node>(
on_disk, on_disk,
header.root, root.root_nodes,
)?), )?),
nodes_with_entry_count: header.nodes_with_entry_count.get(), nodes_with_entry_count: root.nodes_with_entry_count.get(),
nodes_with_copy_source_count: header nodes_with_copy_source_count: root.nodes_with_copy_source_count.get(),
.nodes_with_copy_source_count ignore_patterns_hash: root.ignore_patterns_hash,
.get(),
ignore_patterns_hash: header.ignore_patterns_hash,
}; };
Ok(dirstate_map) Ok(dirstate_map)
} }
impl Node { impl Node {
pub(super) fn full_path<'on_disk>( pub(super) fn full_path<'on_disk>(
&self, &self,
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
.map(|(slice, _rest)| slice) .map(|(slice, _rest)| slice)
.ok_or_else(|| DirstateV2ParseError) .ok_or_else(|| DirstateV2ParseError)
} }
pub(crate) fn for_each_tracked_path<'on_disk>( pub(crate) fn for_each_tracked_path<'on_disk>(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
mut f: impl FnMut(&'on_disk HgPath), mut f: impl FnMut(&'on_disk HgPath),
) -> Result<(), DirstateV2ParseError> { ) -> Result<(), DirstateV2ParseError> {
let (header, _) = let root = read_root(on_disk)?;
Header::from_bytes(on_disk).map_err(|_| DirstateV2ParseError)?;
fn recur<'on_disk>( fn recur<'on_disk>(
on_disk: &'on_disk [u8], on_disk: &'on_disk [u8],
nodes: Slice, nodes: Slice,
f: &mut impl FnMut(&'on_disk HgPath), f: &mut impl FnMut(&'on_disk HgPath),
) -> Result<(), DirstateV2ParseError> { ) -> Result<(), DirstateV2ParseError> {
for node in read_slice::<Node>(on_disk, nodes)? { for node in read_slice::<Node>(on_disk, nodes)? {
if let Some(state) = node.state()? { if let Some(state) = node.state()? {
if state.is_tracked() { if state.is_tracked() {
f(node.full_path(on_disk)?) f(node.full_path(on_disk)?)
} }
} }
recur(on_disk, node.children, f)? recur(on_disk, node.children, f)?
} }
Ok(()) Ok(())
} }
recur(on_disk, header.root, &mut f) recur(on_disk, root.root_nodes, &mut f)
} }
pub(super) fn write( pub(super) fn write(
dirstate_map: &mut DirstateMap, dirstate_map: &mut DirstateMap,
) -> Result<Vec<u8>, DirstateError> { ) -> Result<Vec<u8>, DirstateError> {
let header_len = std::mem::size_of::<Header>(); let root_len = std::mem::size_of::<Root>();
// This ignores the space for paths, and for nodes without an entry. // This ignores the space for paths, and for nodes without an entry.
// TODO: better estimate? Skip the `Vec` and write to a file directly? // TODO: better estimate? Skip the `Vec` and write to a file directly?
let size_guess = header_len let size_guess = root_len
+ std::mem::size_of::<Node>() + std::mem::size_of::<Node>()
* dirstate_map.nodes_with_entry_count as usize; * dirstate_map.nodes_with_entry_count as usize;
let mut out = Vec::with_capacity(size_guess); let mut out = Vec::with_capacity(size_guess);
// Keep space for the header. We’ll fill it out at the end when we know the let root_nodes =
// actual offset for the root nodes.
out.resize(header_len, 0_u8);
let root =
write_nodes(dirstate_map, dirstate_map.root.as_ref(), &mut out)?; write_nodes(dirstate_map, dirstate_map.root.as_ref(), &mut out)?;
let header = Header { let root = Root {
root, root_nodes,
nodes_with_entry_count: dirstate_map.nodes_with_entry_count.into(), nodes_with_entry_count: dirstate_map.nodes_with_entry_count.into(),
nodes_with_copy_source_count: dirstate_map nodes_with_copy_source_count: dirstate_map
.nodes_with_copy_source_count .nodes_with_copy_source_count
.into(), .into(),
ignore_patterns_hash: dirstate_map.ignore_patterns_hash, ignore_patterns_hash: dirstate_map.ignore_patterns_hash,
}; };
out[..header_len].copy_from_slice(header.as_bytes()); out.extend(root.as_bytes());
Ok(out) Ok(out)
} }
fn write_nodes( fn write_nodes(
dirstate_map: &DirstateMap, dirstate_map: &DirstateMap,
nodes: dirstate_map::ChildNodesRef, nodes: dirstate_map::ChildNodesRef,
out: &mut Vec<u8>, out: &mut Vec<u8>,
) -> Result<ChildNodes, DirstateError> { ) -> Result<ChildNodes, DirstateError> {