This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10486

dirstate-tree: Serialize to disk
ClosedPublic

Authored by SimonSapin on Apr 20 2021, 10:17 AM.

Download Raw Diff

Details

Reviewers

Alphare

Group Reviewers

hg-reviewers

Commits

rHGd6c94ca40863: dirstate-tree: Serialize to disk
rHG1ae8c997b5ab: dirstate-tree: Serialize to disk

Summary

The existing pack_dirstate function relies on implementation details
of DirstateMap, so extract some parts of it as separate functions
for us in the tree-based DirstateMap.

The bytes-cast crate is updated to a version that has an as_bytes method,
not just from_bytes:
https://docs.rs/bytes-cast/0.2.0/bytes_cast/trait.BytesCast.html#method.as_bytes

Drive-by refactor clear_ambiguous_times which does part of the same thing.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

SimonSapin created this revision.Apr 20 2021, 10:17 AM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptApr 20 2021, 10:17 AM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

SimonSapin added a child revision: D10487: dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources.Apr 20 2021, 10:17 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare accepted this revision.Apr 22 2021, 4:43 AM

Alphare added a subscriber: Alphare.

Alphare added inline comments.

rust/hg-core/Cargo.toml
12	I know you're the author of that crate (and that this is because `0.2` adds supports for casting as bytes), but a general good practice when updating dependencies is to explain why you've done so in the commit message.

This revision is now accepted and ready to land.Apr 22 2021, 4:43 AM

SimonSapin marked an inline comment as done.May 3 2021, 5:55 AM

SimonSapin added inline comments.

rust/hg-core/Cargo.toml
12	Good point, I’ll amend the message.

SimonSapin edited the summary of this revision. (Show Details)May 3 2021, 6:24 AM

SimonSapin marked an inline comment as done.May 6 2021, 12:51 PM

SimonSapin added a commit: rHG1ae8c997b5ab: dirstate-tree: Serialize to disk.

Closed by commit rHG1ae8c997b5ab: dirstate-tree: Serialize to disk (authored by SimonSapin). · Explain Why

This revision was automatically updated to reflect the committed changes.

SimonSapin added a commit: rHGd6c94ca40863: dirstate-tree: Serialize to disk.May 6 2021, 3:42 PM

Revision Contents
Changeset List

			Path	Packages
M			rust/Cargo.lock (4 lines)
M			rust/hg-core/Cargo.toml (2 lines)
M			rust/hg-core/src/dirstate/parsers.rs (105 lines)
M			rust/hg-core/src/dirstate_tree/dirstate_map.rs (42 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	27106		Apr 20 2021, 10:17 AM	★	★
Diff 2	27136		Apr 20 2021, 1:03 PM	★	★
Diff 3	27645	rHG1ae8c997b5ab6c0eefebaaf0f7a36bdeb7ac737d	Apr 12 2021, 8:21 AM	★	★

Status	Author	Revision
Closed	SimonSapin	D10560 dirstate-tree: Borrow paths from the "on disk" bytes
Closed	SimonSapin	D10559 dirstate-tree: Borrow copy source paths from the "on disk" bytes
Closed	SimonSapin	D10558 rust: Use `&HgPath` instead of `&HgPathBuf` in may APIs
Closed	SimonSapin	D10557 dirstate-tree: Make `DirstateMap` borrow from a bytes buffer
Closed	SimonSapin	D10556 rust: Read dirstate from disk in DirstateMap constructor
Closed	SimonSapin	D10555 rust: Remove handling of `parents` in `DirstateMap`
Closed	SimonSapin	D10554 dirstate-tree: Fold "tracked descendants" counter update in main walk
Closed	SimonSapin	D10553 dirstate-tree: Use HashMap instead of BTreeMap
Closed	SimonSapin	D10552 dirstate-tree: Add #[timed] attribute to `status` and `DirstateMap::read`
Closed	SimonSapin	D10551 dirstate-tree: Paralellize the status algorithm with Rayon
Closed	SimonSapin	D10550 dirstate-tree: Avoid BTreeMap double-lookup when inserting a dirstate entry
Closed	SimonSapin	D10549 dirstate-tree: Handle I/O errors in status
Closed	SimonSapin	D10548 dirstate-tree: Ignore FIFOs etc. in the status algorithm
Closed	SimonSapin	D10547 dirstate-tree: Add the new `status()` algorithm
Closed	SimonSapin	D10546 dirstate-tree: Give to `status()` mutable access to the `DirstateMap`
Closed	SimonSapin	D10495 rust: Add doc-comments to DirstateStatus fields
Closed	SimonSapin	D10494 rust: Move "lookup" a.k.a. "unsure" paths into `DirstateStatus` struct
Closed	SimonSapin	D10493 rust: Remove DirstateMap::file_fold_map
Closed	SimonSapin	D10492 dirstate-tree: Add "non normal" and "from other parent" sets
Closed	SimonSapin	D10491 dirstate-tree: Add add_file, remove_file, and drop_file
Closed	SimonSapin	D10490 dirstate-tree: Add has_dir and has_tracked_dir
Closed	SimonSapin	D10489 dirstate-tree: Add clear_ambiguous_times in the new DirstateMap
Closed	SimonSapin	D10488 dirstate-tree: Add copy_map_insert and copy_map_remove
Closed	SimonSapin	D10487 dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources
Closed	SimonSapin	D10486 dirstate-tree: Serialize to disk
Closed	SimonSapin	D10485 rust: Add a Timestamp struct instead of abusing Duration
Closed	SimonSapin	D10370 dirstate-tree: Add tree traversal/iteration
Closed	SimonSapin	D10369 dirstate-tree: Add map `get` and `contains_key` methods
Closed	SimonSapin	D10368 dirstate-tree: Add parsing only dirstate parents from disk
Closed	SimonSapin	D10367 dirstate-tree: Implement DirstateMap::read
Abandoned	SimonSapin	D10366 rust: Refactor parse_dirstate() to take a callback
Closed	SimonSapin	D10365 dirstate-tree: Add `WithBasename` wrapper for `HgPath`
Closed	SimonSapin	D10364 dirstate-tree: Empty shell for a second Rust DirstateMap implementation
Closed	SimonSapin	D10363 dirstate-tree: Abstract "non-normal" and "other parent" sets
Closed	SimonSapin	D10362 dirstate-tree: Make Rust DirstateMap bindings go through a trait object

Diff 27645

rust/Cargo.lock

	[[package]]			[[package]]
	name = "byteorder"			name = "byteorder"
	version = "1.3.4"			version = "1.3.4"
	source = "registry+https://github.com/rust-lang/crates.io-index"			source = "registry+https://github.com/rust-lang/crates.io-index"
	checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"			checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"

	[[package]]			[[package]]
	name = "bytes-cast"			name = "bytes-cast"
	version = "0.1.0"			version = "0.2.0"
	source = "registry+https://github.com/rust-lang/crates.io-index"			source = "registry+https://github.com/rust-lang/crates.io-index"
	checksum = "3196ba300c7bc9282a4331e878496cb3e9603a898a8f1446601317163e16ca52"			checksum = "0d434f9a4ecbe987e7ccfda7274b6f82ea52c9b63742565a65cb5e8ba0f2c452"
	dependencies = [			dependencies = [
	"bytes-cast-derive",			"bytes-cast-derive",
	]			]

	[[package]]			[[package]]
	name = "bytes-cast-derive"			name = "bytes-cast-derive"
	version = "0.1.0"			version = "0.1.0"
	source = "registry+https://github.com/rust-lang/crates.io-index"			source = "registry+https://github.com/rust-lang/crates.io-index"

rust/hg-core/Cargo.toml

	[package]			[package]
	name = "hg-core"			name = "hg-core"
	version = "0.1.0"			version = "0.1.0"
	authors = ["Georges Racinet <gracinet@anybox.fr>"]			authors = ["Georges Racinet <gracinet@anybox.fr>"]
	description = "Mercurial pure Rust core library, with no assumption on Python bindings (FFI)"			description = "Mercurial pure Rust core library, with no assumption on Python bindings (FFI)"
	edition = "2018"			edition = "2018"

	[lib]			[lib]
	name = "hg"			name = "hg"

	[dependencies]			[dependencies]
	bytes-cast = "0.1"			bytes-cast = "0.2"
				AlphareUnsubmitted Done I know you're the author of that crate (and that this is because `0.2` adds supports for casting as bytes), but a general good practice when updating dependencies is to explain why you've done so in the commit message. Alphare: I know you're the author of that crate (and that this is because `0.2` adds supports for…
				SimonSapinAuthorUnsubmitted Done Good point, I’ll amend the message. SimonSapin: Good point, I’ll amend the message.
	byteorder = "1.3.4"			byteorder = "1.3.4"
	derive_more = "0.99"			derive_more = "0.99"
	home = "0.5"			home = "0.5"
	im-rc = "15.0.*"			im-rc = "15.0.*"
	lazy_static = "1.4.0"			lazy_static = "1.4.0"
	rand = "0.7.3"			rand = "0.7.3"
	rand_pcg = "0.2.1"			rand_pcg = "0.2.1"
	rand_distr = "0.2.2"			rand_distr = "0.2.2"

rust/hg-core/src/dirstate/parsers.rs

	// Copyright 2019 Raphaël Gomès <rgomes@octobus.net>			// Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
	//			//
	// This software may be used and distributed according to the terms of the			// This software may be used and distributed according to the terms of the
	// GNU General Public License version 2 or any later version.			// GNU General Public License version 2 or any later version.

	use crate::errors::HgError;			use crate::errors::HgError;
	use crate::utils::hg_path::HgPath;			use crate::utils::hg_path::{HgPath, HgPathBuf};
	use crate::{			use crate::{
	dirstate::{CopyMap, EntryState, RawEntry, StateMap},			dirstate::{CopyMap, EntryState, RawEntry, StateMap},
	DirstateEntry, DirstateParents,			DirstateEntry, DirstateParents,
	};			};
	use byteorder::{BigEndian, WriteBytesExt};			use byteorder::{BigEndian, WriteBytesExt};
	use bytes_cast::BytesCast;			use bytes_cast::BytesCast;
	use micro_timer::timed;			use micro_timer::timed;
	use std::convert::{TryFrom, TryInto};			use std::convert::{TryFrom, TryInto};
	let copy_source = iter.next().map(HgPath::new);			let copy_source = iter.next().map(HgPath::new);
	each_entry(path, &entry, copy_source);			each_entry(path, &entry, copy_source);

	contents = rest;			contents = rest;
	}			}
	Ok(parents)			Ok(parents)
	}			}

				fn packed_filename_and_copy_source_size(
				filename: &HgPathBuf,
				copy_source: Option<&HgPathBuf>,
				) -> usize {
				filename.len()
				+ if let Some(source) = copy_source {
				b"\0".len() + source.len()
				} else {
				0
				}
				}

				pub fn packed_entry_size(
				filename: &HgPathBuf,
				copy_source: Option<&HgPathBuf>,
				) -> usize {
				MIN_ENTRY_SIZE
				+ packed_filename_and_copy_source_size(filename, copy_source)
				}

				pub fn pack_entry(
				filename: &HgPathBuf,
				entry: &DirstateEntry,
				copy_source: Option<&HgPathBuf>,
				packed: &mut Vec<u8>,
				) {
				let length = packed_filename_and_copy_source_size(filename, copy_source);

				// Unwrapping because `impl std::io::Write for Vec<u8>` never errors
				packed.write_u8(entry.state.into()).unwrap();
				packed.write_i32::<BigEndian>(entry.mode).unwrap();
				packed.write_i32::<BigEndian>(entry.size).unwrap();
				packed.write_i32::<BigEndian>(entry.mtime).unwrap();
				packed.write_i32::<BigEndian>(length as i32).unwrap();
				packed.extend(filename.as_bytes());
				if let Some(source) = copy_source {
				packed.push(b'\0');
				packed.extend(source.as_bytes());
				}
				}

	/// Seconds since the Unix epoch			/// Seconds since the Unix epoch
	pub struct Timestamp(pub u64);			pub struct Timestamp(pub u64);

				pub fn clear_ambiguous_mtime(
				entry: &mut DirstateEntry,
				mtime_now: i32,
				) -> bool {
				let ambiguous =
				entry.state == EntryState::Normal && entry.mtime == mtime_now;
				if ambiguous {
				// The file was last modified "simultaneously" with the current
				// write to dirstate (i.e. within the same second for file-
				// systems with a granularity of 1 sec). This commonly happens
				// for at least a couple of files on 'update'.
				// The user could change the file without changing its size
				// within the same second. Invalidate the file's mtime in
				// dirstate, forcing future 'status' calls to compare the
				// contents of the file if the size is the same. This prevents
				// mistakenly treating such files as clean.
				entry.mtime = -1;
				}
				ambiguous
				}

	pub fn pack_dirstate(			pub fn pack_dirstate(
	state_map: &mut StateMap,			state_map: &mut StateMap,
	copy_map: &CopyMap,			copy_map: &CopyMap,
	parents: DirstateParents,			parents: DirstateParents,
	now: Timestamp,			now: Timestamp,
	) -> Result<Vec<u8>, HgError> {			) -> Result<Vec<u8>, HgError> {
	// TODO move away from i32 before 2038.			// TODO move away from i32 before 2038.
	let now: i32 = now.0.try_into().expect("time overflow");			let now: i32 = now.0.try_into().expect("time overflow");

	let expected_size: usize = state_map			let expected_size: usize = state_map
	.iter()			.iter()
	.map(\|(filename, _)\| {			.map(\|(filename, _)\| {
	let mut length = MIN_ENTRY_SIZE + filename.len();			packed_entry_size(filename, copy_map.get(filename))
	if let Some(copy) = copy_map.get(filename) {
	length += copy.len() + 1;
	}
	length
	})			})
	.sum();			.sum();
	let expected_size = expected_size + PARENT_SIZE * 2;			let expected_size = expected_size + PARENT_SIZE * 2;

	let mut packed = Vec::with_capacity(expected_size);			let mut packed = Vec::with_capacity(expected_size);

	packed.extend(parents.p1.as_bytes());			packed.extend(parents.p1.as_bytes());
	packed.extend(parents.p2.as_bytes());			packed.extend(parents.p2.as_bytes());

	for (filename, entry) in state_map.iter_mut() {			for (filename, entry) in state_map.iter_mut() {
	let new_filename = filename.to_owned();			clear_ambiguous_mtime(entry, now);
	let mut new_mtime: i32 = entry.mtime;			pack_entry(filename, entry, copy_map.get(filename), &mut packed)
	if entry.state == EntryState::Normal && entry.mtime == now {
	// The file was last modified "simultaneously" with the current
	// write to dirstate (i.e. within the same second for file-
	// systems with a granularity of 1 sec). This commonly happens
	// for at least a couple of files on 'update'.
	// The user could change the file without changing its size
	// within the same second. Invalidate the file's mtime in
	// dirstate, forcing future 'status' calls to compare the
	// contents of the file if the size is the same. This prevents
	// mistakenly treating such files as clean.
	new_mtime = -1;
	*entry = DirstateEntry {
	mtime: new_mtime,
	..*entry
	};
	}
	let mut new_filename = new_filename.into_vec();
	if let Some(copy) = copy_map.get(filename) {
	new_filename.push(b'\0');
	new_filename.extend(copy.bytes());
	}

	// Unwrapping because `impl std::io::Write for Vec<u8>` never errors
	packed.write_u8(entry.state.into()).unwrap();
	packed.write_i32::<BigEndian>(entry.mode).unwrap();
	packed.write_i32::<BigEndian>(entry.size).unwrap();
	packed.write_i32::<BigEndian>(new_mtime).unwrap();
	packed
	.write_i32::<BigEndian>(new_filename.len() as i32)
	.unwrap();
	packed.extend(new_filename)
	}			}

	if packed.len() != expected_size {			if packed.len() != expected_size {
	return Err(HgError::CorruptedRepository(format!(			return Err(HgError::CorruptedRepository(format!(
	"bad dirstate size: {} != {}",			"bad dirstate size: {} != {}",
	expected_size,			expected_size,
	packed.len()			packed.len()
	)));			)));

rust/hg-core/src/dirstate_tree/dirstate_map.rs

	use std::collections::BTreeMap;			use bytes_cast::BytesCast;
	use std::path::PathBuf;			use std::path::PathBuf;
				use std::{collections::BTreeMap, convert::TryInto};

	use super::path_with_basename::WithBasename;			use super::path_with_basename::WithBasename;
				use crate::dirstate::parsers::clear_ambiguous_mtime;
				use crate::dirstate::parsers::pack_entry;
				use crate::dirstate::parsers::packed_entry_size;
	use crate::dirstate::parsers::parse_dirstate_entries;			use crate::dirstate::parsers::parse_dirstate_entries;
	use crate::dirstate::parsers::parse_dirstate_parents;			use crate::dirstate::parsers::parse_dirstate_parents;
	use crate::dirstate::parsers::Timestamp;			use crate::dirstate::parsers::Timestamp;

	use crate::matchers::Matcher;			use crate::matchers::Matcher;
	use crate::revlog::node::NULL_NODE;			use crate::revlog::node::NULL_NODE;
	use crate::utils::hg_path::{HgPath, HgPathBuf};			use crate::utils::hg_path::{HgPath, HgPathBuf};
	use crate::CopyMapIter;			use crate::CopyMapIter;
	use crate::DirstateEntry;			use crate::DirstateEntry;
	use crate::DirstateError;			use crate::DirstateError;
	use crate::DirstateMapError;			use crate::DirstateMapError;
	use crate::DirstateParents;			use crate::DirstateParents;
	self.set_parents(parents);			self.set_parents(parents);
	}			}

	Ok(Some(parents))			Ok(Some(parents))
	}			}

	fn pack(			fn pack(
	&mut self,			&mut self,
	_parents: DirstateParents,			parents: DirstateParents,
	_now: Timestamp,			now: Timestamp,
	) -> Result<Vec<u8>, DirstateError> {			) -> Result<Vec<u8>, DirstateError> {
	let _ = self.iter_node_data_mut();			// Optizimation (to be measured?): pre-compute size to avoid `Vec`
	todo!()			// reallocations
				let mut size = parents.as_bytes().len();
				for (path, node) in self.iter_nodes() {
				if node.entry.is_some() {
				size += packed_entry_size(
				path.full_path(),
				node.copy_source.as_ref(),
				)
				}
				}

				let mut packed = Vec::with_capacity(size);
				packed.extend(parents.as_bytes());

				let now: i32 = now.0.try_into().expect("time overflow");
				for (path, opt_entry, copy_source) in self.iter_node_data_mut() {
				if let Some(entry) = opt_entry {
				clear_ambiguous_mtime(entry, now);
				pack_entry(
				path.full_path(),
				entry,
				copy_source.as_ref(),
				&mut packed,
				);
				}
				}
				self.dirty_parents = false;
				Ok(packed)
	}			}

	fn build_file_fold_map(&mut self) -> &FastHashMap<HgPathBuf, HgPathBuf> {			fn build_file_fold_map(&mut self) -> &FastHashMap<HgPathBuf, HgPathBuf> {
	todo!()			todo!()
	}			}

	fn set_all_dirs(&mut self) -> Result<(), DirstateMapError> {			fn set_all_dirs(&mut self) -> Result<(), DirstateMapError> {
	todo!()			todo!()