This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10486

dirstate-tree: Serialize to disk
ClosedPublic

Authored by SimonSapin on Apr 20 2021, 10:17 AM.

Download Raw Diff

Details

Reviewers

Alphare

Group Reviewers

hg-reviewers

Commits

rHGd6c94ca40863: dirstate-tree: Serialize to disk
rHG1ae8c997b5ab: dirstate-tree: Serialize to disk

Summary

The existing pack_dirstate function relies on implementation details
of DirstateMap, so extract some parts of it as separate functions
for us in the tree-based DirstateMap.

The bytes-cast crate is updated to a version that has an as_bytes method,
not just from_bytes:
https://docs.rs/bytes-cast/0.2.0/bytes_cast/trait.BytesCast.html#method.as_bytes

Drive-by refactor clear_ambiguous_times which does part of the same thing.

Diff Detail

Repository

rHG Mercurial

Branch

default

Lint

No Linters Available

Unit

No Unit Test Coverage

Event Timeline

SimonSapin created this revision.Apr 20 2021, 10:17 AM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptApr 20 2021, 10:17 AM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

SimonSapin added a child revision: D10487: dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources.Apr 20 2021, 10:17 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare accepted this revision.Apr 22 2021, 4:43 AM

Alphare added a subscriber: Alphare.

Alphare added inline comments.

rust/hg-core/Cargo.toml
12	I know you're the author of that crate (and that this is because `0.2` adds supports for casting as bytes), but a general good practice when updating dependencies is to explain why you've done so in the commit message.

This revision is now accepted and ready to land.Apr 22 2021, 4:43 AM

SimonSapin marked an inline comment as done.May 3 2021, 5:55 AM

SimonSapin added inline comments.

rust/hg-core/Cargo.toml
12	Good point, I’ll amend the message.

SimonSapin edited the summary of this revision. (Show Details)May 3 2021, 6:24 AM

SimonSapin marked an inline comment as done.May 6 2021, 12:51 PM

SimonSapin added a commit: rHG1ae8c997b5ab: dirstate-tree: Serialize to disk.

Closed by commit rHG1ae8c997b5ab: dirstate-tree: Serialize to disk (authored by SimonSapin). · Explain Why

This revision was automatically updated to reflect the committed changes.

SimonSapin added a commit: rHGd6c94ca40863: dirstate-tree: Serialize to disk.May 6 2021, 3:42 PM

Revision Contents
Changeset List

			Path	Packages
M			rust/Cargo.lock (4 lines)
M			rust/hg-core/Cargo.toml (2 lines)
M			rust/hg-core/src/dirstate/parsers.rs (105 lines)
M			rust/hg-core/src/dirstate_tree/dirstate_map.rs (42 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	27106		Apr 20 2021, 10:17 AM	★	★
Diff 2	27136		Apr 20 2021, 1:03 PM	★	★
Diff 3	27645	rHG1ae8c997b5ab6c0eefebaaf0f7a36bdeb7ac737d	Apr 12 2021, 8:21 AM	★	★

Commit	Parents	Author	Summary	Date
19909f9e1e64	c8adee8e8c71	Simon Sapin		Apr 12 2021, 8:21 AM

Status	Author	Revision
Closed	SimonSapin	D10560 dirstate-tree: Borrow paths from the "on disk" bytes
Closed	SimonSapin	D10559 dirstate-tree: Borrow copy source paths from the "on disk" bytes
Closed	SimonSapin	D10558 rust: Use `&HgPath` instead of `&HgPathBuf` in may APIs
Closed	SimonSapin	D10557 dirstate-tree: Make `DirstateMap` borrow from a bytes buffer
Closed	SimonSapin	D10556 rust: Read dirstate from disk in DirstateMap constructor
Closed	SimonSapin	D10555 rust: Remove handling of `parents` in `DirstateMap`
Closed	SimonSapin	D10554 dirstate-tree: Fold "tracked descendants" counter update in main walk
Closed	SimonSapin	D10553 dirstate-tree: Use HashMap instead of BTreeMap
Closed	SimonSapin	D10552 dirstate-tree: Add #[timed] attribute to `status` and `DirstateMap::read`
Closed	SimonSapin	D10551 dirstate-tree: Paralellize the status algorithm with Rayon
Closed	SimonSapin	D10550 dirstate-tree: Avoid BTreeMap double-lookup when inserting a dirstate entry
Closed	SimonSapin	D10549 dirstate-tree: Handle I/O errors in status
Closed	SimonSapin	D10548 dirstate-tree: Ignore FIFOs etc. in the status algorithm
Closed	SimonSapin	D10547 dirstate-tree: Add the new `status()` algorithm
Closed	SimonSapin	D10546 dirstate-tree: Give to `status()` mutable access to the `DirstateMap`
Closed	SimonSapin	D10495 rust: Add doc-comments to DirstateStatus fields
Closed	SimonSapin	D10494 rust: Move "lookup" a.k.a. "unsure" paths into `DirstateStatus` struct
Closed	SimonSapin	D10493 rust: Remove DirstateMap::file_fold_map
Closed	SimonSapin	D10492 dirstate-tree: Add "non normal" and "from other parent" sets
Closed	SimonSapin	D10491 dirstate-tree: Add add_file, remove_file, and drop_file
Closed	SimonSapin	D10490 dirstate-tree: Add has_dir and has_tracked_dir
Closed	SimonSapin	D10489 dirstate-tree: Add clear_ambiguous_times in the new DirstateMap
Closed	SimonSapin	D10488 dirstate-tree: Add copy_map_insert and copy_map_remove
Closed	SimonSapin	D10487 dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources
Closed	SimonSapin	D10486 dirstate-tree: Serialize to disk
Closed	SimonSapin	D10485 rust: Add a Timestamp struct instead of abusing Duration
Closed	SimonSapin	D10370 dirstate-tree: Add tree traversal/iteration
Closed	SimonSapin	D10369 dirstate-tree: Add map `get` and `contains_key` methods
Closed	SimonSapin	D10368 dirstate-tree: Add parsing only dirstate parents from disk
Closed	SimonSapin	D10367 dirstate-tree: Implement DirstateMap::read
Abandoned	SimonSapin	D10366 rust: Refactor parse_dirstate() to take a callback
Closed	SimonSapin	D10365 dirstate-tree: Add `WithBasename` wrapper for `HgPath`
Closed	SimonSapin	D10364 dirstate-tree: Empty shell for a second Rust DirstateMap implementation
Closed	SimonSapin	D10363 dirstate-tree: Abstract "non-normal" and "other parent" sets
Closed	SimonSapin	D10362 dirstate-tree: Make Rust DirstateMap bindings go through a trait object

Diff 27136

rust/Cargo.lock

	[[package]]			[[package]]
	name = "byteorder"			name = "byteorder"
	version = "1.3.4"			version = "1.3.4"
	source = "registry+https://github.com/rust-lang/crates.io-index"			source = "registry+https://github.com/rust-lang/crates.io-index"
	checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"			checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"

	[[package]]			[[package]]
	name = "bytes-cast"			name = "bytes-cast"
	version = "0.1.0"			version = "0.2.0"
	source = "registry+https://github.com/rust-lang/crates.io-index"			source = "registry+https://github.com/rust-lang/crates.io-index"
	checksum = "3196ba300c7bc9282a4331e878496cb3e9603a898a8f1446601317163e16ca52"			checksum = "0d434f9a4ecbe987e7ccfda7274b6f82ea52c9b63742565a65cb5e8ba0f2c452"
	dependencies = [			dependencies = [
	"bytes-cast-derive",			"bytes-cast-derive",
	]			]

	[[package]]			[[package]]
	name = "bytes-cast-derive"			name = "bytes-cast-derive"
	version = "0.1.0"			version = "0.1.0"
	source = "registry+https://github.com/rust-lang/crates.io-index"			source = "registry+https://github.com/rust-lang/crates.io-index"

rust/hg-core/Cargo.toml

	[package]			[package]
	name = "hg-core"			name = "hg-core"
	version = "0.1.0"			version = "0.1.0"
	authors = ["Georges Racinet <gracinet@anybox.fr>"]			authors = ["Georges Racinet <gracinet@anybox.fr>"]
	description = "Mercurial pure Rust core library, with no assumption on Python bindings (FFI)"			description = "Mercurial pure Rust core library, with no assumption on Python bindings (FFI)"
	edition = "2018"			edition = "2018"

	[lib]			[lib]
	name = "hg"			name = "hg"

	[dependencies]			[dependencies]
	bytes-cast = "0.1"			bytes-cast = "0.2"
				AlphareUnsubmitted Done I know you're the author of that crate (and that this is because `0.2` adds supports for casting as bytes), but a general good practice when updating dependencies is to explain why you've done so in the commit message. Alphare: I know you're the author of that crate (and that this is because `0.2` adds supports for…
				SimonSapinAuthorUnsubmitted Done Good point, I’ll amend the message. SimonSapin: Good point, I’ll amend the message.
	byteorder = "1.3.4"			byteorder = "1.3.4"
	derive_more = "0.99"			derive_more = "0.99"
	home = "0.5"			home = "0.5"
	im-rc = "15.0.*"			im-rc = "15.0.*"
	lazy_static = "1.4.0"			lazy_static = "1.4.0"
	rand = "0.7.3"			rand = "0.7.3"
	rand_pcg = "0.2.1"			rand_pcg = "0.2.1"
	rand_distr = "0.2.2"			rand_distr = "0.2.2"

rust/hg-core/src/dirstate/parsers.rs

	// Copyright 2019 Raphaël Gomès <rgomes@octobus.net>			// Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
	//			//
	// This software may be used and distributed according to the terms of the			// This software may be used and distributed according to the terms of the
	// GNU General Public License version 2 or any later version.			// GNU General Public License version 2 or any later version.

	use crate::errors::HgError;			use crate::errors::HgError;
	use crate::utils::hg_path::HgPath;			use crate::utils::hg_path::{HgPath, HgPathBuf};
	use crate::{			use crate::{
	dirstate::{CopyMap, EntryState, RawEntry, StateMap},			dirstate::{CopyMap, EntryState, RawEntry, StateMap},
	DirstateEntry, DirstateParents,			DirstateEntry, DirstateParents,
	};			};
	use byteorder::{BigEndian, WriteBytesExt};			use byteorder::{BigEndian, WriteBytesExt};
	use bytes_cast::BytesCast;			use bytes_cast::BytesCast;
	use micro_timer::timed;			use micro_timer::timed;
	use std::convert::{TryFrom, TryInto};			use std::convert::{TryFrom, TryInto};
	let copy_source = iter.next().map(HgPath::new);			let copy_source = iter.next().map(HgPath::new);
	each_entry(path, &entry, copy_source);			each_entry(path, &entry, copy_source);

	contents = rest;			contents = rest;
	}			}
	Ok(parents)			Ok(parents)
	}			}

				fn packed_filename_and_copy_source_size(
				filename: &HgPathBuf,
				copy_source: Option<&HgPathBuf>,
				) -> usize {
				filename.len()
				+ if let Some(source) = copy_source {
				b"\0".len() + source.len()
				} else {
				0
				}
				}

				pub fn packed_entry_size(
				filename: &HgPathBuf,
				copy_source: Option<&HgPathBuf>,
				) -> usize {
				MIN_ENTRY_SIZE
				+ packed_filename_and_copy_source_size(filename, copy_source)
				}

				pub fn pack_entry(
				filename: &HgPathBuf,
				entry: &DirstateEntry,
				copy_source: Option<&HgPathBuf>,
				packed: &mut Vec<u8>,
				) {
				let length = packed_filename_and_copy_source_size(filename, copy_source);

				// Unwrapping because `impl std::io::Write for Vec<u8>` never errors
				packed.write_u8(entry.state.into()).unwrap();
				packed.write_i32::<BigEndian>(entry.mode).unwrap();
				packed.write_i32::<BigEndian>(entry.size).unwrap();
				packed.write_i32::<BigEndian>(entry.mtime).unwrap();
				packed.write_i32::<BigEndian>(length as i32).unwrap();
				packed.extend(filename.as_bytes());
				if let Some(source) = copy_source {
				packed.push(b'\0');
				packed.extend(source.as_bytes());
				}
				}

	/// Seconds since the Unix epoch			/// Seconds since the Unix epoch
	pub struct Timestamp(pub u64);			pub struct Timestamp(pub u64);

				pub fn clear_ambiguous_mtime(
				entry: &mut DirstateEntry,
				mtime_now: i32,
				) -> bool {
				let ambiguous =
				entry.state == EntryState::Normal && entry.mtime == mtime_now;
				if ambiguous {
				// The file was last modified "simultaneously" with the current
				// write to dirstate (i.e. within the same second for file-
				// systems with a granularity of 1 sec). This commonly happens
				// for at least a couple of files on 'update'.
				// The user could change the file without changing its size
				// within the same second. Invalidate the file's mtime in
				// dirstate, forcing future 'status' calls to compare the
				// contents of the file if the size is the same. This prevents
				// mistakenly treating such files as clean.
				entry.mtime = -1;
				}
				ambiguous
				}

	pub fn pack_dirstate(			pub fn pack_dirstate(
	state_map: &mut StateMap,			state_map: &mut StateMap,
	copy_map: &CopyMap,			copy_map: &CopyMap,
	parents: DirstateParents,			parents: DirstateParents,
	now: Timestamp,			now: Timestamp,
	) -> Result<Vec<u8>, HgError> {			) -> Result<Vec<u8>, HgError> {
	// TODO move away from i32 before 2038.			// TODO move away from i32 before 2038.
	let now: i32 = now.0.try_into().expect("time overflow");			let now: i32 = now.0.try_into().expect("time overflow");

	let expected_size: usize = state_map			let expected_size: usize = state_map
	.iter()			.iter()
	.map(\|(filename, _)\| {			.map(\|(filename, _)\| {
	let mut length = MIN_ENTRY_SIZE + filename.len();			packed_entry_size(filename, copy_map.get(filename))
	if let Some(copy) = copy_map.get(filename) {
	length += copy.len() + 1;
	}
	length
	})			})
	.sum();			.sum();
	let expected_size = expected_size + PARENT_SIZE * 2;			let expected_size = expected_size + PARENT_SIZE * 2;

	let mut packed = Vec::with_capacity(expected_size);			let mut packed = Vec::with_capacity(expected_size);

	packed.extend(parents.p1.as_bytes());			packed.extend(parents.p1.as_bytes());
	packed.extend(parents.p2.as_bytes());			packed.extend(parents.p2.as_bytes());

	for (filename, entry) in state_map.iter_mut() {			for (filename, entry) in state_map.iter_mut() {
	let new_filename = filename.to_owned();			clear_ambiguous_mtime(entry, now);
	let mut new_mtime: i32 = entry.mtime;			pack_entry(filename, entry, copy_map.get(filename), &mut packed)
	if entry.state == EntryState::Normal && entry.mtime == now {
	// The file was last modified "simultaneously" with the current
	// write to dirstate (i.e. within the same second for file-
	// systems with a granularity of 1 sec). This commonly happens
	// for at least a couple of files on 'update'.
	// The user could change the file without changing its size
	// within the same second. Invalidate the file's mtime in
	// dirstate, forcing future 'status' calls to compare the
	// contents of the file if the size is the same. This prevents
	// mistakenly treating such files as clean.
	new_mtime = -1;
	*entry = DirstateEntry {
	mtime: new_mtime,
	..*entry
	};
	}
	let mut new_filename = new_filename.into_vec();
	if let Some(copy) = copy_map.get(filename) {
	new_filename.push(b'\0');
	new_filename.extend(copy.bytes());
	}

	// Unwrapping because `impl std::io::Write for Vec<u8>` never errors
	packed.write_u8(entry.state.into()).unwrap();
	packed.write_i32::<BigEndian>(entry.mode).unwrap();
	packed.write_i32::<BigEndian>(entry.size).unwrap();
	packed.write_i32::<BigEndian>(new_mtime).unwrap();
	packed
	.write_i32::<BigEndian>(new_filename.len() as i32)
	.unwrap();
	packed.extend(new_filename)
	}			}

	if packed.len() != expected_size {			if packed.len() != expected_size {
	return Err(HgError::CorruptedRepository(format!(			return Err(HgError::CorruptedRepository(format!(
	"bad dirstate size: {} != {}",			"bad dirstate size: {} != {}",
	expected_size,			expected_size,
	packed.len()			packed.len()
	)));			)));

rust/hg-core/src/dirstate_tree/dirstate_map.rs

	use std::collections::BTreeMap;			use bytes_cast::BytesCast;
	use std::path::PathBuf;			use std::path::PathBuf;
				use std::{collections::BTreeMap, convert::TryInto};

	use super::path_with_basename::WithBasename;			use super::path_with_basename::WithBasename;
				use crate::dirstate::parsers::clear_ambiguous_mtime;
				use crate::dirstate::parsers::pack_entry;
				use crate::dirstate::parsers::packed_entry_size;
	use crate::dirstate::parsers::parse_dirstate_entries;			use crate::dirstate::parsers::parse_dirstate_entries;
	use crate::dirstate::parsers::parse_dirstate_parents;			use crate::dirstate::parsers::parse_dirstate_parents;
	use crate::dirstate::parsers::Timestamp;			use crate::dirstate::parsers::Timestamp;

	use crate::matchers::Matcher;			use crate::matchers::Matcher;
	use crate::revlog::node::NULL_NODE;			use crate::revlog::node::NULL_NODE;
	use crate::utils::hg_path::{HgPath, HgPathBuf};			use crate::utils::hg_path::{HgPath, HgPathBuf};
	use crate::CopyMapIter;			use crate::CopyMapIter;
	use crate::DirstateEntry;			use crate::DirstateEntry;
	use crate::DirstateError;			use crate::DirstateError;
	use crate::DirstateMapError;			use crate::DirstateMapError;
	use crate::DirstateParents;			use crate::DirstateParents;
	self.set_parents(parents);			self.set_parents(parents);
	}			}

	Ok(Some(parents))			Ok(Some(parents))
	}			}

	fn pack(			fn pack(
	&mut self,			&mut self,
	_parents: DirstateParents,			parents: DirstateParents,
	_now: Timestamp,			now: Timestamp,
	) -> Result<Vec<u8>, DirstateError> {			) -> Result<Vec<u8>, DirstateError> {
	let _ = self.iter_node_data_mut();			// Optizimation (to be measured?): pre-compute size to avoid `Vec`
	todo!()			// reallocations
				let mut size = parents.as_bytes().len();
				for (path, node) in self.iter_nodes() {
				if node.entry.is_some() {
				size += packed_entry_size(
				path.full_path(),
				node.copy_source.as_ref(),
				)
				}
				}

				let mut packed = Vec::with_capacity(size);
				packed.extend(parents.as_bytes());

				let now: i32 = now.0.try_into().expect("time overflow");
				for (path, opt_entry, copy_source) in self.iter_node_data_mut() {
				if let Some(entry) = opt_entry {
				clear_ambiguous_mtime(entry, now);
				pack_entry(
				path.full_path(),
				entry,
				copy_source.as_ref(),
				&mut packed,
				);
				}
				}
				self.dirty_parents = false;
				Ok(packed)
	}			}

	fn build_file_fold_map(&mut self) -> &FastHashMap<HgPathBuf, HgPathBuf> {			fn build_file_fold_map(&mut self) -> &FastHashMap<HgPathBuf, HgPathBuf> {
	todo!()			todo!()
	}			}

	fn set_all_dirs(&mut self) -> Result<(), DirstateMapError> {			fn set_all_dirs(&mut self) -> Result<(), DirstateMapError> {
	todo!()			todo!()