This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11688

dirstate-v2: preserve the fallback values on disk
ClosedPublic

Authored by marmoute on Oct 18 2021, 9:20 PM.

Download Raw Diff

Details

Reviewers

None

Group Reviewers

hg-reviewers

Commits

rHGb874e8d81a98: dirstate-v2: preserve the fallback values on disk
rHG5f8ee8157dcc: dirstate-v2: preserve the fallback values on disk

Summary

When the fallback values are set, they are now read and written to disk.

See format documentation for details.

Diff Detail

Repository

rHG Mercurial

Branch

default

Lint

No Linters Available

Unit

No Unit Test Coverage

Event Timeline

marmoute created this revision.Oct 18 2021, 9:20 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptOct 18 2021, 9:20 PM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

LGTM

mercurial/helptext/internals/dirstate-v2.txt
576–579	Looking further ahead to the UI, it would feel weird if it was a command that only worked on Windows. Is the plan to simply chmod+x the file on !windows, and then continue on looking to the filesystem like is done now? Is that too complicated with symlinks (where you'd have to delete the existing file/symlink and create a symlink/file when the filesystem supports symlinks)?

Alphare added a subscriber: Alphare.Oct 19 2021, 4:29 AM

Alphare added inline comments.

mercurial/helptext/internals/dirstate-v2.txt
567	Maybe `the entry carries "fallback" information`?
582	Same change as above

marmoute updated this revision to Diff 30893.Oct 19 2021, 7:49 AM

marmoute updated this revision to Diff 30895.Oct 19 2021, 9:20 AM

marmoute added inline comments.Oct 19 2021, 9:21 AM

mercurial/helptext/internals/dirstate-v2.txt
576–579	The plan will always be to look at things in this order: the file system the fallback value the parents manifests (this series introduce a (2) here). The first user of this logic will be the merge code so that they can properly apply mode changes on system where the mode is unsupported. It will also make sense to add some command to manually set this. In such case, this would also update -on-disk- status on the file for consistency. I don't think either of the above will be present in 6.0 for time constraint reason, but it is important to have the semantic added early. Does this address your question ?

marmoute added a commit: rHG5f8ee8157dcc: dirstate-v2: preserve the fallback values on disk.Oct 19 2021, 1:01 PM

marmoute added a commit: rHGb874e8d81a98: dirstate-v2: preserve the fallback values on disk.

This revision was not accepted when it landed; it landed in state Needs Review.

Closed by commit rHG5f8ee8157dcc: dirstate-v2: preserve the fallback values on disk (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/helptext/internals/dirstate-v2.txt (34 lines)
M			mercurial/pure/parsers.py (25 lines)
M			rust/hg-core/src/dirstate/entry.rs (20 lines)
M			rust/hg-core/src/dirstate_tree/on_disk.rs (27 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	30887		Oct 18 2021, 9:20 PM	★	★
Diff 2	30893		Oct 19 2021, 7:49 AM	★	★
Diff 3	30895		Oct 19 2021, 9:20 AM	★	★
Diff 4	30903	rHG5f8ee8157dcc69e65251e95d55b26b148ae82365	Oct 18 2021, 4:56 AM	★	★

Commit	Parents	Author	Summary	Date
cc10bfedcd6e	ba9e848ae2f7	Pierre-Yves David		Oct 18 2021, 4:56 AM

Status	Author	Revision
Closed	marmoute	D11688 dirstate-v2: preserve the fallback values on disk
Closed	marmoute	D11687 dirstate: make DirstateItem constructor accept fallback value
Closed	marmoute	D11686 dirstate: add a concept of "fallback" flags to dirstate item
Closed	marmoute	D11682 dirstate-v2: adds two flag to track the presence of some unrecorded files
Closed	marmoute	D11681 dirstate-v2: adds a flag to mark a file as modified

Diff 30893

mercurial/helptext/internals/dirstate-v2.txt

	HAS_MODE_AND_SIZE = 1 << 3			HAS_MODE_AND_SIZE = 1 << 3
	HAS_FILE_MTIME = 1 << 4			HAS_FILE_MTIME = 1 << 4
	HAS_DIRECTORY_MTIME = 1 << 5			HAS_DIRECTORY_MTIME = 1 << 5
	MODE_EXEC_PERM = 1 << 6			MODE_EXEC_PERM = 1 << 6
	MODE_IS_SYMLINK = 1 << 7			MODE_IS_SYMLINK = 1 << 7
	EXPECTED_STATE_IS_MODIFIED = 1 << 8			EXPECTED_STATE_IS_MODIFIED = 1 << 8
	ALL_UNKNOWN_RECORDED = 1 << 9			ALL_UNKNOWN_RECORDED = 1 << 9
	ALL_IGNORED_RECORDED = 1 << 10			ALL_IGNORED_RECORDED = 1 << 10
				HAS_FALLBACK_EXEC = 1 << 11
				FALLBACK_EXEC = 1 << 12
				HAS_FALLBACK_SYMLINK = 1 << 13
				FALLBACK_SYMLINK = 1 << 14

	The meaning of each bit is described below.			The meaning of each bit is described below.

	Other bits are unset.			Other bits are unset.
	They may be assigned meaning if the future,			They may be assigned meaning if the future,
	with the limitation that Mercurial versions that pre-date such meaning			with the limitation that Mercurial versions that pre-date such meaning
	will always reset those bits to unset when writing nodes.			will always reset those bits to unset when writing nodes.
	(A new node is written for any mutation in its subtree,			(A new node is written for any mutation in its subtree,
	are listed.			are listed.

	Note that the amount recorded "ignored" children can still be zero if None			Note that the amount recorded "ignored" children can still be zero if None
	where present.			where present.

	Also note that having this flag unset does not imply that no "ignored"			Also note that having this flag unset does not imply that no "ignored"
	children have been recorded. Some might be present, but there is no garantee			children have been recorded. Some might be present, but there is no garantee
	that is will be all of them.			that is will be all of them.

				`HAS_FALLBACK_EXEC`
				If this flag is set, the entry carry as "fallback" information for the
				AlphareUnsubmitted Not Done Maybe `the entry carries "fallback" information`? Alphare: Maybe `the entry carries "fallback" information`?
				executable bit in the `FALLBACK_EXEC` flag.

				Fallback information can be stored in the dirstate to keep track of
				filesystem attribute tracked by Mercurial when the underlying file
				system or operating system does not support that property, (e.g.
				Windows).

				`FALLBACK_EXEC`
				Should be ignored if `HAS_FALLBACK_EXEC` is unset. If set the file for this
				entry should be considered executable if that information cannot be
				extracted from the file system. If unset it should be considered
				non-executable instead.
				mharbison72Unsubmitted Not Done Looking further ahead to the UI, it would feel weird if it was a command that only worked on Windows. Is the plan to simply chmod+x the file on !windows, and then continue on looking to the filesystem like is done now? Is that too complicated with symlinks (where you'd have to delete the existing file/symlink and create a symlink/file when the filesystem supports symlinks)? mharbison72: Looking further ahead to the UI, it would feel weird if it was a command that only worked on…
				marmouteAuthorUnsubmitted Done The plan will always be to look at things in this order: the file system the fallback value the parents manifests (this series introduce a (2) here). The first user of this logic will be the merge code so that they can properly apply mode changes on system where the mode is unsupported. It will also make sense to add some command to manually set this. In such case, this would also update -on-disk- status on the file for consistency. I don't think either of the above will be present in 6.0 for time constraint reason, but it is important to have the semantic added early. Does this address your question ? marmoute: The plan will always be to look at things in this order: 1) the file system 2) the fallback…

				`HAS_FALLBACK_SYMLINK`
				If this flag is set, the entry carry as "fallback" information for symbolic
				AlphareUnsubmitted Not Done Same change as above Alphare: Same change as above
				link status in the `FALLBACK_SYMLINK` flag.

				Fallback information can be stored in the dirstate to keep track of
				filesystem attribute tracked by Mercurial when the underlying file
				system or operating system does not support that property, (e.g.
				Windows).

				`FALLBACK_SYMLINK`
				Should be ignored if `HAS_FALLBACK_SYMLINK` is unset. If set the file for
				this entry should be considered a symlink if that information cannot be
				extracted from the file system. If unset it should be considered a normal
				file instead.

mercurial/pure/parsers.py

	DIRSTATE_V2_HAS_MODE_AND_SIZE = 1 << 3			DIRSTATE_V2_HAS_MODE_AND_SIZE = 1 << 3
	DIRSTATE_V2_HAS_FILE_MTIME = 1 << 4			DIRSTATE_V2_HAS_FILE_MTIME = 1 << 4
	_DIRSTATE_V2_HAS_DIRCTORY_MTIME = 1 << 5 # Unused when Rust is not available			_DIRSTATE_V2_HAS_DIRCTORY_MTIME = 1 << 5 # Unused when Rust is not available
	DIRSTATE_V2_MODE_EXEC_PERM = 1 << 6			DIRSTATE_V2_MODE_EXEC_PERM = 1 << 6
	DIRSTATE_V2_MODE_IS_SYMLINK = 1 << 7			DIRSTATE_V2_MODE_IS_SYMLINK = 1 << 7
	DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED = 1 << 8			DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED = 1 << 8
	DIRSTATE_V2_ALL_UNKNOWN_RECORDED = 1 << 9			DIRSTATE_V2_ALL_UNKNOWN_RECORDED = 1 << 9
	DIRSTATE_V2_ALL_IGNORED_RECORDED = 1 << 10			DIRSTATE_V2_ALL_IGNORED_RECORDED = 1 << 10
				DIRSTATE_V2_HAS_FALLBACK_EXEC = 1 << 11
				DIRSTATE_V2_FALLBACK_EXEC = 1 << 12
				DIRSTATE_V2_HAS_FALLBACK_SYMLINK = 1 << 13
				DIRSTATE_V2_FALLBACK_SYMLINK = 1 << 14


	@attr.s(slots=True, init=False)			@attr.s(slots=True, init=False)
	class DirstateItem(object):			class DirstateItem(object):
	"""represent a dirstate entry			"""represent a dirstate entry

	It hold multiple attributes			It hold multiple attributes

	mode = None			mode = None

	if flags & +DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED:			if flags & +DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED:
	# we do not have support for this flag in the code yet,			# we do not have support for this flag in the code yet,
	# force a lookup for this file.			# force a lookup for this file.
	has_mode_size = False			has_mode_size = False
	has_meaningful_mtime = False			has_meaningful_mtime = False

				fallback_exec = None
				if flags & DIRSTATE_V2_HAS_FALLBACK_EXEC:
				fallback_exec = flags & DIRSTATE_V2_FALLBACK_EXEC

				fallback_symlink = None
				if flags & DIRSTATE_V2_HAS_FALLBACK_SYMLINK:
				fallback_symlink = flags & DIRSTATE_V2_FALLBACK_SYMLINK

	if has_mode_size:			if has_mode_size:
	assert stat.S_IXUSR == 0o100			assert stat.S_IXUSR == 0o100
	if flags & DIRSTATE_V2_MODE_EXEC_PERM:			if flags & DIRSTATE_V2_MODE_EXEC_PERM:
	mode = 0o755			mode = 0o755
	else:			else:
	mode = 0o644			mode = 0o644
	if flags & DIRSTATE_V2_MODE_IS_SYMLINK:			if flags & DIRSTATE_V2_MODE_IS_SYMLINK:
	mode \|= stat.S_IFLNK			mode \|= stat.S_IFLNK
	else:			else:
	mode \|= stat.S_IFREG			mode \|= stat.S_IFREG
	return cls(			return cls(
	wc_tracked=bool(flags & DIRSTATE_V2_WDIR_TRACKED),			wc_tracked=bool(flags & DIRSTATE_V2_WDIR_TRACKED),
	p1_tracked=bool(flags & DIRSTATE_V2_P1_TRACKED),			p1_tracked=bool(flags & DIRSTATE_V2_P1_TRACKED),
	p2_info=bool(flags & DIRSTATE_V2_P2_INFO),			p2_info=bool(flags & DIRSTATE_V2_P2_INFO),
	has_meaningful_data=has_mode_size,			has_meaningful_data=has_mode_size,
	has_meaningful_mtime=has_meaningful_mtime,			has_meaningful_mtime=has_meaningful_mtime,
	parentfiledata=(mode, size, mtime),			parentfiledata=(mode, size, mtime),
				fallback_exec=fallback_exec,
				fallback_symlink=fallback_symlink,
	)			)

	@classmethod			@classmethod
	def from_v1_data(cls, state, mode, size, mtime):			def from_v1_data(cls, state, mode, size, mtime):
	"""Build a new DirstateItem object from V1 data			"""Build a new DirstateItem object from V1 data

	Since the dirstate-v1 format is frozen, the signature of this function			Since the dirstate-v1 format is frozen, the signature of this function
	is not expected to change, unlike the __init__ one.			is not expected to change, unlike the __init__ one.
	if self._mode is not None and self._size is not None:			if self._mode is not None and self._size is not None:
	flags \|= DIRSTATE_V2_HAS_MODE_AND_SIZE			flags \|= DIRSTATE_V2_HAS_MODE_AND_SIZE
	if self.mode & stat.S_IXUSR:			if self.mode & stat.S_IXUSR:
	flags \|= DIRSTATE_V2_MODE_EXEC_PERM			flags \|= DIRSTATE_V2_MODE_EXEC_PERM
	if stat.S_ISLNK(self.mode):			if stat.S_ISLNK(self.mode):
	flags \|= DIRSTATE_V2_MODE_IS_SYMLINK			flags \|= DIRSTATE_V2_MODE_IS_SYMLINK
	if self._mtime is not None:			if self._mtime is not None:
	flags \|= DIRSTATE_V2_HAS_FILE_MTIME			flags \|= DIRSTATE_V2_HAS_FILE_MTIME

				if self._fallback_exec is not None:
				flags \|= DIRSTATE_V2_HAS_FALLBACK_EXEC
				if self._fallback_exec:
				flags \|= DIRSTATE_V2_FALLBACK_EXEC

				if self._fallback_symlink is not None:
				flags \|= DIRSTATE_V2_HAS_FALLBACK_SYMLINK
				if self._fallback_symlink:
				flags \|= DIRSTATE_V2_FALLBACK_SYMLINK

	# Note: we do not need to do anything regarding			# Note: we do not need to do anything regarding
	# DIRSTATE_V2_ALL_UNKNOWN_RECORDED and DIRSTATE_V2_ALL_IGNORED_RECORDED			# DIRSTATE_V2_ALL_UNKNOWN_RECORDED and DIRSTATE_V2_ALL_IGNORED_RECORDED
	# since we never set _DIRSTATE_V2_HAS_DIRCTORY_MTIME			# since we never set _DIRSTATE_V2_HAS_DIRCTORY_MTIME
	return (flags, self._size or 0, self._mtime or 0)			return (flags, self._size or 0, self._mtime or 0)

	def v1_state(self):			def v1_state(self):
	"""return a "state" suitable for v1 serialization"""			"""return a "state" suitable for v1 serialization"""
	if not self.any_tracked:			if not self.any_tracked:

rust/hg-core/src/dirstate/entry.rs

	self.flags.intersects(			self.flags.intersects(
	Flags::WDIR_TRACKED \| Flags::P1_TRACKED \| Flags::P2_INFO,			Flags::WDIR_TRACKED \| Flags::P1_TRACKED \| Flags::P2_INFO,
	)			)
	}			}

	/// Returns `(wdir_tracked, p1_tracked, p2_info, mode_size, mtime)`			/// Returns `(wdir_tracked, p1_tracked, p2_info, mode_size, mtime)`
	pub(crate) fn v2_data(			pub(crate) fn v2_data(
	&self,			&self,
	) -> (bool, bool, bool, Option<(u32, u32)>, Option<u32>) {			) -> (
				bool,
				bool,
				bool,
				Option<(u32, u32)>,
				Option<u32>,
				Option<bool>,
				Option<bool>,
				) {
	if !self.any_tracked() {			if !self.any_tracked() {
	// TODO: return an Option instead?			// TODO: return an Option instead?
	panic!("Accessing v1_state of an untracked DirstateEntry")			panic!("Accessing v1_state of an untracked DirstateEntry")
	}			}
	let wdir_tracked = self.flags.contains(Flags::WDIR_TRACKED);			let wdir_tracked = self.flags.contains(Flags::WDIR_TRACKED);
	let p1_tracked = self.flags.contains(Flags::P1_TRACKED);			let p1_tracked = self.flags.contains(Flags::P1_TRACKED);
	let p2_info = self.flags.contains(Flags::P2_INFO);			let p2_info = self.flags.contains(Flags::P2_INFO);
	let mode_size = self.mode_size;			let mode_size = self.mode_size;
	let mtime = self.mtime;			let mtime = self.mtime;
	(wdir_tracked, p1_tracked, p2_info, mode_size, mtime)			(
				wdir_tracked,
				p1_tracked,
				p2_info,
				mode_size,
				mtime,
				self.get_fallback_exec(),
				self.get_fallback_symlink(),
				)
	}			}

	fn v1_state(&self) -> EntryState {			fn v1_state(&self) -> EntryState {
	if !self.any_tracked() {			if !self.any_tracked() {
	// TODO: return an Option instead?			// TODO: return an Option instead?
	panic!("Accessing v1_state of an untracked DirstateEntry")			panic!("Accessing v1_state of an untracked DirstateEntry")
	}			}
	if self.removed() {			if self.removed() {

rust/hg-core/src/dirstate_tree/on_disk.rs

	const HAS_MODE_AND_SIZE = 1 << 3;			const HAS_MODE_AND_SIZE = 1 << 3;
	const HAS_FILE_MTIME = 1 << 4;			const HAS_FILE_MTIME = 1 << 4;
	const HAS_DIRECTORY_MTIME = 1 << 5;			const HAS_DIRECTORY_MTIME = 1 << 5;
	const MODE_EXEC_PERM = 1 << 6;			const MODE_EXEC_PERM = 1 << 6;
	const MODE_IS_SYMLINK = 1 << 7;			const MODE_IS_SYMLINK = 1 << 7;
	const EXPECTED_STATE_IS_MODIFIED = 1 << 8;			const EXPECTED_STATE_IS_MODIFIED = 1 << 8;
	const ALL_UNKNOWN_RECORDED = 1 << 9;			const ALL_UNKNOWN_RECORDED = 1 << 9;
	const ALL_IGNORED_RECORDED = 1 << 10;			const ALL_IGNORED_RECORDED = 1 << 10;
				const HAS_FALLBACK_EXEC = 1 << 11;
				const FALLBACK_EXEC = 1 << 12;
				const HAS_FALLBACK_SYMLINK = 1 << 13;
				const FALLBACK_SYMLINK = 1 << 14;
	}			}
	}			}

	/// Duration since the Unix epoch			/// Duration since the Unix epoch
	#[derive(BytesCast, Copy, Clone)]			#[derive(BytesCast, Copy, Clone)]
	#[repr(C)]			#[repr(C)]
	struct PackedTruncatedTimestamp {			struct PackedTruncatedTimestamp {
	truncated_seconds: U32Be,			truncated_seconds: U32Be,
	.get(),			.get(),
	tracked_descendants_count: self.tracked_descendants_count.get(),			tracked_descendants_count: self.tracked_descendants_count.get(),
	})			})
	}			}

	fn from_dirstate_entry(			fn from_dirstate_entry(
	entry: &DirstateEntry,			entry: &DirstateEntry,
	) -> (Flags, U32Be, PackedTruncatedTimestamp) {			) -> (Flags, U32Be, PackedTruncatedTimestamp) {
	let (wdir_tracked, p1_tracked, p2_info, mode_size_opt, mtime_opt) =			let (
	entry.v2_data();			wdir_tracked,
				p1_tracked,
				p2_info,
				mode_size_opt,
				mtime_opt,
				fallback_exec,
				fallback_symlink,
				) = entry.v2_data();
	// TODO: convert throug raw flag bits instead?			// TODO: convert throug raw flag bits instead?
	let mut flags = Flags::empty();			let mut flags = Flags::empty();
	flags.set(Flags::WDIR_TRACKED, wdir_tracked);			flags.set(Flags::WDIR_TRACKED, wdir_tracked);
	flags.set(Flags::P1_TRACKED, p1_tracked);			flags.set(Flags::P1_TRACKED, p1_tracked);
	flags.set(Flags::P2_INFO, p2_info);			flags.set(Flags::P2_INFO, p2_info);
	let size = if let Some((m, s)) = mode_size_opt {			let size = if let Some((m, s)) = mode_size_opt {
	let exec_perm = m & libc::S_IXUSR != 0;			let exec_perm = m & libc::S_IXUSR != 0;
	let is_symlink = m & libc::S_IFMT == libc::S_IFLNK;			let is_symlink = m & libc::S_IFMT == libc::S_IFLNK;
	flags.set(Flags::MODE_EXEC_PERM, exec_perm);			flags.set(Flags::MODE_EXEC_PERM, exec_perm);
	flags.set(Flags::MODE_IS_SYMLINK, is_symlink);			flags.set(Flags::MODE_IS_SYMLINK, is_symlink);
	flags.insert(Flags::HAS_MODE_AND_SIZE);			flags.insert(Flags::HAS_MODE_AND_SIZE);
	s.into()			s.into()
	} else {			} else {
	0.into()			0.into()
	};			};
	let mtime = if let Some(m) = mtime_opt {			let mtime = if let Some(m) = mtime_opt {
	flags.insert(Flags::HAS_FILE_MTIME);			flags.insert(Flags::HAS_FILE_MTIME);
	PackedTruncatedTimestamp {			PackedTruncatedTimestamp {
	truncated_seconds: m.into(),			truncated_seconds: m.into(),
	nanoseconds: 0.into(),			nanoseconds: 0.into(),
	}			}
	} else {			} else {
	PackedTruncatedTimestamp::null()			PackedTruncatedTimestamp::null()
	};			};
				if let Some(f_exec) = fallback_exec {
				flags.insert(Flags::HAS_FALLBACK_EXEC);
				if f_exec {
				flags.insert(Flags::FALLBACK_EXEC);
				}
				}
				if let Some(f_symlink) = fallback_symlink {
				flags.insert(Flags::HAS_FALLBACK_SYMLINK);
				if f_symlink {
				flags.insert(Flags::FALLBACK_SYMLINK);
				}
				}
	(flags, size, mtime)			(flags, size, mtime)
	}			}
	}			}

	fn read_hg_path(			fn read_hg_path(
	on_disk: &[u8],			on_disk: &[u8],
	slice: PathSlice,			slice: PathSlice,
	) -> Result<&HgPath, DirstateV2ParseError> {			) -> Result<&HgPath, DirstateV2ParseError> {