This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11764

rhg: add support for narrow clones and sparse checkouts
ClosedPublic

Authored by aalekseyev on Nov 16 2021, 7:11 AM.

Download Raw Diff

Details

Reviewers

Alphare

Group Reviewers

hg-reviewers

Commits

rHG005ae1a343f8: rhg: add support for narrow clones and sparse checkouts

Summary

This adds a minimal support that can be implemented without parsing the narrowspec.
We can parse the narrowspec and add support for more operations later.

The reason we need so few code changes is as follows:

Most operations need no special treatment of sparse because
some of them only read dirstate (rhg files without -r),
which bakes in the filtering,
some of them only read store (rhg files -r, rhg cat),
and some of them read no data at all (rhg root, rhg debugrequirements).

status is the command that might care about sparse, so we just disable
rhg on it.

For narrow clones, rhg files clearly needs the narrowspec to work
correctly, so we fall back.
rhg cat seems to work consistently with hg cat if the file exists.
If the file is hidden by narrow spec, the error message is different and confusing, so
that's something that we should improve in follow-up patches.

Diff Detail

Repository

rHG Mercurial

Branch

default

Lint

No Linters Available

Unit

No Unit Test Coverage

Event Timeline

aalekseyev created this revision.Nov 16 2021, 7:11 AM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptNov 16 2021, 7:11 AM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

aalekseyev mentioned this in D11752: rhg: allow rhg in sparse repos when the operations only need the store.Nov 16 2021, 7:19 AM

Small nit, but this is user-facing, so I'd rather have it corrected in the same breath as the other patches instead of forgetting to amend it myself :)

rust/rhg/src/commands/files.rs
58	s/support/supported/

This revision now requires changes to proceed.Nov 24 2021, 6:04 AM

aalekseyev updated this revision to Diff 31195.Nov 29 2021, 12:39 PM

Fixed the wording along with some other issues discovered in tests and review (we're using this patch in our internal fork).

Seems like this is broken: https://foss.heptapod.net/mercurial/mercurial-devel/-/jobs/269980

This revision now requires changes to proceed.Nov 30 2021, 9:51 AM

aalekseyev updated this revision to Diff 31219.Nov 30 2021, 11:27 AM

I'm sorry. I forgot to backport a patch for shell compatibility. :-(
Did now.

aalekseyev added a commit: rHG005ae1a343f8: rhg: add support for narrow clones and sparse checkouts.Dec 3 2021, 10:42 AM

This revision was not accepted when it landed; it landed in state Needs Review.

Closed by commit rHG005ae1a343f8: rhg: add support for narrow clones and sparse checkouts (authored by aalekseyev). · Explain Why

This revision was automatically updated to reflect the committed changes.

Thanks

Revision Contents
Changeset List

		Path
M		rust/hg-core/src/repo.rs (8 lines)
M		rust/hg-core/src/requirements.rs (4 lines)
M		rust/rhg/src/commands/files.rs (28 lines)
M		rust/rhg/src/commands/status.rs (7 lines)
M		rust/rhg/src/main.rs (3 lines)
A	M	tests/test-rhg-sparse-narrow.t (120 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	31080		Nov 16 2021, 7:11 AM	★	★
Diff 2	31195		Nov 29 2021, 12:39 PM	★	★
Diff 3	31219		Nov 30 2021, 11:27 AM	★	★
Diff 4	31303	rHG005ae1a343f87c27f278a6a0a639458947398325	Nov 16 2021, 6:53 AM	★	★

Commit	Parents	Author	Summary	Date
958912e9d774	645ae3ee63f9	Arseniy Alekseyev		Nov 16 2021, 6:53 AM

Diff 31195

rust/hg-core/src/repo.rs

	}			}
	}			}

	pub fn has_dirstate_v2(&self) -> bool {			pub fn has_dirstate_v2(&self) -> bool {
	self.requirements			self.requirements
	.contains(requirements::DIRSTATE_V2_REQUIREMENT)			.contains(requirements::DIRSTATE_V2_REQUIREMENT)
	}			}

				pub fn has_sparse(&self) -> bool {
				self.requirements.contains(requirements::SPARSE_REQUIREMENT)
				}

				pub fn has_narrow(&self) -> bool {
				self.requirements.contains(requirements::NARROW_REQUIREMENT)
				}

	fn dirstate_file_contents(&self) -> Result<Vec<u8>, HgError> {			fn dirstate_file_contents(&self) -> Result<Vec<u8>, HgError> {
	Ok(self			Ok(self
	.hg_vfs()			.hg_vfs()
	.read("dirstate")			.read("dirstate")
	.io_not_found_as_none()?			.io_not_found_as_none()?
	.unwrap_or(Vec::new()))			.unwrap_or(Vec::new()))
	}			}

rust/hg-core/src/requirements.rs

	SPARSEREVLOG_REQUIREMENT,			SPARSEREVLOG_REQUIREMENT,
	RELATIVE_SHARED_REQUIREMENT,			RELATIVE_SHARED_REQUIREMENT,
	REVLOG_COMPRESSION_ZSTD,			REVLOG_COMPRESSION_ZSTD,
	DIRSTATE_V2_REQUIREMENT,			DIRSTATE_V2_REQUIREMENT,
	// As of this writing everything rhg does is read-only.			// As of this writing everything rhg does is read-only.
	// When it starts writing to the repository, it’ll need to either keep the			// When it starts writing to the repository, it’ll need to either keep the
	// persistent nodemap up to date or remove this entry:			// persistent nodemap up to date or remove this entry:
	NODEMAP_REQUIREMENT,			NODEMAP_REQUIREMENT,
				// Not all commands support `sparse` and `narrow`. The commands that do
				// not should opt out by checking `has_sparse` and `has_narrow`.
				SPARSE_REQUIREMENT,
				NARROW_REQUIREMENT,
	];			];

	// Copied from mercurial/requirements.py:			// Copied from mercurial/requirements.py:

	pub(crate) const DIRSTATE_V2_REQUIREMENT: &str = "dirstate-v2";			pub(crate) const DIRSTATE_V2_REQUIREMENT: &str = "dirstate-v2";

	/// When narrowing is finalized and no longer subject to format changes,			/// When narrowing is finalized and no longer subject to format changes,
	/// we should move this to just "narrow" or similar.			/// we should move this to just "narrow" or similar.

rust/rhg/src/commands/files.rs

	return Err(CommandError::unsupported(			return Err(CommandError::unsupported(
	"non-default ui.relative-paths",			"non-default ui.relative-paths",
	));			));
	}			}

	let rev = invocation.subcommand_args.value_of("rev");			let rev = invocation.subcommand_args.value_of("rev");

	let repo = invocation.repo?;			let repo = invocation.repo?;

				// It seems better if this check is removed: this would correspond to
				// automatically enabling the extension if the repo requires it.
				// However we need this check to be in sync with vanilla hg so hg tests
				// pass.
				if repo.has_sparse()
				&& invocation.config.get(b"extensions", b"sparse").is_none()
				{
				return Err(CommandError::unsupported(
				"repo is using sparse, but sparse extension is not enabled",
				));
				}

	if let Some(rev) = rev {			if let Some(rev) = rev {
				if repo.has_narrow() {
				return Err(CommandError::unsupported(
				"rhg files -r <rev> is not supported in narrow clones",
				AlphareUnsubmitted Not Done s/support/supported/ Alphare: s/support/supported/
				));
				}
	let files = list_rev_tracked_files(repo, rev).map_err(\|e\| (e, rev))?;			let files = list_rev_tracked_files(repo, rev).map_err(\|e\| (e, rev))?;
	display_files(invocation.ui, repo, files.iter())			display_files(invocation.ui, repo, files.iter())
	} else {			} else {
				// The dirstate always reflects the sparse narrowspec, so if
				// we only have sparse without narrow all is fine.
				// If we have narrow, then [hg files] needs to check if
				// the store narrowspec is in sync with the one of the dirstate,
				// so we can't support that without explicit code.
				if repo.has_narrow() {
				return Err(CommandError::unsupported(
				"rhg files is not supported in narrow clones",
				));
				}
	let distate = Dirstate::new(repo)?;			let distate = Dirstate::new(repo)?;
	let files = distate.tracked_files()?;			let files = distate.tracked_files()?;
	display_files(invocation.ui, repo, files)			display_files(invocation.ui, repo, files)
	}			}
	}			}

	fn display_files<'a>(			fn display_files<'a>(
	ui: &Ui,			ui: &Ui,

rust/rhg/src/commands/status.rs

	if requested.is_empty() {			if requested.is_empty() {
	DEFAULT_DISPLAY_STATES			DEFAULT_DISPLAY_STATES
	} else {			} else {
	requested			requested
	}			}
	};			};

	let repo = invocation.repo?;			let repo = invocation.repo?;

				if repo.has_sparse() \|\| repo.has_narrow() {
				return Err(CommandError::unsupported(
				"rhg status is not supported for sparse checkouts or narrow clones yet"
				));
				}

	let mut dmap = repo.dirstate_map_mut()?;			let mut dmap = repo.dirstate_map_mut()?;

	let options = StatusOptions {			let options = StatusOptions {
	// TODO should be provided by the dirstate parsing and			// TODO should be provided by the dirstate parsing and
	// hence be stored on dmap. Using a value that assumes we aren't			// hence be stored on dmap. Using a value that assumes we aren't
	// below the time resolution granularity of the FS and the			// below the time resolution granularity of the FS and the
	// dirstate.			// dirstate.
	last_normal_time: TruncatedTimestamp::new_truncate(0, 0),			last_normal_time: TruncatedTimestamp::new_truncate(0, 0),

rust/rhg/src/main.rs

	Some(_) => {			Some(_) => {
	// TODO: warn about unknown config value			// TODO: warn about unknown config value
	Self::DEFAULT			Self::DEFAULT
	}			}
	}			}
	}			}
	}			}

	const SUPPORTED_EXTENSIONS: &[&[u8]] = &[b"blackbox", b"share"];			const SUPPORTED_EXTENSIONS: &[&[u8]] =
				&[b"blackbox", b"share", b"sparse", b"narrow"];

	fn check_extensions(config: &Config) -> Result<(), CommandError> {			fn check_extensions(config: &Config) -> Result<(), CommandError> {
	let enabled = config.get_section_keys(b"extensions");			let enabled = config.get_section_keys(b"extensions");

	let mut unsupported = enabled;			let mut unsupported = enabled;
	for supported in SUPPORTED_EXTENSIONS {			for supported in SUPPORTED_EXTENSIONS {
	unsupported.remove(supported);			unsupported.remove(supported);
	}			}

tests/test-rhg-sparse-narrow.t

This file was added.

				#require rhg

				$ NO_FALLBACK="env RHG_ON_UNSUPPORTED=abort"

				Rhg works well when sparse working copy is enabled.

				$ cd "$TESTTMP"
				$ hg init repo-sparse
				$ cd repo-sparse
				$ cat > .hg/hgrc <<EOF
				> [extensions]
				> sparse=
				> EOF

				$ echo a > show
				$ echo x > hide
				$ mkdir dir1 dir2
				$ echo x > dir1/x
				$ echo y > dir1/y
				$ echo z > dir2/z

				$ hg ci -Aqm 'initial'
				$ hg debugsparse --include 'show'
				$ ls -A
				.hg
				show

				$ tip=$(hg log -r . --template '{node}')
				$ $NO_FALLBACK rhg files -r "$tip"
				dir1/x
				dir1/y
				dir2/z
				hide
				show
				$ $NO_FALLBACK rhg files
				show

				$ $NO_FALLBACK rhg cat -r "$tip" hide
				x

				$ cd ..

				We support most things when narrow is enabled, too, with a couple of caveats.

				$ . "$TESTDIR/narrow-library.sh"
				$ real_hg=$RHG_FALLBACK_EXECUTABLE

				$ cat >> $HGRCPATH <<EOF
				> [extensions]
				> narrow=
				> EOF

				$ hg clone --narrow ./repo-sparse repo-narrow --include dir1
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 2 changes to 2 files
				new changesets 6d714a4a2998
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ cd repo-narrow

				$ $NO_FALLBACK rhg cat -r "$tip" dir1/x
				x
				$ "$real_hg" cat -r "$tip" dir1/x
				x

				TODO: bad error message

				$ $NO_FALLBACK rhg cat -r "$tip" hide
				abort: invalid revision identifier: 6d714a4a2998cbfd0620db44da58b749f6565d63
				[255]
				$ "$real_hg" cat -r "$tip" hide
				[1]

				A naive implementation of [rhg files] leaks the paths that are supposed to be
				hidden by narrow, so we just fall back to hg.

				$ $NO_FALLBACK rhg files -r "$tip"
				unsupported feature: rhg files -r <rev> is not supported in narrow clones
				[252]
				$ "$real_hg" files -r "$tip"
				dir1/x
				dir1/y

				Hg status needs to do some filtering based on narrow spec, so we don't
				support it in rhg for narrow clones yet.

				$ mkdir dir2
				$ touch dir2/q
				$ "$real_hg" status
				$ $NO_FALLBACK rhg --config rhg.status=true status
				unsupported feature: rhg status is not supported for sparse checkouts or narrow clones yet
				[252]

				Adding "orphaned" index files:

				$ (cd ..; cp {repo-sparse,repo-narrow}/.hg/store/data/hide.i)
				$ (cd ..; mkdir repo-narrow/.hg/store/data/dir2; cp {repo-sparse,repo-narrow}/.hg/store/data/dir2/z.i)
				$ "$real_hg" verify
				checking changesets
				checking manifests
				crosschecking files in changesets and manifests
				checking files
				checked 1 changesets with 2 changes to 2 files

				$ "$real_hg" files -r "$tip"
				dir1/x
				dir1/y

				# TODO: even though [hg files] hides the orphaned dir2/z, [hg cat] still shows it.
				# rhg has the same issue, but at least it's not specific to rhg.
				# This is despite [hg verify] succeeding above.

				$ $NO_FALLBACK rhg cat -r "$tip" dir2/z
				z
				$ "$real_hg" cat -r "$tip" dir2/z
				z