( )⚙ D11764 rhg: add support for narrow clones and sparse checkouts

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11764

rhg: add support for narrow clones and sparse checkouts
ClosedPublic

Authored by aalekseyev on Nov 16 2021, 7:11 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG005ae1a343f8: rhg: add support for narrow clones and sparse checkouts
Summary

This adds a minimal support that can be implemented without parsing the narrowspec.
We can parse the narrowspec and add support for more operations later.

The reason we need so few code changes is as follows:

Most operations need no special treatment of sparse because
some of them only read dirstate (rhg files without -r),
which bakes in the filtering,
some of them only read store (rhg files -r, rhg cat),
and some of them read no data at all (rhg root, rhg debugrequirements).

status is the command that might care about sparse, so we just disable
rhg on it.

For narrow clones, rhg files clearly needs the narrowspec to work
correctly, so we fall back.
rhg cat seems to work consistently with hg cat if the file exists.
If the file is hidden by narrow spec, the error message is different and confusing, so
that's something that we should improve in follow-up patches.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

aalekseyev created this revision.Nov 16 2021, 7:11 AM
Alphare requested changes to this revision.Nov 24 2021, 6:04 AM
Alphare added a subscriber: Alphare.

Small nit, but this is user-facing, so I'd rather have it corrected in the same breath as the other patches instead of forgetting to amend it myself :)

rust/rhg/src/commands/files.rs
59

s/support/supported/

This revision now requires changes to proceed.Nov 24 2021, 6:04 AM
aalekseyev added a comment.Nov 29 2021, 12:47 PM

Fixed the wording along with some other issues discovered in tests and review (we're using this patch in our internal fork).

This revision now requires changes to proceed.Nov 30 2021, 9:51 AM
aalekseyev added a comment.Nov 30 2021, 11:29 AM

I'm sorry. I forgot to backport a patch for shell compatibility. :-(
Did now.

aalekseyev added a commit: rHG005ae1a343f8: rhg: add support for narrow clones and sparse checkouts.Dec 3 2021, 10:42 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG005ae1a343f8: rhg: add support for narrow clones and sparse checkouts (authored by aalekseyev). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 31303

rust/hg-core/src/repo.rs

} }
} }
pub fn has_dirstate_v2(&self) -> bool { pub fn has_dirstate_v2(&self) -> bool {
self.requirements self.requirements
.contains(requirements::DIRSTATE_V2_REQUIREMENT) .contains(requirements::DIRSTATE_V2_REQUIREMENT)
} }
pub fn has_sparse(&self) -> bool {
self.requirements.contains(requirements::SPARSE_REQUIREMENT)
}
pub fn has_narrow(&self) -> bool {
self.requirements.contains(requirements::NARROW_REQUIREMENT)
}
fn dirstate_file_contents(&self) -> Result<Vec<u8>, HgError> { fn dirstate_file_contents(&self) -> Result<Vec<u8>, HgError> {
Ok(self Ok(self
.hg_vfs() .hg_vfs()
.read("dirstate") .read("dirstate")
.io_not_found_as_none()? .io_not_found_as_none()?
.unwrap_or(Vec::new())) .unwrap_or(Vec::new()))
} }

rust/hg-core/src/requirements.rs

SPARSEREVLOG_REQUIREMENT, SPARSEREVLOG_REQUIREMENT,
RELATIVE_SHARED_REQUIREMENT, RELATIVE_SHARED_REQUIREMENT,
REVLOG_COMPRESSION_ZSTD, REVLOG_COMPRESSION_ZSTD,
DIRSTATE_V2_REQUIREMENT, DIRSTATE_V2_REQUIREMENT,
// As of this writing everything rhg does is read-only. // As of this writing everything rhg does is read-only.
// When it starts writing to the repository, it’ll need to either keep the // When it starts writing to the repository, it’ll need to either keep the
// persistent nodemap up to date or remove this entry: // persistent nodemap up to date or remove this entry:
NODEMAP_REQUIREMENT, NODEMAP_REQUIREMENT,
// Not all commands support `sparse` and `narrow`. The commands that do
// not should opt out by checking `has_sparse` and `has_narrow`.
SPARSE_REQUIREMENT,
NARROW_REQUIREMENT,
]; ];
// Copied from mercurial/requirements.py: // Copied from mercurial/requirements.py:
pub(crate) const DIRSTATE_V2_REQUIREMENT: &str = "dirstate-v2"; pub(crate) const DIRSTATE_V2_REQUIREMENT: &str = "dirstate-v2";
/// When narrowing is finalized and no longer subject to format changes, /// When narrowing is finalized and no longer subject to format changes,
/// we should move this to just "narrow" or similar. /// we should move this to just "narrow" or similar.

rust/rhg/src/commands/files.rs

return Err(CommandError::unsupported( return Err(CommandError::unsupported(
"non-default ui.relative-paths", "non-default ui.relative-paths",
)); ));
} }
let rev = invocation.subcommand_args.value_of("rev"); let rev = invocation.subcommand_args.value_of("rev");
let repo = invocation.repo?; let repo = invocation.repo?;
// It seems better if this check is removed: this would correspond to
// automatically enabling the extension if the repo requires it.
// However we need this check to be in sync with vanilla hg so hg tests
// pass.
if repo.has_sparse()
&& invocation.config.get(b"extensions", b"sparse").is_none()
{
return Err(CommandError::unsupported(
"repo is using sparse, but sparse extension is not enabled",
));
}
if let Some(rev) = rev { if let Some(rev) = rev {
if repo.has_narrow() {
return Err(CommandError::unsupported(
"rhg files -r <rev> is not supported in narrow clones",
AlphareUnsubmitted
Not Done

s/support/supported/

Alphare: s/support/supported/
));
}
let files = list_rev_tracked_files(repo, rev).map_err(|e| (e, rev))?; let files = list_rev_tracked_files(repo, rev).map_err(|e| (e, rev))?;
display_files(invocation.ui, repo, files.iter()) display_files(invocation.ui, repo, files.iter())
} else { } else {
// The dirstate always reflects the sparse narrowspec, so if
// we only have sparse without narrow all is fine.
// If we have narrow, then [hg files] needs to check if
// the store narrowspec is in sync with the one of the dirstate,
// so we can't support that without explicit code.
if repo.has_narrow() {
return Err(CommandError::unsupported(
"rhg files is not supported in narrow clones",
));
}
let distate = Dirstate::new(repo)?; let distate = Dirstate::new(repo)?;
let files = distate.tracked_files()?; let files = distate.tracked_files()?;
display_files(invocation.ui, repo, files.into_iter().map(Ok)) display_files(invocation.ui, repo, files.into_iter().map(Ok))
} }
} }
fn display_files<'a>( fn display_files<'a>(
ui: &Ui, ui: &Ui,

rust/rhg/src/commands/status.rs

DEFAULT_DISPLAY_STATES DEFAULT_DISPLAY_STATES
} else { } else {
requested requested
} }
}; };
let no_status = args.is_present("no-status"); let no_status = args.is_present("no-status");
let repo = invocation.repo?; let repo = invocation.repo?;
if repo.has_sparse() || repo.has_narrow() {
return Err(CommandError::unsupported(
"rhg status is not supported for sparse checkouts or narrow clones yet"
));
}
let mut dmap = repo.dirstate_map_mut()?; let mut dmap = repo.dirstate_map_mut()?;
let options = StatusOptions { let options = StatusOptions {
// we're currently supporting file systems with exec flags only // we're currently supporting file systems with exec flags only
// anyway // anyway
check_exec: true, check_exec: true,
list_clean: display_states.clean, list_clean: display_states.clean,
list_unknown: display_states.unknown, list_unknown: display_states.unknown,

rust/rhg/src/main.rs

Some(_) => { Some(_) => {
// TODO: warn about unknown config value // TODO: warn about unknown config value
Self::DEFAULT Self::DEFAULT
} }
} }
} }
} }
const SUPPORTED_EXTENSIONS: &[&[u8]] = &[b"blackbox", b"share"]; const SUPPORTED_EXTENSIONS: &[&[u8]] =
&[b"blackbox", b"share", b"sparse", b"narrow"];
fn check_extensions(config: &Config) -> Result<(), CommandError> { fn check_extensions(config: &Config) -> Result<(), CommandError> {
let enabled = config.get_section_keys(b"extensions"); let enabled = config.get_section_keys(b"extensions");
let mut unsupported = enabled; let mut unsupported = enabled;
for supported in SUPPORTED_EXTENSIONS { for supported in SUPPORTED_EXTENSIONS {
unsupported.remove(supported); unsupported.remove(supported);
} }

tests/test-rhg-sparse-narrow.t

  • This file was added.
#require rhg
$ NO_FALLBACK="env RHG_ON_UNSUPPORTED=abort"
Rhg works well when sparse working copy is enabled.
$ cd "$TESTTMP"
$ hg init repo-sparse
$ cd repo-sparse
$ cat > .hg/hgrc <<EOF
> [extensions]
> sparse=
> EOF
$ echo a > show
$ echo x > hide
$ mkdir dir1 dir2
$ echo x > dir1/x
$ echo y > dir1/y
$ echo z > dir2/z
$ hg ci -Aqm 'initial'
$ hg debugsparse --include 'show'
$ ls -A
.hg
show
$ tip=$(hg log -r . --template '{node}')
$ $NO_FALLBACK rhg files -r "$tip"
dir1/x
dir1/y
dir2/z
hide
show
$ $NO_FALLBACK rhg files
show
$ $NO_FALLBACK rhg cat -r "$tip" hide
x
$ cd ..
We support most things when narrow is enabled, too, with a couple of caveats.
$ . "$TESTDIR/narrow-library.sh"
$ real_hg=$RHG_FALLBACK_EXECUTABLE
$ cat >> $HGRCPATH <<EOF
> [extensions]
> narrow=
> EOF
$ hg clone --narrow ./repo-sparse repo-narrow --include dir1
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 2 changes to 2 files
new changesets 6d714a4a2998
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd repo-narrow
$ $NO_FALLBACK rhg cat -r "$tip" dir1/x
x
$ "$real_hg" cat -r "$tip" dir1/x
x
TODO: bad error message
$ $NO_FALLBACK rhg cat -r "$tip" hide
abort: invalid revision identifier: 6d714a4a2998cbfd0620db44da58b749f6565d63
[255]
$ "$real_hg" cat -r "$tip" hide
[1]
A naive implementation of [rhg files] leaks the paths that are supposed to be
hidden by narrow, so we just fall back to hg.
$ $NO_FALLBACK rhg files -r "$tip"
unsupported feature: rhg files -r <rev> is not supported in narrow clones
[252]
$ "$real_hg" files -r "$tip"
dir1/x
dir1/y
Hg status needs to do some filtering based on narrow spec, so we don't
support it in rhg for narrow clones yet.
$ mkdir dir2
$ touch dir2/q
$ "$real_hg" status
$ $NO_FALLBACK rhg --config rhg.status=true status
unsupported feature: rhg status is not supported for sparse checkouts or narrow clones yet
[252]
Adding "orphaned" index files:
$ (cd ..; cp repo-sparse/.hg/store/data/hide.i repo-narrow/.hg/store/data/hide.i)
$ (cd ..; mkdir repo-narrow/.hg/store/data/dir2; cp repo-sparse/.hg/store/data/dir2/z.i repo-narrow/.hg/store/data/dir2/z.i)
$ "$real_hg" verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 1 changesets with 2 changes to 2 files
$ "$real_hg" files -r "$tip"
dir1/x
dir1/y
# TODO: even though [hg files] hides the orphaned dir2/z, [hg cat] still shows it.
# rhg has the same issue, but at least it's not specific to rhg.
# This is despite [hg verify] succeeding above.
$ $NO_FALLBACK rhg cat -r "$tip" dir2/z
z
$ "$real_hg" cat -r "$tip" dir2/z
z