( )⚙ D10320 remotefilelog: include file contents in bundles produced during strip

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10320

remotefilelog: include file contents in bundles produced during strip
ClosedPublic

Authored by spectral on Apr 6 2021, 8:21 PM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG47a9527731c3: remotefilelog: include file contents in bundles produced during strip
Summary

hg strip and other things that use repair.strip (such as the narrow
extension's hg tracked --removeinclude) will "save" some commits that have a
higher revision number than the oldest commit we're stripping, but aren't
actually descended from any of the commits that we're stripping. It saves them
in a bundle, and then reapplies them to the repo.

Remotefilelog doesn't generally participate in strip, it doesn't contribute
files to either the backup bundle or the "saved" bundle, and doesn't adjust
linknodes when commits are stripped. This can break things like push, which
rely on the linknodes.

This change makes it so that remotefilelog includes files in these bundles
during strip operations. During reapplication, the files are reapplied from the
bundle, and the linknode is properly updated.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

spectral created this revision.Apr 6 2021, 8:21 PM
pulkit accepted this revision.Apr 8 2021, 2:57 AM
This revision is now accepted and ready to land.Apr 8 2021, 2:57 AM

Revision Contents
Changeset List

Diff 26669

hgext/remotefilelog/__init__.py

configitem(b'remotefilelog', b'servercachepath', default=None) configitem(b'remotefilelog', b'servercachepath', default=None)
configitem(b"remotefilelog", b"serverexpiration", default=30) configitem(b"remotefilelog", b"serverexpiration", default=30)
configitem(b'remotefilelog', b'backgroundrepack', default=False) configitem(b'remotefilelog', b'backgroundrepack', default=False)
configitem(b'remotefilelog', b'bgprefetchrevs', default=None) configitem(b'remotefilelog', b'bgprefetchrevs', default=None)
configitem(b'remotefilelog', b'pullprefetch', default=None) configitem(b'remotefilelog', b'pullprefetch', default=None)
configitem(b'remotefilelog', b'backgroundprefetch', default=False) configitem(b'remotefilelog', b'backgroundprefetch', default=False)
configitem(b'remotefilelog', b'prefetchdelay', default=120) configitem(b'remotefilelog', b'prefetchdelay', default=120)
configitem(b'remotefilelog', b'prefetchdays', default=14) configitem(b'remotefilelog', b'prefetchdays', default=14)
# Other values include 'local' or 'none'. Any unrecognized value is 'all'.
configitem(b'remotefilelog', b'strip.includefiles', default='all')
configitem(b'remotefilelog', b'getfilesstep', default=10000) configitem(b'remotefilelog', b'getfilesstep', default=10000)
configitem(b'remotefilelog', b'getfilestype', default=b'optimistic') configitem(b'remotefilelog', b'getfilestype', default=b'optimistic')
configitem(b'remotefilelog', b'batchsize', configitems.dynamicdefault) configitem(b'remotefilelog', b'batchsize', configitems.dynamicdefault)
configitem(b'remotefilelog', b'fetchwarning', default=b'') configitem(b'remotefilelog', b'fetchwarning', default=b'')
configitem(b'remotefilelog', b'includepattern', default=None) configitem(b'remotefilelog', b'includepattern', default=None)
configitem(b'remotefilelog', b'excludepattern', default=None) configitem(b'remotefilelog', b'excludepattern', default=None)

hgext/remotefilelog/shallowbundle.py

def shouldaddfilegroups(self, source): def shouldaddfilegroups(self, source):
repo = self._repo repo = self._repo
if not shallowutil.isenabled(repo): if not shallowutil.isenabled(repo):
return AllFiles return AllFiles
if source == b"push" or source == b"bundle": if source == b"push" or source == b"bundle":
return AllFiles return AllFiles
# We won't actually strip the files, but we should put them in any
# backup bundle generated by strip (especially for cases like narrow's
# `hg tracked --removeinclude`, as failing to do so means that the
# "saved" changesets during a strip won't have their files reapplied and
# thus their linknode adjusted, if necessary).
if source == b"strip":
cfg = repo.ui.config(b'remotefilelog', b'strip.includefiles')
if cfg == b'local':
return LocalFiles
elif cfg != b'none':
return AllFiles
caps = self._bundlecaps or [] caps = self._bundlecaps or []
if source == b"serve" or source == b"pull": if source == b"serve" or source == b"pull":
if constants.BUNDLE2_CAPABLITY in caps: if constants.BUNDLE2_CAPABLITY in caps:
return LocalFiles return LocalFiles
else: else:
# Serving to a full repo requires us to serve everything # Serving to a full repo requires us to serve everything
repo.ui.warn(_(b"pulling from a shallow repo\n")) repo.ui.warn(_(b"pulling from a shallow repo\n"))
return AllFiles return AllFiles

tests/test-remotefilelog-bgprefetch.t

$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> pullprefetch=bookmark() > pullprefetch=bookmark()
> backgroundprefetch=True > backgroundprefetch=True
> EOF > EOF
$ hg strip tip $ hg strip tip
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob)
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ clearcache $ clearcache
$ hg pull $ hg pull
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes

tests/test-remotefilelog-bundles.t

1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ cd shallow $ cd shallow
Unbundling a shallow bundle Unbundling a shallow bundle
$ hg strip -r 66ee28d0328c $ hg strip -r 66ee28d0328c
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob) 2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
$ hg unbundle .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg $ hg unbundle .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 2 changesets with 0 changes to 0 files added 2 changesets with 2 changes to 1 files
new changesets 66ee28d0328c:16db62c5946f new changesets 66ee28d0328c:16db62c5946f
(run 'hg update' to get a working copy) (run 'hg update' to get a working copy)
Unbundling a full bundle Unbundling a full bundle
$ hg -R ../master bundle -r 66ee28d0328c:: --base "66ee28d0328c^" ../fullbundle.hg $ hg -R ../master bundle -r 66ee28d0328c:: --base "66ee28d0328c^" ../fullbundle.hg
2 changesets found 2 changesets found
$ hg strip -r 66ee28d0328c $ hg strip -r 66ee28d0328c
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
$ hg unbundle ../fullbundle.hg $ hg unbundle ../fullbundle.hg
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 2 changesets with 2 changes to 1 files added 2 changesets with 2 changes to 1 files
new changesets 66ee28d0328c:16db62c5946f (2 drafts) new changesets 66ee28d0328c:16db62c5946f (2 drafts)
(run 'hg update' to get a working copy) (run 'hg update' to get a working copy)
Pulling from a shallow bundle Pulling from a shallow bundle
$ hg strip -r 66ee28d0328c $ hg strip -r 66ee28d0328c --config remotefilelog.strip.includefiles=none
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
$ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg $ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
pulling from .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg pulling from .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 1 changesets with 0 changes to 0 files added 1 changesets with 0 changes to 0 files
new changesets 66ee28d0328c (1 drafts) new changesets 66ee28d0328c (1 drafts)
(run 'hg update' to get a working copy) (run 'hg update' to get a working copy)
Pulling from a full bundle Pulling from a full bundle, also testing that strip produces a full bundle by
default.
$ hg strip -r 66ee28d0328c $ hg strip -r 66ee28d0328c
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg (glob)
$ hg pull -r 66ee28d0328c ../fullbundle.hg $ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg
pulling from ../fullbundle.hg pulling from .hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg
searching for changes searching for changes
abort: cannot pull from full bundles abort: cannot pull from full bundles
(use `hg unbundle` instead) (use `hg unbundle` instead)
[255] [255]

tests/test-remotefilelog-local.t

# strip # strip
$ clearcache $ clearcache
$ hg debugrebuilddirstate # fixes dirstate non-determinism $ hg debugrebuilddirstate # fixes dirstate non-determinism
$ hg strip -r . $ hg strip -r .
2 files updated, 0 files merged, 1 files removed, 0 files unresolved 2 files updated, 0 files merged, 1 files removed, 0 files unresolved
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg (glob)
4 files fetched over 2 fetches - (4 misses, 0.00% hit ratio) over *s (glob) 3 files fetched over 2 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
# unbundle # unbundle
$ clearcache $ clearcache
$ ls -A $ ls -A
.hg .hg
w w
x x
y y
z z
$ hg debugrebuilddirstate # fixes dirstate non-determinism $ hg debugrebuilddirstate # fixes dirstate non-determinism
$ hg unbundle .hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg $ hg unbundle .hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 1 changesets with 0 changes to 0 files added 1 changesets with 3 changes to 3 files
new changesets 19edf50f4de7 (1 drafts) new changesets 19edf50f4de7 (1 drafts)
(run 'hg update' to get a working copy) (run 'hg update' to get a working copy)
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
$ hg up $ hg up
3 files updated, 0 files merged, 0 files removed, 0 files unresolved 3 files updated, 0 files merged, 0 files removed, 0 files unresolved
4 files fetched over 1 fetches - (4 misses, 0.00% hit ratio) over *s (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ cat a $ cat a
a a
# revert # revert
$ clearcache $ clearcache
$ hg revert -r .~2 y z $ hg revert -r .~2 y z
no changes needed to z no changes needed to z
2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ hg checkout -C -r . -q $ hg checkout -C -r . -q
# explicit bundle should produce full bundle file # explicit bundle should produce full bundle file
$ hg bundle -r 2 --base 1 ../local.bundle $ hg bundle -r 2 --base 1 ../local.bundle
1 changesets found 1 changesets found
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ cd .. $ cd ..
$ hgcloneshallow ssh://user@dummy/master shallow2 -q $ hgcloneshallow ssh://user@dummy/master shallow2 -q
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob) 2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
$ cd shallow2 $ cd shallow2
$ hg unbundle ../local.bundle $ hg unbundle ../local.bundle
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 1 changesets with 3 changes to 3 files added 1 changesets with 3 changes to 3 files
new changesets 19edf50f4de7 (1 drafts) new changesets 19edf50f4de7 (1 drafts)
(run 'hg update' to get a working copy) (run 'hg update' to get a working copy)

tests/test-remotefilelog-prefetch.t

y y
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
# prefetch on pull when configured # prefetch on pull when configured
$ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc $ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc
$ hg strip tip $ hg strip tip
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/109c3a557a73-3f43405e-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/109c3a557a73-3f43405e-backup.hg (glob)
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ clearcache $ clearcache
$ hg pull $ hg pull
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes

tests/test-remotefilelog-sparse.t

$ hg cat -r 0 z $ hg cat -r 0 z
z z
# prefetch sparse only on pull when configured # prefetch sparse only on pull when configured
$ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc $ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc
$ hg strip tip $ hg strip tip
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/876b1317060d-b2e91d8d-backup.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/876b1317060d-b2e91d8d-backup.hg (glob)
2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
$ hg debugsparse --delete z $ hg debugsparse --delete z
$ clearcache $ clearcache
$ hg pull $ hg pull
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
adding changesets adding changesets

tests/test-remotefilelog-strip.t

0 files updated, 0 files merged, 1 files removed, 0 files unresolved 0 files updated, 0 files merged, 1 files removed, 0 files unresolved
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/df91f74b871e-c94d67be-backup.hg saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/df91f74b871e-c94d67be-backup.hg
$ hg --hidden log -G -T '{rev} {node|short}' $ hg --hidden log -G -T '{rev} {node|short}'
o 1 70494d7ec5ef o 1 70494d7ec5ef
| |
@ 0 b292c1e3311f @ 0 b292c1e3311f
FIXME: This should point to a commit that actually exists in the repo. Otherwise Demonstrate that the linknode points to a commit that is actually in the repo
remotefilelog has to search every commit in the repository looking for a valid after the strip operation. Otherwise remotefilelog has to search every commit in
linkrev every time it's queried, such as during push. the repository looking for a valid linkrev every time it's queried, such as
during push.
$ hg debug-file-linknode -r 70494d a $ hg debug-file-linknode -r 70494d a
df91f74b871e064c89afa1fe9e2f66afa2c125df 70494d7ec5ef6cd3cd6939a9fd2812f9956bf553