This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10320

remotefilelog: include file contents in bundles produced during strip
ClosedPublic

Authored by spectral on Apr 6 2021, 8:21 PM.

Download Raw Diff

Details

Reviewers

pulkit

Group Reviewers

hg-reviewers

Commits

rHG47a9527731c3: remotefilelog: include file contents in bundles produced during strip

Summary

hg strip and other things that use repair.strip (such as the narrow
extension's hg tracked --removeinclude) will "save" some commits that have a
higher revision number than the oldest commit we're stripping, but aren't
actually descended from any of the commits that we're stripping. It saves them
in a bundle, and then reapplies them to the repo.

Remotefilelog doesn't generally participate in strip, it doesn't contribute
files to either the backup bundle or the "saved" bundle, and doesn't adjust
linknodes when commits are stripped. This can break things like push, which
rely on the linknodes.

This change makes it so that remotefilelog includes files in these bundles
during strip operations. During reapplication, the files are reapplied from the
bundle, and the linknode is properly updated.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

spectral created this revision.Apr 6 2021, 8:21 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptApr 6 2021, 8:21 PM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

pulkit accepted this revision.Apr 8 2021, 2:57 AM

This revision is now accepted and ready to land.Apr 8 2021, 2:57 AM

spectral added a commit: rHG47a9527731c3: remotefilelog: include file contents in bundles produced during strip.Apr 8 2021, 6:55 AM

Closed by commit rHG47a9527731c3: remotefilelog: include file contents in bundles produced during strip (authored by spectral). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			hgext/remotefilelog/__init__.py (2 lines)
M			hgext/remotefilelog/shallowbundle.py (12 lines)
M			tests/test-remotefilelog-bgprefetch.t (1 line)
M			tests/test-remotefilelog-bundles.t (13 lines)
M			tests/test-remotefilelog-local.t (11 lines)
M			tests/test-remotefilelog-prefetch.t (1 line)
M			tests/test-remotefilelog-sparse.t (1 line)
M			tests/test-remotefilelog-strip.t (9 lines)

	Status	Author	Revision
	Closed	spectral	D10320 remotefilelog: include file contents in bundles produced during strip
	Closed	spectral	D10319 tests: add test-remotefilelog-strip.t to demonstrate an issue with linknodes

Diff 26684

hgext/remotefilelog/init.py

	configitem(b'remotefilelog', b'servercachepath', default=None)			configitem(b'remotefilelog', b'servercachepath', default=None)
	configitem(b"remotefilelog", b"serverexpiration", default=30)			configitem(b"remotefilelog", b"serverexpiration", default=30)
	configitem(b'remotefilelog', b'backgroundrepack', default=False)			configitem(b'remotefilelog', b'backgroundrepack', default=False)
	configitem(b'remotefilelog', b'bgprefetchrevs', default=None)			configitem(b'remotefilelog', b'bgprefetchrevs', default=None)
	configitem(b'remotefilelog', b'pullprefetch', default=None)			configitem(b'remotefilelog', b'pullprefetch', default=None)
	configitem(b'remotefilelog', b'backgroundprefetch', default=False)			configitem(b'remotefilelog', b'backgroundprefetch', default=False)
	configitem(b'remotefilelog', b'prefetchdelay', default=120)			configitem(b'remotefilelog', b'prefetchdelay', default=120)
	configitem(b'remotefilelog', b'prefetchdays', default=14)			configitem(b'remotefilelog', b'prefetchdays', default=14)
				# Other values include 'local' or 'none'. Any unrecognized value is 'all'.
				configitem(b'remotefilelog', b'strip.includefiles', default='all')

	configitem(b'remotefilelog', b'getfilesstep', default=10000)			configitem(b'remotefilelog', b'getfilesstep', default=10000)
	configitem(b'remotefilelog', b'getfilestype', default=b'optimistic')			configitem(b'remotefilelog', b'getfilestype', default=b'optimistic')
	configitem(b'remotefilelog', b'batchsize', configitems.dynamicdefault)			configitem(b'remotefilelog', b'batchsize', configitems.dynamicdefault)
	configitem(b'remotefilelog', b'fetchwarning', default=b'')			configitem(b'remotefilelog', b'fetchwarning', default=b'')

	configitem(b'remotefilelog', b'includepattern', default=None)			configitem(b'remotefilelog', b'includepattern', default=None)
	configitem(b'remotefilelog', b'excludepattern', default=None)			configitem(b'remotefilelog', b'excludepattern', default=None)

hgext/remotefilelog/shallowbundle.py

	def shouldaddfilegroups(self, source):			def shouldaddfilegroups(self, source):
	repo = self._repo			repo = self._repo
	if not shallowutil.isenabled(repo):			if not shallowutil.isenabled(repo):
	return AllFiles			return AllFiles

	if source == b"push" or source == b"bundle":			if source == b"push" or source == b"bundle":
	return AllFiles			return AllFiles

				# We won't actually strip the files, but we should put them in any
				# backup bundle generated by strip (especially for cases like narrow's
				# `hg tracked --removeinclude`, as failing to do so means that the
				# "saved" changesets during a strip won't have their files reapplied and
				# thus their linknode adjusted, if necessary).
				if source == b"strip":
				cfg = repo.ui.config(b'remotefilelog', b'strip.includefiles')
				if cfg == b'local':
				return LocalFiles
				elif cfg != b'none':
				return AllFiles

	caps = self._bundlecaps or []			caps = self._bundlecaps or []
	if source == b"serve" or source == b"pull":			if source == b"serve" or source == b"pull":
	if constants.BUNDLE2_CAPABLITY in caps:			if constants.BUNDLE2_CAPABLITY in caps:
	return LocalFiles			return LocalFiles
	else:			else:
	# Serving to a full repo requires us to serve everything			# Serving to a full repo requires us to serve everything
	repo.ui.warn(_(b"pulling from a shallow repo\n"))			repo.ui.warn(_(b"pulling from a shallow repo\n"))
	return AllFiles			return AllFiles

tests/test-remotefilelog-bgprefetch.t


	$ cat >> .hg/hgrc <<EOF			$ cat >> .hg/hgrc <<EOF
	> [remotefilelog]			> [remotefilelog]
	> pullprefetch=bookmark()			> pullprefetch=bookmark()
	> backgroundprefetch=True			> backgroundprefetch=True
	> EOF			> EOF
	$ hg strip tip			$ hg strip tip
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob)
				1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)

	$ clearcache			$ clearcache
	$ hg pull			$ hg pull
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes

tests/test-remotefilelog-bundles.t

	1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)			1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
	$ cd shallow			$ cd shallow

	Unbundling a shallow bundle			Unbundling a shallow bundle

	$ hg strip -r 66ee28d0328c			$ hg strip -r 66ee28d0328c
	1 files updated, 0 files merged, 0 files removed, 0 files unresolved			1 files updated, 0 files merged, 0 files removed, 0 files unresolved
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
	1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)			2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
	$ hg unbundle .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg			$ hg unbundle .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes
	added 2 changesets with 0 changes to 0 files			added 2 changesets with 2 changes to 1 files
	new changesets 66ee28d0328c:16db62c5946f			new changesets 66ee28d0328c:16db62c5946f
	(run 'hg update' to get a working copy)			(run 'hg update' to get a working copy)

	Unbundling a full bundle			Unbundling a full bundle

	$ hg -R ../master bundle -r 66ee28d0328c:: --base "66ee28d0328c^" ../fullbundle.hg			$ hg -R ../master bundle -r 66ee28d0328c:: --base "66ee28d0328c^" ../fullbundle.hg
	2 changesets found			2 changesets found
	$ hg strip -r 66ee28d0328c			$ hg strip -r 66ee28d0328c
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
	$ hg unbundle ../fullbundle.hg			$ hg unbundle ../fullbundle.hg
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes
	added 2 changesets with 2 changes to 1 files			added 2 changesets with 2 changes to 1 files
	new changesets 66ee28d0328c:16db62c5946f (2 drafts)			new changesets 66ee28d0328c:16db62c5946f (2 drafts)
	(run 'hg update' to get a working copy)			(run 'hg update' to get a working copy)

	Pulling from a shallow bundle			Pulling from a shallow bundle

	$ hg strip -r 66ee28d0328c			$ hg strip -r 66ee28d0328c --config remotefilelog.strip.includefiles=none
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
	$ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg			$ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
	pulling from .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg			pulling from .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
	searching for changes			searching for changes
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes
	added 1 changesets with 0 changes to 0 files			added 1 changesets with 0 changes to 0 files
	new changesets 66ee28d0328c (1 drafts)			new changesets 66ee28d0328c (1 drafts)
	(run 'hg update' to get a working copy)			(run 'hg update' to get a working copy)

	Pulling from a full bundle			Pulling from a full bundle, also testing that strip produces a full bundle by
				default.

	$ hg strip -r 66ee28d0328c			$ hg strip -r 66ee28d0328c
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg (glob)
	$ hg pull -r 66ee28d0328c ../fullbundle.hg			$ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg
	pulling from ../fullbundle.hg			pulling from .hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg
	searching for changes			searching for changes
	abort: cannot pull from full bundles			abort: cannot pull from full bundles
	(use `hg unbundle` instead)			(use `hg unbundle` instead)
	[255]			[255]

tests/test-remotefilelog-local.t


	# strip			# strip

	$ clearcache			$ clearcache
	$ hg debugrebuilddirstate # fixes dirstate non-determinism			$ hg debugrebuilddirstate # fixes dirstate non-determinism
	$ hg strip -r .			$ hg strip -r .
	2 files updated, 0 files merged, 1 files removed, 0 files unresolved			2 files updated, 0 files merged, 1 files removed, 0 files unresolved
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg (glob)
	4 files fetched over 2 fetches - (4 misses, 0.00% hit ratio) over *s (glob)			3 files fetched over 2 fetches - (3 misses, 0.00% hit ratio) over *s (glob)

	# unbundle			# unbundle

	$ clearcache			$ clearcache
	$ ls -A			$ ls -A
	.hg			.hg
	w			w
	x			x
	y			y
	z			z

	$ hg debugrebuilddirstate # fixes dirstate non-determinism			$ hg debugrebuilddirstate # fixes dirstate non-determinism
	$ hg unbundle .hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg			$ hg unbundle .hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes
	added 1 changesets with 0 changes to 0 files			added 1 changesets with 3 changes to 3 files
	new changesets 19edf50f4de7 (1 drafts)			new changesets 19edf50f4de7 (1 drafts)
	(run 'hg update' to get a working copy)			(run 'hg update' to get a working copy)
				2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)

	$ hg up			$ hg up
	3 files updated, 0 files merged, 0 files removed, 0 files unresolved			3 files updated, 0 files merged, 0 files removed, 0 files unresolved
	4 files fetched over 1 fetches - (4 misses, 0.00% hit ratio) over *s (glob)			1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
	$ cat a			$ cat a
	a			a

	# revert			# revert

	$ clearcache			$ clearcache
	$ hg revert -r .~2 y z			$ hg revert -r .~2 y z
	no changes needed to z			no changes needed to z
	2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)			1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
	$ hg checkout -C -r . -q			$ hg checkout -C -r . -q

	# explicit bundle should produce full bundle file			# explicit bundle should produce full bundle file

	$ hg bundle -r 2 --base 1 ../local.bundle			$ hg bundle -r 2 --base 1 ../local.bundle
	1 changesets found			1 changesets found
	1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)			1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
	$ cd ..			$ cd ..

	$ hgcloneshallow ssh://user@dummy/master shallow2 -q			$ hgcloneshallow ssh://user@dummy/master shallow2 -q
	1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)			2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
	$ cd shallow2			$ cd shallow2
	$ hg unbundle ../local.bundle			$ hg unbundle ../local.bundle
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes
	added 1 changesets with 3 changes to 3 files			added 1 changesets with 3 changes to 3 files
	new changesets 19edf50f4de7 (1 drafts)			new changesets 19edf50f4de7 (1 drafts)
	(run 'hg update' to get a working copy)			(run 'hg update' to get a working copy)

tests/test-remotefilelog-prefetch.t

	y			y
	1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)			1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)

	# prefetch on pull when configured			# prefetch on pull when configured

	$ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc			$ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc
	$ hg strip tip			$ hg strip tip
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/109c3a557a73-3f43405e-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/109c3a557a73-3f43405e-backup.hg (glob)
				1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)

	$ clearcache			$ clearcache
	$ hg pull			$ hg pull
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes

tests/test-remotefilelog-sparse.t

	$ hg cat -r 0 z			$ hg cat -r 0 z
	z			z

	# prefetch sparse only on pull when configured			# prefetch sparse only on pull when configured

	$ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc			$ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc
	$ hg strip tip			$ hg strip tip
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/876b1317060d-b2e91d8d-backup.hg (glob)			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/876b1317060d-b2e91d8d-backup.hg (glob)
				2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)

	$ hg debugsparse --delete z			$ hg debugsparse --delete z

	$ clearcache			$ clearcache
	$ hg pull			$ hg pull
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	adding changesets			adding changesets

tests/test-remotefilelog-strip.t

	0 files updated, 0 files merged, 1 files removed, 0 files unresolved			0 files updated, 0 files merged, 1 files removed, 0 files unresolved
	saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/df91f74b871e-c94d67be-backup.hg			saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/df91f74b871e-c94d67be-backup.hg

	$ hg --hidden log -G -T '{rev} {node\|short}'			$ hg --hidden log -G -T '{rev} {node\|short}'
	o 1 70494d7ec5ef			o 1 70494d7ec5ef
	\|			\|
	@ 0 b292c1e3311f			@ 0 b292c1e3311f

	FIXME: This should point to a commit that actually exists in the repo. Otherwise			Demonstrate that the linknode points to a commit that is actually in the repo
	remotefilelog has to search every commit in the repository looking for a valid			after the strip operation. Otherwise remotefilelog has to search every commit in
	linkrev every time it's queried, such as during push.			the repository looking for a valid linkrev every time it's queried, such as
				during push.
	$ hg debug-file-linknode -r 70494d a			$ hg debug-file-linknode -r 70494d a
	df91f74b871e064c89afa1fe9e2f66afa2c125df			70494d7ec5ef6cd3cd6939a9fd2812f9956bf553

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	26669		Apr 6 2021, 8:21 PM	★	★
Diff 2	26684	rHG47a9527731c3c2aca078f949ce0980ea462f3c66	Apr 6 2021, 5:21 PM	★	★