This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1255

repack: add blob size to metadata if it's missing
ClosedPublic

Authored by phillco on Oct 26 2017, 7:57 PM.

Download Raw Diff

Details

Reviewers

durham

Group Reviewers

Restricted Project

Commits

rFBHGXbd434ad44ff6: repack: add blob size to metadata if it's missing

Summary

When trees are fetched from the server as packs, metadata isn't included, as
it's not supported in the protocol. Fast size information is useful, since
we have access to the fulltext during a repack, add the metadatan then.

This will be needed for sized-based sorting of manifest entries.

Diff Detail

Repository

rFBHGX Facebook Mercurial Extensions

Lint

Lint Skipped

Unit

Unit Tests Skipped

Event Timeline

phillco created this revision.Oct 26 2017, 7:57 PM

Herald added a reviewer: Restricted Project. · View Herald TranscriptOct 26 2017, 7:57 PM

phillco updated this revision to Diff 3133.Oct 26 2017, 8:03 PM

phillco edited the summary of this revision. (Show Details)Oct 26 2017, 8:05 PM

phillco edited the summary of this revision. (Show Details)

phillco edited the summary of this revision. (Show Details)Oct 27 2017, 6:05 PM

phillco added a child revision: D1259: debugdatapack: print delta and blob size totals and compression %.

phillco edited the summary of this revision. (Show Details)Oct 30 2017, 11:49 PM

phillco added a reviewer: durham.

phillco updated this revision to Diff 3170.Oct 31 2017, 2:02 AM

durham accepted this revision.Oct 31 2017, 3:56 PM

This revision is now accepted and ready to land.Oct 31 2017, 3:56 PM

landed

Revision Contents
Changeset List

			Path	Packages
M			remotefilelog/repack.py (4 lines)
M			tests/test-remotefilelog-repack-fast.t (6 lines)
M			tests/test-remotefilelog-repack.t (16 lines)
M			tests/test-treemanifest-prefetch.t (116 lines)
M			tests/test-treemanifest-repack.t (16 lines)

Status	Author	Revision
Closed	phillco	D1272 repack: sort orphan nodes by size
Closed	phillco	D1271 debugdatapack: return 1 and warn if there are invalid entries
Closed	phillco	D1259 debugdatapack: print delta and blob size totals and compression %
Closed	phillco	D1255 repack: add blob size to metadata if it's missing
Closed	phillco	D1254 debugdatapack: add blob size

Diff 3170

remotefilelog/repack.py

	# Compute delta			# Compute delta
	# TODO: Optimize the deltachain fetching. Since we're			# TODO: Optimize the deltachain fetching. Since we're
	# iterating over the different version of the file, we may			# iterating over the different version of the file, we may
	# be fetching the same deltachain over and over again.			# be fetching the same deltachain over and over again.
	# TODO: reuse existing deltas if it matches our deltabase			# TODO: reuse existing deltas if it matches our deltabase
	if deltabase != nullid:			if deltabase != nullid:
	deltabasetext = self.data.get(filename, deltabase)			deltabasetext = self.data.get(filename, deltabase)
	original = self.data.get(filename, node)			original = self.data.get(filename, node)
				size = len(original)
	delta = mdiff.textdiff(deltabasetext, original)			delta = mdiff.textdiff(deltabasetext, original)
	else:			else:
	delta = self.data.get(filename, node)			delta = self.data.get(filename, node)
				size = len(delta)

	# TODO: don't use the delta if it's larger than the fulltext			# TODO: don't use the delta if it's larger than the fulltext
	# TODO: don't use the delta if the chain is already long			# TODO: don't use the delta if the chain is already long
	meta = self.data.getmeta(filename, node)			meta = self.data.getmeta(filename, node)
				if constants.METAKEYSIZE not in meta:
				meta[constants.METAKEYSIZE] = size
	target.add(filename, node, deltabase, delta, meta)			target.add(filename, node, deltabase, delta, meta)

	entries[node].datarepacked = True			entries[node].datarepacked = True

	ui.progress(_("processing nodes"), None)			ui.progress(_("processing nodes"), None)
	count += 1			count += 1

	ui.progress(_("repacking data"), None)			ui.progress(_("repacking data"), None)

tests/test-remotefilelog-repack-fast.t

	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack			-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
	-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack			-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
	-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack			-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
	$ hg repack --incremental			$ hg repack --incremental
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack
	-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack			-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
	-r--r--r-- 213 f3c56163b762b2931865bbbb1250b4fae09e782c.datapack			-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack

	1 gen3 pack, 1 gen0 pack - does nothing			1 gen3 pack, 1 gen0 pack - does nothing
	$ hg repack --incremental			$ hg repack --incremental
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack
	-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack			-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
	-r--r--r-- 213 f3c56163b762b2931865bbbb1250b4fae09e782c.datapack			-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack

	Pull should run background repack			Pull should run background repack
	$ cat >> .hg/hgrc <<EOF			$ cat >> .hg/hgrc <<EOF
	> [remotefilelog]			> [remotefilelog]
	> backgroundrepack=True			> backgroundrepack=True
	> EOF			> EOF
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	no changes found			no changes found
	(running background incremental repack)			(running background incremental repack)
	$ sleep 0.5			$ sleep 0.5
	$ hg debugwaitonrepack >/dev/null 2>&1			$ hg debugwaitonrepack >/dev/null 2>&1
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack
	-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack			-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
	-r--r--r-- 213 f3c56163b762b2931865bbbb1250b4fae09e782c.datapack			-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack

	Test environment variable resolution			Test environment variable resolution
	$ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'			$ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'
	1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob)			1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob)
	$ find $TESTTMP/envcache \| sort			$ find $TESTTMP/envcache \| sort
	$TESTTMP/envcache			$TESTTMP/envcache

tests/test-remotefilelog-repack.t

	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack			-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
	-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack			-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
	-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack			-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
	$ hg repack --incremental			$ hg repack --incremental
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack
	-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack			-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
	-r--r--r-- 213 f3c56163b762b2931865bbbb1250b4fae09e782c.datapack			-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack

	1 gen3 pack, 1 gen0 pack - does nothing			1 gen3 pack, 1 gen0 pack - does nothing
	$ hg repack --incremental			$ hg repack --incremental
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack
	-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack			-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
	-r--r--r-- 213 f3c56163b762b2931865bbbb1250b4fae09e782c.datapack			-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack

	Pull should run background repack			Pull should run background repack
	$ cat >> .hg/hgrc <<EOF			$ cat >> .hg/hgrc <<EOF
	> [remotefilelog]			> [remotefilelog]
	> backgroundrepack=True			> backgroundrepack=True
	> EOF			> EOF
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	no changes found			no changes found
	(running background incremental repack)			(running background incremental repack)
	$ sleep 0.5			$ sleep 0.5
	$ hg debugwaitonrepack >/dev/null 2>&1			$ hg debugwaitonrepack >/dev/null 2>&1
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep datapack
	-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack			-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
	-r--r--r-- 213 f3c56163b762b2931865bbbb1250b4fae09e782c.datapack			-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
	$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack			$ ls_l $TESTTMP/hgcache/master/packs/ \| grep histpack
	-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack			-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack

	Test environment variable resolution			Test environment variable resolution
	$ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'			$ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'
	1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob)			1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob)
	$ find $TESTTMP/envcache \| sort			$ find $TESTTMP/envcache \| sort
	$TESTTMP/envcache			$TESTTMP/envcache
	1406e7411862 => 000000000000 000000000000 b292c1e3311f			1406e7411862 => 000000000000 000000000000 b292c1e3311f

	Test limiting the max delta chain length			Test limiting the max delta chain length
	$ hg repack --config packs.maxchainlen=1			$ hg repack --config packs.maxchainlen=1
	$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.dataidx			$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.dataidx

	x			x
	Node Delta Base Delta Length Blob Size			Node Delta Base Delta Length Blob Size
	1bb2e6237e03 000000000000 8 (missing)			1bb2e6237e03 000000000000 8 8
	d4a3ed9310e5 1bb2e6237e03 12 (missing)			d4a3ed9310e5 1bb2e6237e03 12 6
	aee31534993a 000000000000 4 (missing)			aee31534993a 000000000000 4 4
	1406e7411862 aee31534993a 12 (missing)			1406e7411862 aee31534993a 12 2

	y			y
	Node Delta Base Delta Length Blob Size			Node Delta Base Delta Length Blob Size
	577959738234 000000000000 8 (missing)			577959738234 000000000000 8 8

tests/test-treemanifest-prefetch.t

	$ ls $CACHEDIR/master/packs/manifests			$ ls $CACHEDIR/master/packs/manifests
	29938257d506f677320d5abec8e34a1a9ed635fe.histidx			29938257d506f677320d5abec8e34a1a9ed635fe.histidx
	29938257d506f677320d5abec8e34a1a9ed635fe.histpack			29938257d506f677320d5abec8e34a1a9ed635fe.histpack
	8adc618d23082c0a5311a4bbf9ac08b9b9672471.dataidx			8adc618d23082c0a5311a4bbf9ac08b9b9672471.dataidx
	8adc618d23082c0a5311a4bbf9ac08b9b9672471.datapack			8adc618d23082c0a5311a4bbf9ac08b9b9672471.datapack
	$ hg debugdatapack --long $CACHEDIR/master/packs/manifests/*.dataidx			$ hg debugdatapack --long $CACHEDIR/master/packs/manifests/*.dataidx

	subdir			subdir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	ddb35f099a648a43a997aef53123bce309c794fd 0000000000000000000000000000000000000000 43			ddb35f099a648a43a997aef53123bce309c794fd 0000000000000000000000000000000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	1be4ab2126dd2252dcae6be2aac2561dd3ddcda0 0000000000000000000000000000000000000000 95			1be4ab2126dd2252dcae6be2aac2561dd3ddcda0 0000000000000000000000000000000000000000 95 (missing)

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	a18d21674e76d6aab2edb46810b20fbdbd10fb4b 0000000000000000000000000000000000000000 43			a18d21674e76d6aab2edb46810b20fbdbd10fb4b 0000000000000000000000000000000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5 0000000000000000000000000000000000000000 95			60a7f7acb6bb5aaf93ca7d9062931b0f6a0d6db5 0000000000000000000000000000000000000000 95 (missing)

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	bc0c2c938b929f98b1c31a8c5994396ebb096bf0 0000000000000000000000000000000000000000 43			bc0c2c938b929f98b1c31a8c5994396ebb096bf0 0000000000000000000000000000000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	ef362f8bbe8aa457b0cfc49f200cbeb7747984ed 0000000000000000000000000000000000000000 46			ef362f8bbe8aa457b0cfc49f200cbeb7747984ed 0000000000000000000000000000000000000000 46 (missing)
	$ hg debughistorypack $CACHEDIR/master/packs/manifests/*.histidx			$ hg debughistorypack $CACHEDIR/master/packs/manifests/*.histidx


	Node P1 Node P2 Node Link Node Copy From			Node P1 Node P2 Node Link Node Copy From
	60a7f7acb6bb 1be4ab2126dd 000000000000 bd6f9b289c01			60a7f7acb6bb 1be4ab2126dd 000000000000 bd6f9b289c01
	1be4ab2126dd ef362f8bbe8a 000000000000 f15c65c6e9bd			1be4ab2126dd ef362f8bbe8a 000000000000 f15c65c6e9bd
	ef362f8bbe8a 000000000000 000000000000 ecfb693caff5			ef362f8bbe8a 000000000000 000000000000 ecfb693caff5

	$ hg prefetchtrees -r '2' --base '1'			$ hg prefetchtrees -r '2' --base '1'
	2 trees fetched over * (glob)			2 trees fetched over * (glob)
	$ ls $CACHEDIR/master/packs/manifests/*.dataidx			$ ls $CACHEDIR/master/packs/manifests/*.dataidx
	$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx			$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx

	$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx			$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	a18d21674e76 000000000000 43			a18d21674e76 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	60a7f7acb6bb 000000000000 95			60a7f7acb6bb 000000000000 95 (missing)

	Test auto prefetch during normal access			Test auto prefetch during normal access
	$ rm -rf $CACHEDIR/master			$ rm -rf $CACHEDIR/master
	\|\| ( exit 1 ) is needed because ls on OSX and Linux exits differently			\|\| ( exit 1 ) is needed because ls on OSX and Linux exits differently
	$ ls $CACHEDIR/master/packs/manifests \|\| ( exit 1 )			$ ls $CACHEDIR/master/packs/manifests \|\| ( exit 1 )
	*No such file or directory (glob)			*No such file or directory (glob)
	[1]			[1]
	$ hg log -r tip --stat --pager=off			$ hg log -r tip --stat --pager=off
	524ab81400c6bc8449e3e720d81f836ebacec539.histidx			524ab81400c6bc8449e3e720d81f836ebacec539.histidx
	524ab81400c6bc8449e3e720d81f836ebacec539.histpack			524ab81400c6bc8449e3e720d81f836ebacec539.histpack
	e5c44a5c1bbfd8841df1c6c4b7cca54536e016db.histidx			e5c44a5c1bbfd8841df1c6c4b7cca54536e016db.histidx
	e5c44a5c1bbfd8841df1c6c4b7cca54536e016db.histpack			e5c44a5c1bbfd8841df1c6c4b7cca54536e016db.histpack

	$ hg debugdatapack $CACHEDIR/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675			$ hg debugdatapack $CACHEDIR/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	bc0c2c938b92 000000000000 43			bc0c2c938b92 000000000000 43 (missing)

	subdir			subdir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	ddb35f099a64 000000000000 43			ddb35f099a64 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	1be4ab2126dd 000000000000 95			1be4ab2126dd 000000000000 95 (missing)

	- Note that subdir/ is not downloaded again			- Note that subdir/ is not downloaded again
	$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c			$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	a18d21674e76 000000000000 43			a18d21674e76 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	60a7f7acb6bb 000000000000 95			60a7f7acb6bb 000000000000 95 (missing)

	Test that auto prefetch scans up the changelog for base trees			Test that auto prefetch scans up the changelog for base trees
	$ rm -rf $CACHEDIR/master			$ rm -rf $CACHEDIR/master
	$ hg prefetchtrees -r 'tip^'			$ hg prefetchtrees -r 'tip^'
	3 trees fetched over * (glob)			3 trees fetched over * (glob)
	$ rm -rf $CACHEDIR/master			$ rm -rf $CACHEDIR/master
	$ hg prefetchtrees -r tip			$ hg prefetchtrees -r tip
	3 trees fetched over * (glob)			3 trees fetched over * (glob)
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	no changes found			no changes found
	prefetching trees			prefetching trees
	6 trees fetched over * (glob)			6 trees fetched over * (glob)
	$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx			$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx

	subdir			subdir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	ddb35f099a64 000000000000 43			ddb35f099a64 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	1be4ab2126dd 000000000000 95			1be4ab2126dd 000000000000 95 (missing)

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	a18d21674e76 000000000000 43			a18d21674e76 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	60a7f7acb6bb 000000000000 95			60a7f7acb6bb 000000000000 95 (missing)

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	bc0c2c938b92 000000000000 43			bc0c2c938b92 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	ef362f8bbe8a 000000000000 46			ef362f8bbe8a 000000000000 46 (missing)

	$ hg strip -q -r 'draft()'			$ hg strip -q -r 'draft()'

	- Prefetch just the top manifest (but the full one)			- Prefetch just the top manifest (but the full one)
	$ rm -rf $CACHEDIR/master			$ rm -rf $CACHEDIR/master
	$ hg pull --config treemanifest.pullprefetchcount=1 --traceback			$ hg pull --config treemanifest.pullprefetchcount=1 --traceback
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	no changes found			no changes found
	prefetching trees			prefetching trees
	3 trees fetched over * (glob)			3 trees fetched over * (glob)
	$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx			$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	a18d21674e76 000000000000 43			a18d21674e76 000000000000 43 (missing)

	subdir			subdir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	ddb35f099a64 000000000000 43			ddb35f099a64 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	60a7f7acb6bb 000000000000 95			60a7f7acb6bb 000000000000 95 (missing)

	- Prefetch commit 1 then minimally prefetch commit 2			- Prefetch commit 1 then minimally prefetch commit 2
	$ rm -rf $CACHEDIR/master			$ rm -rf $CACHEDIR/master
	$ hg prefetchtrees -r 1			$ hg prefetchtrees -r 1
	3 trees fetched over * (glob)			3 trees fetched over * (glob)
	$ ls $CACHEDIR/master/packs/manifests/*dataidx			$ ls $CACHEDIR/master/packs/manifests/*dataidx
	$TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx			$TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx
	$ hg pull --config treemanifest.pullprefetchcount=1 --traceback			$ hg pull --config treemanifest.pullprefetchcount=1 --traceback
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	no changes found			no changes found
	prefetching trees			prefetching trees
	2 trees fetched over * (glob)			2 trees fetched over * (glob)
	$ ls $CACHEDIR/master/packs/manifests/*dataidx			$ ls $CACHEDIR/master/packs/manifests/*dataidx
	$TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx			$TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx
	$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx			$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx
	$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx			$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	a18d21674e76 000000000000 43			a18d21674e76 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	60a7f7acb6bb 000000000000 95			60a7f7acb6bb 000000000000 95 (missing)

	Test prefetching certain revs during pull			Test prefetching certain revs during pull
	$ cd ../master			$ cd ../master
	$ echo x >> dir/x			$ echo x >> dir/x
	$ hg commit -qm "modify dir/x a third time"			$ hg commit -qm "modify dir/x a third time"
	$ echo x >> dir/x			$ echo x >> dir/x
	$ hg commit -qm "modify dir/x a fourth time"			$ hg commit -qm "modify dir/x a fourth time"

	$ cd ../client			$ cd ../client
	$ rm -rf $CACHEDIR/master			$ rm -rf $CACHEDIR/master
	$ hg pull --config treemanifest.pullprefetchrevs='tip~2'			$ hg pull --config treemanifest.pullprefetchrevs='tip~2'
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes
	added 2 changesets with 2 changes to 1 files			added 2 changesets with 2 changes to 1 files
	new changesets dece825f8add:cfacdcc4cee5			new changesets dece825f8add:cfacdcc4cee5
	(run 'hg update' to get a working copy)			(run 'hg update' to get a working copy)
	prefetching trees			prefetching trees
	3 trees fetched over * (glob)			3 trees fetched over * (glob)
	$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx			$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	a18d21674e76 000000000000 43			a18d21674e76 000000000000 43 (missing)

	subdir			subdir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	ddb35f099a64 000000000000 43			ddb35f099a64 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	60a7f7acb6bb 000000000000 95			60a7f7acb6bb 000000000000 95 (missing)

	- Test prefetching only the new tree parts for a commit who's parent tree is not			- Test prefetching only the new tree parts for a commit who's parent tree is not
	- downloaded already. Note that subdir/z was not downloaded this time.			- downloaded already. Note that subdir/z was not downloaded this time.
	$ hg pull --config treemanifest.pullprefetchrevs='tip'			$ hg pull --config treemanifest.pullprefetchrevs='tip'
	pulling from ssh://user@dummy/master			pulling from ssh://user@dummy/master
	searching for changes			searching for changes
	no changes found			no changes found
	prefetching trees			prefetching trees
	2 trees fetched over * (glob)			2 trees fetched over * (glob)
	$ hg debugdatapack $CACHEDIR/master/packs/manifests/99050e724a9236121684523ba3f4db270e62fb58.dataidx			$ hg debugdatapack $CACHEDIR/master/packs/manifests/99050e724a9236121684523ba3f4db270e62fb58.dataidx

	dir			dir
	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	bf22bc15398b 000000000000 43			bf22bc15398b 000000000000 43 (missing)


	Node Delta Base Delta Length			Node Delta Base Delta Length Blob Size
	aa52a49be522 000000000000 95			aa52a49be522 000000000000 95 (missing)

	Test that prefetch refills just part of a tree when the cache is deleted			Test that prefetch refills just part of a tree when the cache is deleted

	$ echo >> dir/x			$ echo >> dir/x
	$ hg commit -m 'edit x locally'			$ hg commit -m 'edit x locally'
	created new head			created new head
	$ rm -rf $CACHEDIR/master/*			$ rm -rf $CACHEDIR/master/*
	$ hg cat subdir/z			$ hg cat subdir/z

tests/test-treemanifest-repack.t

	dir			dir
	Node P1 Node P2 Node Link Node Copy From			Node P1 Node P2 Node Link Node Copy From
	23226e7a252c 000000000000 000000000000 8e83608cbe60			23226e7a252c 000000000000 000000000000 8e83608cbe60

	- Repack and reverify			- Repack and reverify
	$ hg repack			$ hg repack

	$ ls_l $CACHEDIR/master/packs/manifests \| grep pack			$ ls_l $CACHEDIR/master/packs/manifests \| grep pack
				-r--r--r-- 339 56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.datapack
	-r--r--r-- 262 7535b6084226436bbdff33043969e7fa963e8428.histpack			-r--r--r-- 262 7535b6084226436bbdff33043969e7fa963e8428.histpack
	-r--r--r-- 327 be1a1ae9828d9bd09ab8ba765db358f99fd09ea7.datapack

	$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.datapack			$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.datapack


	Node Delta Base Delta Length Blob Size			Node Delta Base Delta Length Blob Size
	1832e0765de9 000000000000 89 (missing)			1832e0765de9 000000000000 89 89
	a0c8bcbbb45c 1832e0765de9 12 (missing)			a0c8bcbbb45c 1832e0765de9 12 43

	dir			dir
	Node Delta Base Delta Length Blob Size			Node Delta Base Delta Length Blob Size
	23226e7a252c 000000000000 43 (missing)			23226e7a252c 000000000000 43 43

	$ hg debughistorypack $CACHEDIR/master/packs/manifests/*.histpack			$ hg debughistorypack $CACHEDIR/master/packs/manifests/*.histpack


	Node P1 Node P2 Node Link Node Copy From			Node P1 Node P2 Node Link Node Copy From
	1832e0765de9 a0c8bcbbb45c 000000000000 8e83608cbe60			1832e0765de9 a0c8bcbbb45c 000000000000 8e83608cbe60
	a0c8bcbbb45c 000000000000 000000000000 1f0dee641bb7			a0c8bcbbb45c 000000000000 000000000000 1f0dee641bb7

	dir			dir
	Node P1 Node P2 Node Link Node Copy From			Node P1 Node P2 Node Link Node Copy From
	23226e7a252c 000000000000 000000000000 8e83608cbe60			23226e7a252c 000000000000 000000000000 8e83608cbe60

	# Test repacking local manifest packs			# Test repacking local manifest packs
	$ hg up -q 1			$ hg up -q 1
	$ echo a >> a && hg commit -Aqm 'modify a'			$ echo a >> a && hg commit -Aqm 'modify a'
	$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'			$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
	$ ls_l .hg/store/packs/manifests \| grep datapack			$ ls_l .hg/store/packs/manifests \| grep datapack
	-r--r--r-- 248 5d1716bbef6e7200192de6509055d1ee31a4172c.datapack			-r--r--r-- 248 5d1716bbef6e7200192de6509055d1ee31a4172c.datapack
	-r--r--r-- 146 cffef142da32f3e52c1779490e5d0ddac5f9b82b.datapack			-r--r--r-- 146 cffef142da32f3e52c1779490e5d0ddac5f9b82b.datapack
	$ hg repack			$ hg repack
	$ ls_l .hg/store/packs/manifests \| grep datapack			$ ls_l .hg/store/packs/manifests \| grep datapack
	-r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack			-r--r--r-- 386 d15c09a9a5a13bb689bd9764455a415a20dc885e.datapack

	# Test incremental repacking of trees			# Test incremental repacking of trees
	$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'			$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
	$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'			$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
	$ ls_l .hg/store/packs/manifests \| grep datapack			$ ls_l .hg/store/packs/manifests \| grep datapack
	-r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
	-r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack			-r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack
				-r--r--r-- 386 d15c09a9a5a13bb689bd9764455a415a20dc885e.datapack
	-r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack			-r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack

	- repack incremental does nothing here because there are so few packs			- repack incremental does nothing here because there are so few packs
	$ hg repack --incremental --config remotefilelog.data.generations=300,200 --config remotefilelog.data.repacksizelimit=300			$ hg repack --incremental --config remotefilelog.data.generations=300,200 --config remotefilelog.data.repacksizelimit=300
	$ ls_l .hg/store/packs/manifests \| grep datapack			$ ls_l .hg/store/packs/manifests \| grep datapack
	-r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
	-r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack			-r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack
				-r--r--r-- 386 d15c09a9a5a13bb689bd9764455a415a20dc885e.datapack
	-r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack			-r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack

	$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'			$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
	$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'			$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
	$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'			$ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
	$ ls_l .hg/store/packs/manifests \| grep datapack			$ ls_l .hg/store/packs/manifests \| grep datapack
	-r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
	-r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack			-r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack
	-r--r--r-- 248 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack			-r--r--r-- 248 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack
	-r--r--r-- 248 544a3b46a61732209116ae50847ec333b75e3765.datapack			-r--r--r-- 248 544a3b46a61732209116ae50847ec333b75e3765.datapack
	-r--r--r-- 248 863908ef8149261ab0d891c2344d8e8766c39441.datapack			-r--r--r-- 248 863908ef8149261ab0d891c2344d8e8766c39441.datapack
				-r--r--r-- 386 d15c09a9a5a13bb689bd9764455a415a20dc885e.datapack
	-r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack			-r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack
	$ cd .hg/store/packs/manifests			$ cd .hg/store/packs/manifests
	$ cp d7e689a91ac63385be120a118af9ce8663748f28.datapack x7e689a91ac63385be120a118af9ce8663748f28.datapack			$ cp d7e689a91ac63385be120a118af9ce8663748f28.datapack x7e689a91ac63385be120a118af9ce8663748f28.datapack
	$ cp d7e689a91ac63385be120a118af9ce8663748f28.dataidx x7e689a91ac63385be120a118af9ce8663748f28.dataidx			$ cp d7e689a91ac63385be120a118af9ce8663748f28.dataidx x7e689a91ac63385be120a118af9ce8663748f28.dataidx
	$ cp 21501384df03b8489b366c5218be639fa08830e4.datapack x1501384df03b8489b366c5218be639fa08830e4.datapack			$ cp 21501384df03b8489b366c5218be639fa08830e4.datapack x1501384df03b8489b366c5218be639fa08830e4.datapack
	$ cp 21501384df03b8489b366c5218be639fa08830e4.dataidx x1501384df03b8489b366c5218be639fa08830e4.dataidx			$ cp 21501384df03b8489b366c5218be639fa08830e4.dataidx x1501384df03b8489b366c5218be639fa08830e4.dataidx
	$ cp 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack x47263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack			$ cp 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack x47263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack
	$ cp 347263bf1efbdb5bf7e1d1565b6b504073fb9093.dataidx x47263bf1efbdb5bf7e1d1565b6b504073fb9093.dataidx			$ cp 347263bf1efbdb5bf7e1d1565b6b504073fb9093.dataidx x47263bf1efbdb5bf7e1d1565b6b504073fb9093.dataidx