This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8410

nodemap: also use persistent nodemap for manifest
ClosedPublic

Authored by marmoute on Apr 14 2020, 11:48 AM.

Download Raw Diff

Details

Reviewers

Alphare

Group Reviewers

hg-reviewers

Commits

rHG640d5b3bd060: nodemap: also use persistent nodemap for manifest

Summary

The manifest as a different usage pattern than the changelog. First, while the
lookup in changelog are not garanteed to match, the lookup in the manifest
nodemap come from changelog and will exist in the manifest. In addition, looking
up a manifest almost always result in unpacking a manifest an operation that
rarely come cheap.

Nevertheless, using a persistent nodemap provide a significant gain for some
operations.

For our measurementw, we use hg cat --rev REV FILE on the our reference
mozilla-try. On this repository the persistent nodemap cache is about 29 MB in
side for a total store side of 11,988 MB

File with large history (file: b2g/config/gaia.json, revision: 195a1146daa0)

no optimisation:                             0.358s
using mmap for index:                        0.297s (-0.061s)
persistent nodemap for changelog only:       0.275s (-0.024s)
persistent nodemap for manifest too:         0.258s (-0.017s)

File with small history (file: .hgignore, revision: 195a1146daa0)

no optimisation:                             0.377s
using mmap for index:                        0.296s (-0.061s)
persistent nodemap for changelog only:       0.274s (-0.022s)
persistent nodemap for manifest too:         0.257s (-0.017s)

Same file but using a revision (8ba995b74e18) with a smaller manifest (3944829
bytes vs 10 bytes)

no optimisation:                             0.192s (-0.185s)
using mmap for index:                        0.131s (-0.061s)
persistent nodemap for changelog only:       0.106s (-0.025s)
persistent nodemap for manifest too:         0.087s (-0.019s)

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Apr 14 2020, 11:48 AM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptApr 14 2020, 11:48 AM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

marmoute added a child revision: D8411: nodemap: also warm manifest nodemap with other caches.Apr 14 2020, 11:48 AM

marmoute updated this revision to Diff 21090.Apr 15 2020, 7:59 AM

marmoute updated this revision to Diff 21145.Apr 16 2020, 1:13 PM

durin42 mentioned this in D8409: nodemap: create files in the repository used in the test.Apr 16 2020, 1:37 PM

Herald added a subscriber: mercurial-patches. · View Herald TranscriptApr 27 2020, 1:12 PM

Alphare accepted this revision.May 7 2020, 8:52 AM

marmoute added a commit: rHG640d5b3bd060: nodemap: also use persistent nodemap for manifest.May 7 2020, 11:33 AM

This revision was not accepted when it landed; it landed in state Needs Review.

Closed by commit rHG640d5b3bd060: nodemap: also use persistent nodemap for manifest (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/manifest.py (3 lines)
M			tests/test-persistent-nodemap.t (3 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	21058		Apr 14 2020, 11:48 AM	★	★
Diff 2	21090		Apr 15 2020, 7:59 AM	★	★
Diff 3	21145		Apr 16 2020, 1:13 PM	★	★
Diff 4	21288	rHG640d5b3bd0603c532e32cef13ae0119f1feeb42a	Apr 5 2020, 7:12 AM	★	★

Status	Author	Revision
Closed	marmoute	D8431 upgrade: support upgrade and downgrade from persistent nodemap
Closed	marmoute	D8430 nodemap: teach `hg debugformat` about the persistent nodemap option
Closed	marmoute	D8429 upgrade: support the --quiet flag
Closed	marmoute	D8428 upgrade: clearly list optimisation
Closed	marmoute	D8422 nodemap: move the mode option to storage.revlog.nodemap.mode
Closed	marmoute	D8421 nodemap: move the option for mmap usage to storage.revlog.nodemap.mmap
Closed	marmoute	D8420 nodemap: move and update the commend about persistence being experimental
Closed	marmoute	D8419 nodemap: move the main switch to the `format` section
Closed	marmoute	D8418 nodemap: drop the 'exp-' prefix for internal opener option
Closed	marmoute	D8417 nodemap: gate the feature behind a new requirement
Closed	marmoute	D8416 nodemap: move on disk file to version 1
Closed	marmoute	D8415 nodemap: add a new mode value, "strict"
Closed	marmoute	D8414 nodemap: add a new mode option, with an optional "warn" value
Closed	marmoute	D8413 nodemap: access the mmap opener option using bytes
Closed	marmoute	D8412 nodemap: rename a variable to prevent shawoding "_"
Closed	marmoute	D8411 nodemap: also warm manifest nodemap with other caches
Closed	marmoute	D8410 nodemap: also use persistent nodemap for manifest
Closed	marmoute	D8409 nodemap: create files in the repository used in the test
Closed	marmoute	D8427 upgrade: properly filter action depending on planned work
Closed	marmoute	D8408 nodemap: skip persistent nodemap warming for revlog not using it
Closed	marmoute	D8453 wait-on-file: adjust the timer counter

Diff 21288

mercurial/manifest.py


	self._revlog = revlog.revlog(			self._revlog = revlog.revlog(
	opener,			opener,
	indexfile,			indexfile,
	# only root indexfile is cached			# only root indexfile is cached
	checkambig=not bool(tree),			checkambig=not bool(tree),
	mmaplargeindex=True,			mmaplargeindex=True,
	upperboundcomp=MAXCOMPRESSION,			upperboundcomp=MAXCOMPRESSION,
				persistentnodemap=opener.options.get(
				b'exp-persistent-nodemap', False
				),
	)			)

	self.index = self._revlog.index			self.index = self._revlog.index
	self.version = self._revlog.version			self.version = self._revlog.version
	self._generaldelta = self._revlog._generaldelta			self._generaldelta = self._revlog._generaldelta

	def _setupmanifestcachehooks(self, repo):			def _setupmanifestcachehooks(self, repo):
	"""Persist the manifestfulltextcache on lock release"""			"""Persist the manifestfulltextcache on lock release"""

tests/test-persistent-nodemap.t

	$ hg log -r "$ANYNODE" --template '{rev}\n'			$ hg log -r "$ANYNODE" --template '{rev}\n'
	5000			5000


	#if rust			#if rust

	$ f --sha256 .hg/store/00changelog-*.nd			$ f --sha256 .hg/store/00changelog-*.nd
	.hg/store/00changelog-????????????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob)			.hg/store/00changelog-????????????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob)

				$ f --sha256 .hg/store/00manifest-*.nd
				.hg/store/00manifest-????????????????.nd: sha256=97117b1c064ea2f86664a124589e47db0e254e8d34739b5c5cc5bf31c9da2b51 (glob)
	$ hg debugnodemap --dump-new \| f --sha256 --size			$ hg debugnodemap --dump-new \| f --sha256 --size
	size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd			size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
	$ hg debugnodemap --dump-disk \| f --sha256 --bytes=256 --hexdump --size			$ hg debugnodemap --dump-disk \| f --sha256 --bytes=256 --hexdump --size
	size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd			size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
	0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 \|....... ........\|			0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 \|....... ........\|
	0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 \|...f..........."\|			0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 \|...f..........."\|
	0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e \|...#...........^\|			0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e \|...#...........^\|
	0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab \|.......N...e....\|			0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab \|.......N...e....\|