This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8410

nodemap: also use persistent nodemap for manifest
ClosedPublic

Authored by marmoute on Apr 14 2020, 11:48 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG640d5b3bd060: nodemap: also use persistent nodemap for manifest
Summary

The manifest as a different usage pattern than the changelog. First, while the
lookup in changelog are not garanteed to match, the lookup in the manifest
nodemap come from changelog and will exist in the manifest. In addition, looking
up a manifest almost always result in unpacking a manifest an operation that
rarely come cheap.

Nevertheless, using a persistent nodemap provide a significant gain for some
operations.

For our measurementw, we use hg cat --rev REV FILE on the our reference
mozilla-try. On this repository the persistent nodemap cache is about 29 MB in
side for a total store side of 11,988 MB

File with large history (file: b2g/config/gaia.json, revision: 195a1146daa0)

no optimisation:                             0.358s
using mmap for index:                        0.297s (-0.061s)
persistent nodemap for changelog only:       0.275s (-0.024s)
persistent nodemap for manifest too:         0.258s (-0.017s)

File with small history (file: .hgignore, revision: 195a1146daa0)

no optimisation:                             0.377s
using mmap for index:                        0.296s (-0.061s)
persistent nodemap for changelog only:       0.274s (-0.022s)
persistent nodemap for manifest too:         0.257s (-0.017s)

Same file but using a revision (8ba995b74e18) with a smaller manifest (3944829
bytes vs 10 bytes)

no optimisation:                             0.192s (-0.185s)
using mmap for index:                        0.131s (-0.061s)
persistent nodemap for changelog only:       0.106s (-0.025s)
persistent nodemap for manifest too:         0.087s (-0.019s)

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Apr 14 2020, 11:48 AM
marmoute updated this revision to Diff 21090.Apr 15 2020, 7:59 AM
marmoute updated this revision to Diff 21145.Apr 16 2020, 1:13 PM
Alphare accepted this revision.May 7 2020, 8:52 AM
marmoute added a commit: rHG640d5b3bd060: nodemap: also use persistent nodemap for manifest.May 7 2020, 11:33 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG640d5b3bd060: nodemap: also use persistent nodemap for manifest (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 21288

mercurial/manifest.py

self._revlog = revlog.revlog( self._revlog = revlog.revlog(
opener, opener,
indexfile, indexfile,
# only root indexfile is cached # only root indexfile is cached
checkambig=not bool(tree), checkambig=not bool(tree),
mmaplargeindex=True, mmaplargeindex=True,
upperboundcomp=MAXCOMPRESSION, upperboundcomp=MAXCOMPRESSION,
persistentnodemap=opener.options.get(
b'exp-persistent-nodemap', False
),
) )
self.index = self._revlog.index self.index = self._revlog.index
self.version = self._revlog.version self.version = self._revlog.version
self._generaldelta = self._revlog._generaldelta self._generaldelta = self._revlog._generaldelta
def _setupmanifestcachehooks(self, repo): def _setupmanifestcachehooks(self, repo):
"""Persist the manifestfulltextcache on lock release""" """Persist the manifestfulltextcache on lock release"""

tests/test-persistent-nodemap.t

$ hg log -r "$ANYNODE" --template '{rev}\n' $ hg log -r "$ANYNODE" --template '{rev}\n'
5000 5000
#if rust #if rust
$ f --sha256 .hg/store/00changelog-*.nd $ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob) .hg/store/00changelog-????????????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob)
$ f --sha256 .hg/store/00manifest-*.nd
.hg/store/00manifest-????????????????.nd: sha256=97117b1c064ea2f86664a124589e47db0e254e8d34739b5c5cc5bf31c9da2b51 (glob)
$ hg debugnodemap --dump-new | f --sha256 --size $ hg debugnodemap --dump-new | f --sha256 --size
size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 |....... ........| 0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 |....... ........|
0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 |...f..........."| 0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 |...f..........."|
0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e |...#...........^| 0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e |...#...........^|
0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab |.......N...e....| 0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab |.......N...e....|