This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8164

rust-nodemap: automatically use the rust index for persistent nodemap
ClosedPublic

Authored by Alphare on Feb 26 2020, 12:37 PM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Commits
rHGe7fff9c3cdac: rust-nodemap: automatically use the rust index for persistent nodemap
Summary

The persistent nodemap requires the rust index to be used to provides any gains.
So we automatically enable it for revlog using the persistent nodemap. We keep
it off for other revset because now that the rust revlog fully initialise the
nodemap using it everywhere introduce a fairly significant regression (eg: hg
diff moving from 0.8s to 2.3s on mozilla-try)

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

Alphare created this revision.Feb 26 2020, 12:37 PM
Alphare updated this revision to Diff 20356.Feb 27 2020, 4:15 AM
marmoute added a subscriber: marmoute.Mar 6 2020, 6:36 PM
marmoute updated this revision to Diff 20579.

eventless rebase past D8255

Alphare updated this revision to Diff 20662.Mar 10 2020, 6:52 AM
marmoute updated this revision to Diff 20679.Mar 10 2020, 9:35 PM

rebase above latest default up to landed-D8182

marmoute added a commit: rHGe7fff9c3cdac: rust-nodemap: automatically use the rust index for persistent nodemap.Mar 11 2020, 11:43 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGe7fff9c3cdac: rust-nodemap: automatically use the rust index for persistent nodemap (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/revlog.py (9 lines)
Mtests/test-persistent-nodemap.t (70 lines)

Diff 20383

mercurial/revlog.py

self._storedeltachains = True self._storedeltachains = True
devel_nodemap = ( devel_nodemap = (
self.nodemap_file self.nodemap_file
and opts.get(b'devel-force-nodemap', False) and opts.get(b'devel-force-nodemap', False)
and NodemapRevlogIO is not None and NodemapRevlogIO is not None
) )
use_rust_index = False
if rustrevlog is not None:
if self.nodemap_file is not None:
use_rust_index = True
else:
use_rust_index = self.opener.options.get(b'rust.index')
self._io = revlogio() self._io = revlogio()
if self.version == REVLOGV0: if self.version == REVLOGV0:
self._io = revlogoldio() self._io = revlogoldio()
elif devel_nodemap: elif devel_nodemap:
self._io = NodemapRevlogIO() self._io = NodemapRevlogIO()
elif rustrevlog is not None and self.opener.options.get(b'rust.index'): elif use_rust_index:
self._io = rustrevlogio() self._io = rustrevlogio()
try: try:
d = self._io.parseindex(indexdata, self._inline) d = self._io.parseindex(indexdata, self._inline)
index, _chunkcache = d index, _chunkcache = d
use_nodemap = ( use_nodemap = (
not self._inline not self._inline
and self.nodemap_file is not None and self.nodemap_file is not None
and util.safehasattr(index, 'update_nodemap_data') and util.safehasattr(index, 'update_nodemap_data')

tests/test-persistent-nodemap.t

=================================== ===================================
Test the persistent on-disk nodemap Test the persistent on-disk nodemap
=================================== ===================================
$ hg init test-repo $ hg init test-repo
$ cd test-repo $ cd test-repo
$ cat << EOF >> .hg/hgrc $ cat << EOF >> .hg/hgrc
> [experimental] > [experimental]
> exp-persistent-nodemap=yes > exp-persistent-nodemap=yes
> [devel] > [devel]
> persistent-nodemap=yes > persistent-nodemap=yes
> EOF > EOF
$ hg debugbuilddag .+5000 $ hg debugbuilddag .+5000
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5000 tip-rev: 5000
data-length: 122880 data-length: 122880
data-unused: 0 data-unused: 0
$ f --size .hg/store/00changelog.n $ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=42 .hg/store/00changelog.n: size=42
Simple lookup works Simple lookup works
$ ANYNODE=`hg log --template '{node|short}\n' --rev tip` $ ANYNODE=`hg log --template '{node|short}\n' --rev tip`
$ hg log -r "$ANYNODE" --template '{rev}\n' $ hg log -r "$ANYNODE" --template '{rev}\n'
5000 5000
#if rust
$ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=1e38e9ffaa45cad13f15c1a9880ad606f4241e8beea2f61b4d5365abadfb55f6 (glob)
$ hg debugnodemap --dump-new | f --sha256 --size
size=122880, sha256=1e38e9ffaa45cad13f15c1a9880ad606f4241e8beea2f61b4d5365abadfb55f6
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=122880, sha256=1e38e9ffaa45cad13f15c1a9880ad606f4241e8beea2f61b4d5365abadfb55f6
0000: 00 00 00 76 00 00 01 65 00 00 00 95 00 00 01 34 |...v...e.......4|
0010: 00 00 00 19 00 00 01 69 00 00 00 ab 00 00 00 4b |.......i.......K|
0020: 00 00 00 07 00 00 01 4c 00 00 00 f8 00 00 00 8f |.......L........|
0030: 00 00 00 c0 00 00 00 a7 00 00 00 89 00 00 01 46 |...............F|
0040: 00 00 00 92 00 00 01 bc 00 00 00 71 00 00 00 ac |...........q....|
0050: 00 00 00 af 00 00 00 b4 00 00 00 34 00 00 01 ca |...........4....|
0060: 00 00 00 23 00 00 01 45 00 00 00 2d 00 00 00 b2 |...#...E...-....|
0070: 00 00 00 56 00 00 01 0f 00 00 00 4e 00 00 02 4c |...V.......N...L|
0080: 00 00 00 e7 00 00 00 cd 00 00 01 5b 00 00 00 78 |...........[...x|
0090: 00 00 00 e3 00 00 01 8e 00 00 00 4f 00 00 00 b1 |...........O....|
00a0: 00 00 00 30 00 00 00 11 00 00 00 25 00 00 00 d2 |...0.......%....|
00b0: 00 00 00 ec 00 00 00 69 00 00 01 2b 00 00 01 2e |.......i...+....|
00c0: 00 00 00 aa 00 00 00 15 00 00 00 3a 00 00 01 4e |...........:...N|
00d0: 00 00 00 4d 00 00 00 9d 00 00 00 8e 00 00 00 a4 |...M............|
00e0: 00 00 00 c3 00 00 00 eb 00 00 00 29 00 00 00 ad |...........)....|
00f0: 00 00 01 3a 00 00 01 32 00 00 00 04 00 00 00 53 |...:...2.......S|
#else
$ f --sha256 .hg/store/00changelog-*.nd $ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7 (glob) .hg/store/00changelog-????????????????.nd: sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7 (glob)
$ hg debugnodemap --dump-new | f --sha256 --size $ hg debugnodemap --dump-new | f --sha256 --size
size=122880, sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7 size=122880, sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=122880, sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7 size=122880, sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: ff ff ff ff ff ff ff ff ff ff fa c2 ff ff ff ff |................| 0010: ff ff ff ff ff ff ff ff ff ff fa c2 ff ff ff ff |................|
0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0030: ff ff ff ff ff ff ed b3 ff ff ff ff ff ff ff ff |................| 0030: ff ff ff ff ff ff ed b3 ff ff ff ff ff ff ff ff |................|
0040: ff ff ff ff ff ff ee 34 00 00 00 00 ff ff ff ff |.......4........| 0040: ff ff ff ff ff ff ee 34 00 00 00 00 ff ff ff ff |.......4........|
0050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0080: ff ff ff ff ff ff f8 50 ff ff ff ff ff ff ff ff |.......P........| 0080: ff ff ff ff ff ff f8 50 ff ff ff ff ff ff ff ff |.......P........|
0090: ff ff ff ff ff ff ff ff ff ff ec c7 ff ff ff ff |................| 0090: ff ff ff ff ff ff ff ff ff ff ec c7 ff ff ff ff |................|
00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00b0: ff ff ff ff ff ff fa be ff ff f2 fc ff ff ff ff |................| 00b0: ff ff ff ff ff ff fa be ff ff f2 fc ff ff ff ff |................|
00c0: ff ff ff ff ff ff ef ea ff ff ff ff ff ff f9 17 |................| 00c0: ff ff ff ff ff ff ef ea ff ff ff ff ff ff f9 17 |................|
00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
#endif
$ hg debugnodemap --check $ hg debugnodemap --check
revision in index: 5001 revision in index: 5001
revision in nodemap: 5001 revision in nodemap: 5001
add a new commit add a new commit
$ hg up $ hg up
0 files updated, 0 files merged, 0 files removed, 0 files unresolved 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo foo > foo $ echo foo > foo
$ hg add foo $ hg add foo
$ hg ci -m 'foo' $ hg ci -m 'foo'
#if pure #if no-pure no-rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5001 tip-rev: 5001
data-length: 123072 data-length: 122880
data-unused: 192 data-unused: 0
#else #else
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5001 tip-rev: 5001
data-length: 122880 data-length: 123072
data-unused: 0 data-unused: 192
#endif #endif
$ f --size .hg/store/00changelog.n $ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=42 .hg/store/00changelog.n: size=42
(The pure code use the debug code that perform incremental update, the C code reencode from scratch) (The pure code use the debug code that perform incremental update, the C code reencode from scratch)
#if pure #if pure
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=123072, sha256=136472751566c8198ff09e306a7d2f9bd18bd32298d614752b73da4d6df23340 (glob) .hg/store/00changelog-????????????????.nd: size=123072, sha256=136472751566c8198ff09e306a7d2f9bd18bd32298d614752b73da4d6df23340 (glob)
#endif
#else #if rust
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=122880, sha256=bfafebd751c4f6d116a76a37a1dee2a251747affe7efbcc4f4842ccc746d4db9 (glob) .hg/store/00changelog-????????????????.nd: size=123072, sha256=ccc8a43310ace13812fcc648683e259346754ef934c12dd238cf9b7fadfe9a4b (glob)
#endif
#if no-pure no-rust
$ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=122880, sha256=bfafebd751c4f6d116a76a37a1dee2a251747affe7efbcc4f4842ccc746d4db9 (glob)
#endif #endif
$ hg debugnodemap --check $ hg debugnodemap --check
revision in index: 5002 revision in index: 5002
revision in nodemap: 5002 revision in nodemap: 5002
Test code path without mmap Test code path without mmap
--------------------------- ---------------------------
#if pure #if pure
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5002 tip-rev: 5002
data-length: 123328 data-length: 123328
data-unused: 384 data-unused: 384
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=123328, sha256=10d26e9776b6596af0f89143a54eba8cc581e929c38242a02a7b0760698c6c70 (glob) .hg/store/00changelog-????????????????.nd: size=123328, sha256=10d26e9776b6596af0f89143a54eba8cc581e929c38242a02a7b0760698c6c70 (glob)
#endif
#else #if rust
$ hg debugnodemap --metadata
uid: ???????????????? (glob)
tip-rev: 5002
data-length: 123328
data-unused: 384
$ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=123328, sha256=081eec9eb6708f2bf085d939b4c97bc0b6762bc8336bc4b93838f7fffa1516bf (glob)
#endif
#if no-pure no-rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5002 tip-rev: 5002
data-length: 122944 data-length: 122944
data-unused: 0 data-unused: 0
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=122944, sha256=755976b22b64ab680401b45395953504e64e7fa8c31ac570f58dee21e15f9bc0 (glob) .hg/store/00changelog-????????????????.nd: size=122944, sha256=755976b22b64ab680401b45395953504e64e7fa8c31ac570f58dee21e15f9bc0 (glob)
#endif #endif