This is an archive of the discontinued Mercurial Phabricator instance.

nodemap: introduce an option to use mmap to read the nodemap mapping
ClosedPublic

Authored by marmoute on Jan 15 2020, 9:56 AM.

Download Raw Diff

Details

Reviewers

durin42
indygreg

Group Reviewers

hg-reviewers

Commits

rHGf7459da77f23: nodemap: introduce an option to use mmap to read the nodemap mapping

Summary

The performance and memory benefit is much greater if we don't have to copy all
the data in memory for each information. So we introduce an option (on by
default) to read the data using mmap.

This changeset is the last one definition the API for index support nodemap
data. (they have to be able to use the mmaping).

Below are some benchmark comparing the best we currently have in 5.3 with the
final step of this series (using the persistent nodemap implementation in
Rust). The benchmark run hg perfindex with various revset and the following
variants:

Before:

do not use the persistent nodemap
use the CPython implementation of the index for nodemap
use mmapping of the changelog index

After:

use the MixedIndex Rust code, with the NodeTree object for nodemap access (still in review)
use the persistent nodemap data from disk
access the persistent nodemap data through mmap
use mmapping of the changelog index

The persistent nodemap greatly speed up most operation on very large
repositories. Some of the previously very fast lookup end up a bit slower because
the persistent nodemap has to be setup. However the absolute slowdown is very
small and won't matters in the big picture.

Here are some numbers (in seconds) for the reference copy of mozilla-try:

Revset Before After abs-change speedup

-10000: 0.004622 0.005532 0.000910 × 0.83
-10: 0.000050 0.000132 0.000082 × 0.37
tip 0.000052 0.000085 0.000033 × 0.61
0 + (-10000:) 0.028222 0.005337 -0.022885 × 5.29
0 0.023521 0.000084 -0.023437 × 280.01
(-10000:) + 0 0.235539 0.005308 -0.230231 × 44.37
(-10:) + :9 0.232883 0.000180 -0.232703 ×1293.79
(-10000:) + (:99) 0.238735 0.005358 -0.233377 × 44.55
:99 + (-10000:) 0.317942 0.005593 -0.312349 × 56.84
:9 + (-10:) 0.313372 0.000179 -0.313193 ×1750.68
:9 0.316450 0.000143 -0.316307 ×2212.93

On smaller repositories, the cost of nodemap related operation is not as big, so
the win is much more modest. Yet it helps shaving a handful of millisecond here
and there.

Here are some numbers (in seconds) for the reference copy of mercurial:

Revset Before After abs-change speedup
-10: 0.000065 0.000097 0.000032 × 0.67
tip 0.000063 0.000078 0.000015 × 0.80
0 0.000561 0.000079 -0.000482 × 7.10
-10000: 0.004609 0.003648 -0.000961 × 1.26
0 + (-10000:) 0.005023 0.003715 -0.001307 × 1.35
(-10:) + :9 0.002187 0.000108 -0.002079 ×20.25
(-10000:) + 0 0.006252 0.003716 -0.002536 × 1.68
(-10000:) + (:99) 0.006367 0.003707 -0.002660 × 1.71
:9 + (-10:) 0.003846 0.000110 -0.003736 ×34.96
:9 0.003854 0.000099 -0.003755 ×38.92
:99 + (-10000:) 0.007644 0.003778 -0.003866 × 2.02

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Jan 15 2020, 9:56 AM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptJan 15 2020, 9:56 AM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

How much does this patch help performance?

I would also like to see performance numbers (even just rough ones) for the Rust version. Sorry about a possibly stupid question, but why will this on-disk nodemap be faster than building it from the index? Is it that the file is smaller and thus faster to read? Or is it more the building of the tree than the reading that's slow? You mentioned you use some private repo for testing this. How large is the 00changelog.n file in that repo and how large is 00changelog.i?

In D7894#116397, @martinvonz wrote:

How much does this patch help performance?
I would also like to see performance numbers (even just rough ones) for the Rust version. Sorry about a possibly stupid question, but why will this on-disk nodemap be faster than building it from the index? Is it that the file is smaller and thus faster to read? Or is it more the building of the tree than the reading that's slow? You mentioned you use some private repo for testing this. How large is the 00changelog.n file in that repo and how large is 00changelog.i?

This save hundreds of milli second at initialization of large repositories. The repositories we are looking at are about 2 millions revisions. (but this will help smaller repository too). Mozilla try is a public repository in that range. It 00changelog.i is 103MB

The information in the .i files is just a flat list of node. So anything that need a mapping needs to build it. Building a mapping for millions of revision is slow. (I think Georges mentionned 300ms to build the mozilla-try nodemap). The nodemap we write on disk is directly usage as such. So we just need to mmap the files (mostly instant is the repository have been busy recently, eg: on server) and directly query the data from disk. We will get more data once the latest version of this series and the latest version of the Rust series (from @gracinet) are connected again. But this greatly boost many operation (small operation, discovery, etc).

In D7894#116407, @marmoute wrote:

In D7894#116397, @martinvonz wrote:

How much does this patch help performance?
I would also like to see performance numbers (even just rough ones) for the Rust version. Sorry about a possibly stupid question, but why will this on-disk nodemap be faster than building it from the index? Is it that the file is smaller and thus faster to read? Or is it more the building of the tree than the reading that's slow? You mentioned you use some private repo for testing this. How large is the 00changelog.n file in that repo and how large is 00changelog.i?

This save hundreds of milli second at initialization of large repositories. The repositories we are looking at are about 2 millions revisions. (but this will help smaller repository too). Mozilla try is a public repository in that range. It 00changelog.i is 103MB

And 00changelog.n?

The information in the .i files is just a flat list of node. So anything that need a mapping needs to build it. Building a mapping for millions of revision is slow. (I think Georges mentionned 300ms to build the mozilla-try nodemap). The nodemap we write on disk is directly usage as such. So we just need to mmap the files (mostly instant is the repository have been busy recently, eg: on server) and directly query the data from disk.

Ah, that's what I was wondering. I was wondering while reviewing this series if your plan was to lazily from disk but I didn't see any mention of that. I guess this mmap business should have been that a hint :)

In D7894#116540, @martinvonz wrote:

In D7894#116407, @marmoute wrote:

In D7894#116397, @martinvonz wrote:

How much does this patch help performance?
I would also like to see performance numbers (even just rough ones) for the Rust version. Sorry about a possibly stupid question, but why will this on-disk nodemap be faster than building it from the index? Is it that the file is smaller and thus faster to read? Or is it more the building of the tree than the reading that's slow? You mentioned you use some private repo for testing this. How large is the 00changelog.n file in that repo and how large is 00changelog.i?

This save hundreds of milli second at initialization of large repositories. The repositories we are looking at are about 2 millions revisions. (but this will help smaller repository too). Mozilla try is a public repository in that range. It 00changelog.i is 103MB

And 00changelog.n?

106973952 bytes for changelog.i
 83123200 bytes for the nodemap rawfiles

The information in the .i files is just a flat list of node. So anything that need a mapping needs to build it. Building a mapping for millions of revision is slow. (I think Georges mentionned 300ms to build the mozilla-try nodemap). The nodemap we write on disk is directly usage as such. So we just need to mmap the files (mostly instant is the repository have been busy recently, eg: on server) and directly query the data from disk.

Ah, that's what I was wondering. I was wondering while reviewing this series if your plan was to lazily from disk but I didn't see any mention of that. I guess this mmap business should have been that a hint :)

Yes, so everythign we are doing is not so really "serialization" since we never actualy "deserialize" it in practice.

marmoute updated this revision to Diff 19435.Jan 17 2020, 1:07 PM

hooo, I am a bot

marmoute updated this revision to Diff 19775.Jan 31 2020, 10:30 AM

marmoute updated this revision to Diff 19802.Jan 31 2020, 6:01 PM

small doc update on .#s[1]

rebase to latest default

marmoute updated this revision to Diff 20018.Feb 7 2020, 5:30 PM

Here are quick performance number on our mozilla-try reference:

For perfindex --rev tip I now get

c:    ! wall 0.000067 comb 0.000000 user 0.000000 sys 0.000000 (median of 36546)
rust: ! wall 0.000197 comb 0.000000 user 0.000000 sys 0.000000 (median of 13090)

For perfindex --rev '-10000: + 0' I now get

c:    ! wall 0.567621 comb 0.570000 user 0.560000 sys 0.010000 (median of 18)
rust: ! wall 0.011586 comb 0.000000 user 0.000000 sys 0.000000 (median of 254)

More throughful benchmarking in progress.

More throughful benchmarking has arrived.

This is a large win all over the board. The win for the larger repository very significant.

+         3.9e-05          7.1e-05     1.82  internal.index.no_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.8e-05            7e-05     1.84  internal.index.no_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.7e-05            7e-05     1.89  internal.index.no_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.002178         0.000111     0.05  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         6.4e-05          9.8e-05     1.53  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000561          7.9e-05     0.14  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.003858         0.000108     0.03  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.003848           0.0001     0.03  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         6.3e-05          7.8e-05     1.24  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.002184          0.00011     0.05  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         6.6e-05          9.9e-05     1.50  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000567            8e-05     0.14  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.00386         0.000106     0.03  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.003892          9.9e-05     0.03  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         6.2e-05          7.7e-05     1.24  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.002187         0.000108     0.05  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         6.5e-05          9.7e-05     1.49  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000561          7.9e-05     0.14  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.003846          0.00011     0.03  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.003854          9.9e-05     0.03  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         6.3e-05          7.8e-05     1.24  internal.index.small_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006298         0.004007     0.64  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006272         0.003874     0.62  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004634         0.003636     0.78  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004998          0.00365     0.73  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.007712         0.003678     0.48  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006313         0.003732     0.59  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.00629         0.003769     0.60  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004632         0.003654     0.79  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.005032         0.003735     0.74  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.007575          0.00379     0.50  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006367         0.003707     0.58  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006252         0.003716     0.59  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004609         0.003648     0.79  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.005023         0.003715     0.74  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.007644         0.003778     0.49  internal.index.large_lookup.track_time('mercurial-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]

+         3.8e-05          7.3e-05     1.92  internal.index.no_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.008179         0.000131     0.02  internal.index.small_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
         0.000106         0.000115     1.08  internal.index.small_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.001654          8.3e-05     0.05  internal.index.small_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.013476         0.000134     0.01  internal.index.small_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.013184         0.000112     0.01  internal.index.small_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000102            8e-05     0.78  internal.index.small_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.012199         0.004018     0.33  internal.index.large_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.011962         0.003782     0.32  internal.index.large_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004641         0.004017     0.87  internal.index.large_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006088          0.00382     0.63  internal.index.large_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.01875         0.003775     0.20  internal.index.large_lookup.track_time('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]

+         3.9e-05          7.2e-05     1.85  internal.index.no_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.9e-05          7.3e-05     1.87  internal.index.no_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.8e-05          7.1e-05     1.87  internal.index.no_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006601          0.00013     0.02  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         9.6e-05          0.00011     1.15  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.001394          7.9e-05     0.06  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010903          0.00013     0.01  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010888         0.000115     0.01  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
          7.4e-05            8e-05     1.08  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006601         0.000129     0.02  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         9.7e-05         0.000108     1.11  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-          0.0014            8e-05     0.06  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010844         0.000128     0.01  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010786         0.000106     0.01  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         9.3e-05            8e-05     0.86  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006475         0.000131     0.02  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         9.6e-05          0.00011     1.15  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.001397          8.2e-05     0.06  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010938         0.000133     0.01  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010665         0.000106     0.01  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         6.5e-05          8.2e-05     1.26  internal.index.small_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010683         0.003773     0.35  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010673         0.003735     0.35  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.00463          0.00382     0.83  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.005921         0.003757     0.63  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.014701         0.004016     0.27  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010648         0.003816     0.36  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010652         0.003766     0.35  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004969         0.003773     0.76  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.005925         0.003782     0.64  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.01466         0.003777     0.26  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010633          0.00386     0.36  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.010727         0.003785     0.35  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004668         0.003688     0.79  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.005846         0.004064     0.70  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.014876          0.00384     0.26  internal.index.large_lookup.track_time('pypy-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]

+         3.7e-05          7.5e-05     2.03  internal.index.no_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.8e-05            7e-05     1.84  internal.index.no_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.9e-05          7.2e-05     1.85  internal.index.no_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.02847         0.000145     0.01  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000239         0.000116     0.49  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004983            8e-05     0.02  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.042669         0.000143     0.00  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.042118         0.000117     0.00  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000244          8.3e-05     0.34  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.027695         0.000146     0.01  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000168         0.000117     0.70  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004967          8.2e-05     0.02  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.042328         0.000144     0.00  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.042587         0.000119     0.00  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000238          8.1e-05     0.34  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.028557         0.000145     0.01  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000165          0.00012     0.73  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004941          8.4e-05     0.02  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.042682          0.00014     0.00  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.042114         0.000118     0.00  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000236          8.1e-05     0.34  internal.index.small_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.032823         0.004066     0.12  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.031798         0.003967     0.12  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004849         0.003907     0.81  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.009529         0.004034     0.42  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.046217         0.004013     0.09  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.031647         0.004058     0.13  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.032553         0.004017     0.12  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004805         0.004136     0.86  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.009515         0.004009     0.42  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.046954         0.003983     0.08  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.03245         0.004078     0.13  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.031685         0.003989     0.13  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.00484         0.003889     0.80  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.009477         0.004019     0.42  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.04654         0.004023     0.09  internal.index.large_lookup.track_time('netbeans-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]

+         4.1e-05          7.3e-05     1.78  internal.index.no_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.7e-05          7.1e-05     1.92  internal.index.no_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.9e-05            7e-05     1.79  internal.index.no_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.041165         0.000153     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000166         0.000119     0.72  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.006994          8.1e-05     0.01  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.060687         0.000157     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.061438         0.000118     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000275          8.1e-05     0.29  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.039947         0.000154     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000164         0.000119     0.73  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.007001          8.3e-05     0.01  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.06026         0.000156     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.061074         0.000121     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000275          8.3e-05     0.30  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.040962         0.000153     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000164         0.000121     0.74  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.007109          8.5e-05     0.01  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.061388         0.000152     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.06147         0.000125     0.00  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.000274          8.2e-05     0.30  internal.index.small_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.045977         0.004112     0.09  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.045198         0.004046     0.09  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004805         0.004118     0.86  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.011548         0.004232     0.37  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.066339          0.00408     0.06  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, False, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.046125         0.004158     0.09  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.046382         0.004132     0.09  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.004815          0.00402     0.83  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.011542         0.004066     0.35  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.066069         0.004116     0.06  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zlib', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.044966         0.004142     0.09  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.045773         0.004113     0.09  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.00484         0.004036     0.83  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.011581         0.004036     0.35  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.065447         0.004214     0.06  internal.index.large_lookup.track_time('mozilla-central-2018-08-01', 'zstd', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]

+         3.7e-05          7.1e-05     1.92  internal.index.no_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         3.8e-05          7.3e-05     1.92  internal.index.no_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.236488         0.000174     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+           5e-05         0.000132     2.64  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.023564          8.6e-05     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.315712         0.000178     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.315888         0.000141     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         4.6e-05          8.4e-05     1.83  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.232883          0.00018     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, '(-10:) + :9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+           5e-05         0.000132     2.64  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, '-10:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.023521          8.4e-05     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, '0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.313372         0.000179     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, ':9 + (-10:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-         0.31645         0.000143     0.00  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, ':9') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+         5.2e-05          8.5e-05     1.63  internal.index.small_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, 'tip') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.242474         0.005356     0.02  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.242892         0.005314     0.02  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+        0.004802         0.005326     1.11  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.028061         0.005319     0.19  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.318157         0.005362     0.02  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zlib', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.238735         0.005358     0.02  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, '(-10000:) + (:99)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.235539         0.005308     0.02  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, '(-10000:) + 0') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
+        0.004622         0.005532     1.20  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, '-10000:') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.028222         0.005337     0.19  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, '0 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
-        0.317942         0.005593     0.02  internal.index.large_lookup.track_time('mozilla-try-2019-02-18', 'zstd', 'default', True, True, True, True, True, ':99 + (-10000:)') [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]

In D7894#120247, @marmoute wrote:
More throughful benchmarking has arrived.
This is a large win all over the board. The win for the larger repository very significant.
+         3.9e-05          7.1e-05     1.82  internal.index.no_lookup.track_time('mercurial-2018-08-01', 'zlib', 'default', True, True, True, True, False) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]

[...]

I literally have no idea what this wall of text is trying to tell me. Some column headers would be a baseline to making this readable, structuring it as a table somehow would be even better, and for actual archaeology it would have been put in some summarized form in the commit message.

I've reviewed up to here and I'm basically happy, but I'll hold this one until the commit message has at least some useful summary of where we're going performance wise.

This revision now requires changes to proceed.Feb 10 2020, 8:15 PM

marmoute updated this revision to Diff 20165.Feb 11 2020, 7:01 AM

Herald added a reviewer: indygreg. · View Herald TranscriptFeb 11 2020, 7:01 AM

(Marking this as wanting changes per discussions about adding some headline numbers in the log message so I stop looking at it until that happens.)

This revision now requires changes to proceed.Feb 12 2020, 11:32 AM

marmoute edited the summary of this revision. (Show Details)Feb 12 2020, 10:01 PM

phabricator seems to be very confused about the performance number formatting…

It is easier to read here: https://foss.heptapod.net/octobus/mercurial-devel/commit/fb04cf42b051ef84bce081f9d10f5ff75c3e0445

and can be pulled using hg pull --rev "nodemap-python-side" https://foss.heptapod.net/octobus/mercurial-devel/

I can also confirm that phabricator will silently devour anything added after the Differential Revision: line

marmoute added a commit: rHGf7459da77f23: nodemap: introduce an option to use mmap to read the nodemap mapping.Feb 15 2020, 2:12 PM

This revision was not accepted when it landed; it landed in state Needs Revision.

Closed by commit rHGf7459da77f23: nodemap: introduce an option to use mmap to read the nodemap mapping (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/configitems.py (3 lines)
M			mercurial/debugcommands.py (2 lines)
M			mercurial/localrepo.py (2 lines)
M			mercurial/revlog.py (4 lines)
M			mercurial/revlogutils/nodemap.py (39 lines)
M			tests/test-persistent-nodemap.t (34 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	19312		Jan 15 2020, 9:56 AM	★	★
Diff 2	19435		Jan 17 2020, 1:07 PM	★	★
Diff 3	19775		Jan 31 2020, 10:30 AM	★	★
Diff 4	19802		Jan 31 2020, 6:01 PM	★	★
Diff 5	19846		Feb 2 2020, 3:20 AM	★	★
Diff 6	19906		Feb 4 2020, 7:26 PM	★	★
Diff 7	20018		Feb 7 2020, 5:30 PM	★	★
Diff 8	20165		Feb 11 2020, 7:01 AM	★	★
Diff 9	20240	rHGf7459da77f234502a2a4b1ce8523cc1886442f54	Feb 11 2020, 5:18 AM	★	★

Status	Author	Revision
Closed	marmoute	D7894 nodemap: introduce an option to use mmap to read the nodemap mapping
Closed	marmoute	D7893 nodemap: update the index with the newly written data (when appropriate)
Closed	marmoute	D7892 nodemap: never read more than the expected data amount
Closed	marmoute	D7891 nodemap: write new data from the expected current data length
Closed	marmoute	D7890 nodemap: double check the source docket when doing incremental update
Closed	marmoute	D7889 nodemap: track the total and unused amount of data in the rawdata file
Closed	marmoute	D7888 nodemap: track the maximum revision tracked in the nodemap
Closed	marmoute	D7887 nodemap: add a flag to dump the details of the docket
Closed	marmoute	D7886 nodemap: introduce append-only incremental update of the persisten data
Closed	marmoute	D7885 nodemap: keep track of the docket for loaded data
Closed	marmoute	D7884 nodemap: introduce an explicit class/object for the docket
Closed	marmoute	D7883 nodemap: keep track of the ondisk id of nodemap blocks
Closed	marmoute	D7847 nodemap: provide the on disk data to indexes who support it
Closed	marmoute	D7846 nodemap: all check that revision and nodes match in the nodemap
Closed	marmoute	D7845 nodemap: add basic checking of the on disk nodemap content
Closed	marmoute	D7844 nodemap: code to parse the persistent binary nodemap data
Closed	marmoute	D7843 nodemap: move the iteratio inside the Block object
Closed	marmoute	D7842 nodemap: use an explicit "Block" object in the reference implementation
Closed	marmoute	D7841 nodemap: add a optional `nodemap_add_full` method on indexes
Closed	marmoute	D7840 nodemap: add a (python) index class for persistent nodemap testing
Closed	marmoute	D7839 nodemap: delete older raw data file when creating a new ones
Closed	marmoute	D7838 nodemap: use an intermediate "docket" file to carry small metadata
Closed	marmoute	D7837 nodemap: only use persistent nodemap for non-inlined revlog
Closed	marmoute	D7836 nodemap: add a function to read the data from disk
Closed	marmoute	D7835 nodemap: write nodemap data on disk
Closed	marmoute	D7834 nodemap: have some python code writing a nodemap in persistent binary form

Diff 20240

mercurial/configitems.py

	)			)
	coreconfigitem(			coreconfigitem(
	b'experimental', b'rust.index', default=False,			b'experimental', b'rust.index', default=False,
	)			)
	coreconfigitem(			coreconfigitem(
	b'experimental', b'exp-persistent-nodemap', default=False,			b'experimental', b'exp-persistent-nodemap', default=False,
	)			)
	coreconfigitem(			coreconfigitem(
				b'experimental', b'exp-persistent-nodemap.mmap', default=True,
				)
				coreconfigitem(
	b'experimental', b'server.filesdata.recommended-batch-size', default=50000,			b'experimental', b'server.filesdata.recommended-batch-size', default=50000,
	)			)
	coreconfigitem(			coreconfigitem(
	b'experimental',			b'experimental',
	b'server.manifestdata.recommended-batch-size',			b'server.manifestdata.recommended-batch-size',
	default=100000,			default=100000,
	)			)
	coreconfigitem(			coreconfigitem(

mercurial/debugcommands.py

	data = nodemap.persistent_data(cl.index)			data = nodemap.persistent_data(cl.index)
	ui.write(data)			ui.write(data)
	elif opts['dump_disk']:			elif opts['dump_disk']:
	unfi = repo.unfiltered()			unfi = repo.unfiltered()
	cl = unfi.changelog			cl = unfi.changelog
	nm_data = nodemap.persisted_data(cl)			nm_data = nodemap.persisted_data(cl)
	if nm_data is not None:			if nm_data is not None:
	docket, data = nm_data			docket, data = nm_data
	ui.write(data)			ui.write(data[:])
	elif opts['check']:			elif opts['check']:
	unfi = repo.unfiltered()			unfi = repo.unfiltered()
	cl = unfi.changelog			cl = unfi.changelog
	nm_data = nodemap.persisted_data(cl)			nm_data = nodemap.persisted_data(cl)
	if nm_data is not None:			if nm_data is not None:
	docket, data = nm_data			docket, data = nm_data
	return nodemap.check_data(ui, cl.index, data)			return nodemap.check_data(ui, cl.index, data)
	elif opts['metadata']:			elif opts['metadata']:

mercurial/localrepo.py


	if repository.NARROW_REQUIREMENT in requirements:			if repository.NARROW_REQUIREMENT in requirements:
	options[b'enableellipsis'] = True			options[b'enableellipsis'] = True

	if ui.configbool(b'experimental', b'rust.index'):			if ui.configbool(b'experimental', b'rust.index'):
	options[b'rust.index'] = True			options[b'rust.index'] = True
	if ui.configbool(b'experimental', b'exp-persistent-nodemap'):			if ui.configbool(b'experimental', b'exp-persistent-nodemap'):
	options[b'exp-persistent-nodemap'] = True			options[b'exp-persistent-nodemap'] = True
				if ui.configbool(b'experimental', b'exp-persistent-nodemap.mmap'):
				options[b'exp-persistent-nodemap.mmap'] = True
	if ui.configbool(b'devel', b'persistent-nodemap'):			if ui.configbool(b'devel', b'persistent-nodemap'):
	options[b'devel-force-nodemap'] = True			options[b'devel-force-nodemap'] = True

	return options			return options


	def makemain(**kwargs):			def makemain(**kwargs):
	"""Produce a type conforming to ``ilocalrepositorymain``."""			"""Produce a type conforming to ``ilocalrepositorymain``."""

mercurial/revlog.py

	raise error.RevlogError(			raise error.RevlogError(
	_(b'revlog chunk cache size %r is not a power of 2')			_(b'revlog chunk cache size %r is not a power of 2')
	% self._chunkcachesize			% self._chunkcachesize
	)			)

	indexdata = b''			indexdata = b''
	self._initempty = True			self._initempty = True
	try:			try:
	nodemap_data = nodemaputil.persisted_data(self)
	if nodemap_data is not None:
	self._nodemap_docket = nodemap_data[0]
	with self._indexfp() as f:			with self._indexfp() as f:
	if (			if (
	mmapindexthreshold is not None			mmapindexthreshold is not None
	and self.opener.fstat(f).st_size >= mmapindexthreshold			and self.opener.fstat(f).st_size >= mmapindexthreshold
	):			):
	# TODO: should .close() to release resources without			# TODO: should .close() to release resources without
	# relying on Python GC			# relying on Python GC
	indexdata = util.buffer(util.mmapread(f))			indexdata = util.buffer(util.mmapread(f))
	use_nodemap = (			use_nodemap = (
	not self._inline			not self._inline
	and self.nodemap_file is not None			and self.nodemap_file is not None
	and util.safehasattr(index, 'update_nodemap_data')			and util.safehasattr(index, 'update_nodemap_data')
	)			)
	if use_nodemap:			if use_nodemap:
	nodemap_data = nodemaputil.persisted_data(self)			nodemap_data = nodemaputil.persisted_data(self)
	if nodemap_data is not None:			if nodemap_data is not None:
				self._nodemap_docket = nodemap_data[0]
	index.update_nodemap_data(*nodemap_data)			index.update_nodemap_data(*nodemap_data)
	except (ValueError, IndexError):			except (ValueError, IndexError):
	raise error.RevlogError(			raise error.RevlogError(
	_(b"index %s is corrupted") % self.indexfile			_(b"index %s is corrupted") % self.indexfile
	)			)
	self.index, self._chunkcache = d			self.index, self._chunkcache = d
	if not self._chunkcache:			if not self._chunkcache:
	self._chunkclear()			self._chunkclear()

mercurial/revlogutils/nodemap.py

	# nodemap.py - nodemap related code and utilities			# nodemap.py - nodemap related code and utilities
	#			#
	# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>			# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
	# Copyright 2019 George Racinet <georges.racinet@octobus.net>			# Copyright 2019 George Racinet <georges.racinet@octobus.net>
	#			#
	# This software may be used and distributed according to the terms of the			# This software may be used and distributed according to the terms of the
	# GNU General Public License version 2 or any later version.			# GNU General Public License version 2 or any later version.

	from __future__ import absolute_import			from __future__ import absolute_import

				import errno
	import os			import os
	import re			import re
	import struct			import struct

	from .. import (			from .. import (
	error,			error,
	node as nodemod,			node as nodemod,
	util,			util,
	uid_size, tip_rev, data_length, data_unused = headers			uid_size, tip_rev, data_length, data_unused = headers
	offset += S_HEADER.size			offset += S_HEADER.size
	docket = NodeMapDocket(pdata[offset : offset + uid_size])			docket = NodeMapDocket(pdata[offset : offset + uid_size])
	docket.tip_rev = tip_rev			docket.tip_rev = tip_rev
	docket.data_length = data_length			docket.data_length = data_length
	docket.data_unused = data_unused			docket.data_unused = data_unused

	filename = _rawdata_filepath(revlog, docket)			filename = _rawdata_filepath(revlog, docket)
	data = revlog.opener.tryread(filename)			use_mmap = revlog.opener.options.get("exp-persistent-nodemap.mmap")
				try:
				with revlog.opener(filename) as fd:
				if use_mmap:
				data = util.buffer(util.mmapread(fd, data_length))
				else:
				data = fd.read(data_length)
				except OSError as e:
				if e.errno != errno.ENOENT:
				raise
	if len(data) < data_length:			if len(data) < data_length:
	return None			return None
	elif len(data) > data_length:
	data = data[:data_length]
	return docket, data			return docket, data


	def setup_persistent_nodemap(tr, revlog):			def setup_persistent_nodemap(tr, revlog):
	"""Install whatever is needed transaction side to persist a nodemap on disk			"""Install whatever is needed transaction side to persist a nodemap on disk

	(only actually persist the nodemap if this is relevant for this revlog)			(only actually persist the nodemap if this is relevant for this revlog)
	"""			"""
	"cannot persist nodemap of a filtered changelog"			"cannot persist nodemap of a filtered changelog"
	)			)
	if revlog.nodemap_file is None:			if revlog.nodemap_file is None:
	msg = "calling persist nodemap on a revlog without the feature enableb"			msg = "calling persist nodemap on a revlog without the feature enableb"
	raise error.ProgrammingError(msg)			raise error.ProgrammingError(msg)

	can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")			can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
	ondisk_docket = revlog._nodemap_docket			ondisk_docket = revlog._nodemap_docket
				feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
				use_mmap = revlog.opener.options.get("exp-persistent-nodemap.mmap")

	data = None			data = None
	# first attemp an incremental update of the data			# first attemp an incremental update of the data
	if can_incremental and ondisk_docket is not None:			if can_incremental and ondisk_docket is not None:
	target_docket = revlog._nodemap_docket.copy()			target_docket = revlog._nodemap_docket.copy()
	(			(
	src_docket,			src_docket,
	data_changed_count,			data_changed_count,
	data,			data,
	) = revlog.index.nodemap_data_incremental()			) = revlog.index.nodemap_data_incremental()
	if src_docket != target_docket:			if src_docket != target_docket:
	data = None			data = None
	else:			else:
	datafile = _rawdata_filepath(revlog, target_docket)			datafile = _rawdata_filepath(revlog, target_docket)
	# EXP-TODO: if this is a cache, this should use a cache vfs, not a			# EXP-TODO: if this is a cache, this should use a cache vfs, not a
	# store vfs			# store vfs
				new_length = target_docket.data_length + len(data)
	with revlog.opener(datafile, b'r+') as fd:			with revlog.opener(datafile, b'r+') as fd:
	fd.seek(target_docket.data_length)			fd.seek(target_docket.data_length)
	fd.write(data)			fd.write(data)
				if feed_data:
				if use_mmap:
	fd.seek(0)			fd.seek(0)
	new_data = fd.read(target_docket.data_length + len(data))			new_data = fd.read(new_length)
	target_docket.data_length += len(data)			else:
				fd.flush()
				new_data = util.buffer(util.mmapread(fd, new_length))
				target_docket.data_length = new_length
	target_docket.data_unused += data_changed_count			target_docket.data_unused += data_changed_count

	if data is None:			if data is None:
	# otherwise fallback to a full new export			# otherwise fallback to a full new export
	target_docket = NodeMapDocket()			target_docket = NodeMapDocket()
	datafile = _rawdata_filepath(revlog, target_docket)			datafile = _rawdata_filepath(revlog, target_docket)
	if util.safehasattr(revlog.index, "nodemap_data_all"):			if util.safehasattr(revlog.index, "nodemap_data_all"):
	data = revlog.index.nodemap_data_all()			data = revlog.index.nodemap_data_all()
	else:			else:
	data = persistent_data(revlog.index)			data = persistent_data(revlog.index)
	# EXP-TODO: if this is a cache, this should use a cache vfs, not a			# EXP-TODO: if this is a cache, this should use a cache vfs, not a
	# store vfs			# store vfs
	new_data = data			with revlog.opener(datafile, b'w+') as fd:
	with revlog.opener(datafile, b'w') as fd:
	fd.write(data)			fd.write(data)
				if feed_data:
				if use_mmap:
				new_data = data
				else:
				fd.flush()
				new_data = util.buffer(util.mmapread(fd, len(data)))
	target_docket.data_length = len(data)			target_docket.data_length = len(data)
	target_docket.tip_rev = revlog.tiprev()			target_docket.tip_rev = revlog.tiprev()
	# EXP-TODO: if this is a cache, this should use a cache vfs, not a			# EXP-TODO: if this is a cache, this should use a cache vfs, not a
	# store vfs			# store vfs
	with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:			with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
	fp.write(target_docket.serialize())			fp.write(target_docket.serialize())
	revlog._nodemap_docket = target_docket			revlog._nodemap_docket = target_docket
	if util.safehasattr(revlog.index, "update_nodemap_data"):			if feed_data:
	revlog.index.update_nodemap_data(target_docket, new_data)			revlog.index.update_nodemap_data(target_docket, new_data)

	# EXP-TODO: if the transaction abort, we should remove the new data and			# EXP-TODO: if the transaction abort, we should remove the new data and
	# reinstall the old one.			# reinstall the old one.

	# search for old index file in all cases, some older process might have			# search for old index file in all cases, some older process might have
	# left one behind.			# left one behind.
	olds = _other_rawdata_filepath(revlog, target_docket)			olds = _other_rawdata_filepath(revlog, target_docket)

tests/test-persistent-nodemap.t

	$ f --sha256 .hg/store/00changelog-*.nd --size			$ f --sha256 .hg/store/00changelog-*.nd --size
	.hg/store/00changelog-????????????????.nd: size=122880, sha256=bfafebd751c4f6d116a76a37a1dee2a251747affe7efbcc4f4842ccc746d4db9 (glob)			.hg/store/00changelog-????????????????.nd: size=122880, sha256=bfafebd751c4f6d116a76a37a1dee2a251747affe7efbcc4f4842ccc746d4db9 (glob)

	#endif			#endif

	$ hg debugnodemap --check			$ hg debugnodemap --check
	revision in index: 5002			revision in index: 5002
	revision in nodemap: 5002			revision in nodemap: 5002

				Test code path without mmap
				---------------------------

				$ echo bar > bar
				$ hg add bar
				$ hg ci -m 'bar' --config experimental.exp-persistent-nodemap.mmap=no

				$ hg debugnodemap --check --config experimental.exp-persistent-nodemap.mmap=yes
				revision in index: 5003
				revision in nodemap: 5003
				$ hg debugnodemap --check --config experimental.exp-persistent-nodemap.mmap=no
				revision in index: 5003
				revision in nodemap: 5003


				#if pure
				$ hg debugnodemap --metadata
				uid: ???????????????? (glob)
				tip-rev: 5002
				data-length: 123328
				data-unused: 384
				$ f --sha256 .hg/store/00changelog-*.nd --size
				.hg/store/00changelog-????????????????.nd: size=123328, sha256=10d26e9776b6596af0f89143a54eba8cc581e929c38242a02a7b0760698c6c70 (glob)

				#else
				$ hg debugnodemap --metadata
				uid: ???????????????? (glob)
				tip-rev: 5002
				data-length: 122944
				data-unused: 0
				$ f --sha256 .hg/store/00changelog-*.nd --size
				.hg/store/00changelog-????????????????.nd: size=122944, sha256=755976b22b64ab680401b45395953504e64e7fa8c31ac570f58dee21e15f9bc0 (glob)
				#endif