( )⚙ D10754 persistent-nodemap: use the intended uuid size

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10754

persistent-nodemap: use the intended uuid size
ClosedPublic

Authored by marmoute on May 20 2021, 10:18 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG5bc6d2fc1cfc: persistent-nodemap: use the intended uuid size
Summary

I overlooked this in the initial implementation. The format already supporting
any uuid size so, nothing horrible should happens.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

marmoute created this revision.May 20 2021, 10:18 AM
pulkit accepted this revision.May 23 2021, 3:38 PM
This revision is now accepted and ready to land.May 23 2021, 3:38 PM
marmoute added a commit: rHG5bc6d2fc1cfc: persistent-nodemap: use the intended uuid size.May 23 2021, 6:06 PM
Closed by commit rHG5bc6d2fc1cfc: persistent-nodemap: use the intended uuid size (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 28155

mercurial/revlogutils/nodemap.py

ID_SIZE = 8 ID_SIZE = 8
def _make_uid(): def _make_uid():
"""return a new unique identifier. """return a new unique identifier.
The identifier is random and composed of ascii characters.""" The identifier is random and composed of ascii characters."""
return hex(os.urandom(ID_SIZE)) # size we "hex" the result we need half the number of bits to have a final
# uuid of size ID_SIZE
return hex(os.urandom(ID_SIZE // 2))
# some special test logic to avoid anoying random output in the test # some special test logic to avoid anoying random output in the test
stable_docket_file = encoding.environ.get(b'HGTEST_DOCKETIDFILE') stable_docket_file = encoding.environ.get(b'HGTEST_DOCKETIDFILE')
if stable_docket_file: if stable_docket_file:
def _make_uid(): def _make_uid():
low_part = (int_seed & low_mask) << 28 low_part = (int_seed & low_mask) << 28
int_seed = high_part + low_part + i int_seed = high_part + low_part + i
r = random.Random() r = random.Random()
if pycompat.ispy3: if pycompat.ispy3:
r.seed(int_seed, version=1) r.seed(int_seed, version=1)
else: else:
r.seed(int_seed) r.seed(int_seed)
# once we drop python 3.8 support we can simply use r.randbytes # once we drop python 3.8 support we can simply use r.randbytes
raw = r.getrandbits(ID_SIZE * 8) raw = r.getrandbits(ID_SIZE * 4)
assert ID_SIZE == 8 assert ID_SIZE == 8
p = struct.pack('>Q', raw) p = struct.pack('>L', raw)
new = hex(p) new = hex(p)
with open(stable_docket_file, 'wb') as f: with open(stable_docket_file, 'wb') as f:
f.write(new) f.write(new)
return new return new
class NodeMapDocket(object): class NodeMapDocket(object):
"""metadata associated with persistent nodemap data """metadata associated with persistent nodemap data

tests/test-persistent-nodemap.t

changelog-v2: no changelog-v2: no
plain-cl-delta: yes plain-cl-delta: yes
compression: zlib (no-zstd !) compression: zlib (no-zstd !)
compression: zstd (zstd !) compression: zstd (zstd !)
compression-level: default compression-level: default
$ hg debugbuilddag .+5000 --new-file $ hg debugbuilddag .+5000 --new-file
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5000 tip-rev: 5000
tip-node: 6b02b8c7b96654c25e86ba69eda198d7e6ad8b3c tip-node: 6b02b8c7b96654c25e86ba69eda198d7e6ad8b3c
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ f --size .hg/store/00changelog.n $ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=70 .hg/store/00changelog.n: size=62
Simple lookup works Simple lookup works
$ ANYNODE=`hg log --template '{node|short}\n' --rev tip` $ ANYNODE=`hg log --template '{node|short}\n' --rev tip`
$ hg log -r "$ANYNODE" --template '{rev}\n' $ hg log -r "$ANYNODE" --template '{rev}\n'
5000 5000
#if rust #if rust
$ f --sha256 .hg/store/00changelog-*.nd $ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob) .hg/store/00changelog-????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob)
$ f --sha256 .hg/store/00manifest-*.nd $ f --sha256 .hg/store/00manifest-*.nd
.hg/store/00manifest-????????????????.nd: sha256=97117b1c064ea2f86664a124589e47db0e254e8d34739b5c5cc5bf31c9da2b51 (glob) .hg/store/00manifest-????????.nd: sha256=97117b1c064ea2f86664a124589e47db0e254e8d34739b5c5cc5bf31c9da2b51 (glob)
$ hg debugnodemap --dump-new | f --sha256 --size $ hg debugnodemap --dump-new | f --sha256 --size
size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 |....... ........| 0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 |....... ........|
0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 |...f..........."| 0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 |...f..........."|
0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e |...#...........^| 0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e |...#...........^|
0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab |.......N...e....| 0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab |.......N...e....|
00d0: 00 00 00 b8 00 00 01 31 00 00 00 2c 00 00 00 55 |.......1...,...U| 00d0: 00 00 00 b8 00 00 01 31 00 00 00 2c 00 00 00 55 |.......1...,...U|
00e0: 00 00 00 8a 00 00 00 9a 00 00 00 0c 00 00 01 1e |................| 00e0: 00 00 00 8a 00 00 00 9a 00 00 00 0c 00 00 01 1e |................|
00f0: 00 00 00 a4 00 00 00 83 00 00 00 c9 00 00 00 8c |................| 00f0: 00 00 00 a4 00 00 00 83 00 00 00 c9 00 00 00 8c |................|
#else #else
$ f --sha256 .hg/store/00changelog-*.nd $ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 (glob) .hg/store/00changelog-????????.nd: sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 (glob)
$ hg debugnodemap --dump-new | f --sha256 --size $ hg debugnodemap --dump-new | f --sha256 --size
size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79
0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0020: ff ff ff ff ff ff f5 06 ff ff ff ff ff ff f3 e7 |................| 0020: ff ff ff ff ff ff f5 06 ff ff ff ff ff ff f3 e7 |................|
0030: ff ff ef ca ff ff ff ff ff ff ff ff ff ff ff ff |................| 0030: ff ff ef ca ff ff ff ff ff ff ff ff ff ff ff ff |................|
6b02b8c7b966+ tip 6b02b8c7b966+ tip
#endif #endif
$ hg ci -m 'foo' $ hg ci -m 'foo'
#if no-pure no-rust #if no-pure no-rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5001 tip-rev: 5001
tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
#else #else
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5001 tip-rev: 5001
tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c
data-length: 121344 data-length: 121344
data-unused: 256 data-unused: 256
data-unused: 0.211% data-unused: 0.211%
#endif #endif
$ f --size .hg/store/00changelog.n $ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=70 .hg/store/00changelog.n: size=62
(The pure code use the debug code that perform incremental update, the C code reencode from scratch) (The pure code use the debug code that perform incremental update, the C code reencode from scratch)
#if pure #if pure
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121344, sha256=cce54c5da5bde3ad72a4938673ed4064c86231b9c64376b082b163fdb20f8f66 (glob) .hg/store/00changelog-????????.nd: size=121344, sha256=cce54c5da5bde3ad72a4938673ed4064c86231b9c64376b082b163fdb20f8f66 (glob)
#endif #endif
#if rust #if rust
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121344, sha256=952b042fcf614ceb37b542b1b723e04f18f83efe99bee4e0f5ccd232ef470e58 (glob) .hg/store/00changelog-????????.nd: size=121344, sha256=952b042fcf614ceb37b542b1b723e04f18f83efe99bee4e0f5ccd232ef470e58 (glob)
#endif #endif
#if no-pure no-rust #if no-pure no-rust
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121088, sha256=df7c06a035b96cb28c7287d349d603baef43240be7736fe34eea419a49702e17 (glob) .hg/store/00changelog-????????.nd: size=121088, sha256=df7c06a035b96cb28c7287d349d603baef43240be7736fe34eea419a49702e17 (glob)
#endif #endif
$ hg debugnodemap --check $ hg debugnodemap --check
revision in index: 5002 revision in index: 5002
revision in nodemap: 5002 revision in nodemap: 5002
Test code path without mmap Test code path without mmap
--------------------------- ---------------------------
$ echo bar > bar $ echo bar > bar
$ hg add bar $ hg add bar
$ hg ci -m 'bar' --config storage.revlog.persistent-nodemap.mmap=no $ hg ci -m 'bar' --config storage.revlog.persistent-nodemap.mmap=no
$ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=yes $ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=yes
revision in index: 5003 revision in index: 5003
revision in nodemap: 5003 revision in nodemap: 5003
$ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=no $ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=no
revision in index: 5003 revision in index: 5003
revision in nodemap: 5003 revision in nodemap: 5003
#if pure #if pure
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121600 data-length: 121600
data-unused: 512 data-unused: 512
data-unused: 0.421% data-unused: 0.421%
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121600, sha256=def52503d049ccb823974af313a98a935319ba61f40f3aa06a8be4d35c215054 (glob) .hg/store/00changelog-????????.nd: size=121600, sha256=def52503d049ccb823974af313a98a935319ba61f40f3aa06a8be4d35c215054 (glob)
#endif #endif
#if rust #if rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121600 data-length: 121600
data-unused: 512 data-unused: 512
data-unused: 0.421% data-unused: 0.421%
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121600, sha256=dacf5b5f1d4585fee7527d0e67cad5b1ba0930e6a0928f650f779aefb04ce3fb (glob) .hg/store/00changelog-????????.nd: size=121600, sha256=dacf5b5f1d4585fee7527d0e67cad5b1ba0930e6a0928f650f779aefb04ce3fb (glob)
#endif #endif
#if no-pure no-rust #if no-pure no-rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121088, sha256=59fcede3e3cc587755916ceed29e3c33748cd1aa7d2f91828ac83e7979d935e8 (glob) .hg/store/00changelog-????????.nd: size=121088, sha256=59fcede3e3cc587755916ceed29e3c33748cd1aa7d2f91828ac83e7979d935e8 (glob)
#endif #endif
Test force warming the cache Test force warming the cache
$ rm .hg/store/00changelog.n $ rm .hg/store/00changelog.n
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
$ hg debugupdatecache $ hg debugupdatecache
#if pure #if pure
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
#else #else
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
#endif #endif
Check out of sync nodemap Check out of sync nodemap
========================= =========================
First copy old data on the side. First copy old data on the side.
$ mkdir ../tmp-copies $ mkdir ../tmp-copies
$ cp .hg/store/00changelog-????????????????.nd .hg/store/00changelog.n ../tmp-copies $ cp .hg/store/00changelog-????????.nd .hg/store/00changelog.n ../tmp-copies
Nodemap lagging behind Nodemap lagging behind
---------------------- ----------------------
make a new commit make a new commit
$ echo bar2 > bar $ echo bar2 > bar
$ hg ci -m 'bar2' $ hg ci -m 'bar2'
$ NODE=`hg log -r tip -T '{node}\n'` $ NODE=`hg log -r tip -T '{node}\n'`
$ hg log -r "$NODE" -T '{rev}\n' $ hg log -r "$NODE" -T '{rev}\n'
5003 5003
If the nodemap is lagging behind, it can catch up fine If the nodemap is lagging behind, it can catch up fine
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5003 tip-rev: 5003
tip-node: c9329770f979ade2d16912267c38ba5f82fd37b3 tip-node: c9329770f979ade2d16912267c38ba5f82fd37b3
data-length: 121344 (pure !) data-length: 121344 (pure !)
data-length: 121344 (rust !) data-length: 121344 (rust !)
data-length: 121152 (no-rust no-pure !) data-length: 121152 (no-rust no-pure !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-rust no-pure !) data-unused: 0 (no-rust no-pure !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-rust no-pure !) data-unused: 0.000% (no-rust no-pure !)
$ cp -f ../tmp-copies/* .hg/store/ $ cp -f ../tmp-copies/* .hg/store/
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ hg log -r "$NODE" -T '{rev}\n' $ hg log -r "$NODE" -T '{rev}\n'
5003 5003
$ hg log -r "$OTHERNODE" -T '{rev}\n' $ hg log -r "$OTHERNODE" -T '{rev}\n'
5004 5004
$ hg --config extensions.strip= strip --rev "$NODE~1" --no-backup $ hg --config extensions.strip= strip --rev "$NODE~1" --no-backup
the nodemap should detect the changelog have been tampered with and recover. the nodemap should detect the changelog have been tampered with and recover.
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: b355ef8adce0949b8bdf6afc72ca853740d65944 tip-node: b355ef8adce0949b8bdf6afc72ca853740d65944
data-length: 121536 (pure !) data-length: 121536 (pure !)
data-length: 121088 (rust !) data-length: 121088 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 448 (pure !) data-unused: 448 (pure !)
data-unused: 0 (rust !) data-unused: 0 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.000% (rust !) data-unused: 0.000% (rust !)
data-unused: 0.369% (pure !) data-unused: 0.369% (pure !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ cp -f ../tmp-copies/* .hg/store/ $ cp -f ../tmp-copies/* .hg/store/
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ hg log -r "$OTHERNODE" -T '{rev}\n' $ hg log -r "$OTHERNODE" -T '{rev}\n'
5002 5002
================================== ==================================
An up to date nodemap should be available to shell hooks, An up to date nodemap should be available to shell hooks,
$ echo dsljfl > a $ echo dsljfl > a
$ hg add a $ hg add a
$ hg ci -m a $ hg ci -m a
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5003 tip-rev: 5003
tip-node: a52c5079765b5865d97b993b303a18740113bbb2 tip-node: a52c5079765b5865d97b993b303a18740113bbb2
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ echo babar2 > babar $ echo babar2 > babar
$ hg ci -m 'babar2' --config "hooks.pretxnclose.nodemap-test=hg debugnodemap --metadata" $ hg ci -m 'babar2' --config "hooks.pretxnclose.nodemap-test=hg debugnodemap --metadata"
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5004 tip-rev: 5004
tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984 tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984
data-length: 121280 (pure !) data-length: 121280 (pure !)
data-length: 121280 (rust !) data-length: 121280 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5004 tip-rev: 5004
tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984 tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984
data-length: 121280 (pure !) data-length: 121280 (pure !)
data-length: 121280 (rust !) data-length: 121280 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
Another process does not see the pending nodemap content during run. Another process does not see the pending nodemap content during run.
$ echo qpoasp > a $ echo qpoasp > a
$ hg ci -m a2 \ $ hg ci -m a2 \
> --config "hooks.pretxnclose=sh \"$RUNTESTDIR/testlib/wait-on-file\" 20 sync-repo-read sync-txn-pending" \ > --config "hooks.pretxnclose=sh \"$RUNTESTDIR/testlib/wait-on-file\" 20 sync-repo-read sync-txn-pending" \
> --config "hooks.txnclose=touch sync-txn-close" > output.txt 2>&1 & > --config "hooks.txnclose=touch sync-txn-close" > output.txt 2>&1 &
(read the repository while the commit transaction is pending) (read the repository while the commit transaction is pending)
$ sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-pending && \ $ sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-pending && \
> hg debugnodemap --metadata && \ > hg debugnodemap --metadata && \
> sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-close sync-repo-read > sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-close sync-repo-read
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5004 tip-rev: 5004
tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984 tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984
data-length: 121280 (pure !) data-length: 121280 (pure !)
data-length: 121280 (rust !) data-length: 121280 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121536 (pure !) data-length: 121536 (pure !)
data-length: 121536 (rust !) data-length: 121536 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 448 (pure !) data-unused: 448 (pure !)
data-unused: 448 (rust !) data-unused: 448 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.369% (pure !) data-unused: 0.369% (pure !)
data-unused: 0.369% (rust !) data-unused: 0.369% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ cat output.txt $ cat output.txt
Check that a failing transaction will properly revert the data Check that a failing transaction will properly revert the data
$ echo plakfe > a $ echo plakfe > a
$ f --size --sha256 .hg/store/00changelog-*.nd $ f --size --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !) .hg/store/00changelog-????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !)
.hg/store/00changelog-????????????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !) .hg/store/00changelog-????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !)
.hg/store/00changelog-????????????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !) .hg/store/00changelog-????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !)
$ hg ci -m a3 --config "extensions.abort=$RUNTESTDIR/testlib/crash_transaction_late.py" $ hg ci -m a3 --config "extensions.abort=$RUNTESTDIR/testlib/crash_transaction_late.py"
transaction abort! transaction abort!
rollback completed rollback completed
abort: This is a late abort abort: This is a late abort
[255] [255]
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121536 (pure !) data-length: 121536 (pure !)
data-length: 121536 (rust !) data-length: 121536 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 448 (pure !) data-unused: 448 (pure !)
data-unused: 448 (rust !) data-unused: 448 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.369% (pure !) data-unused: 0.369% (pure !)
data-unused: 0.369% (rust !) data-unused: 0.369% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ f --size --sha256 .hg/store/00changelog-*.nd $ f --size --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !) .hg/store/00changelog-????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !)
.hg/store/00changelog-????????????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !) .hg/store/00changelog-????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !)
.hg/store/00changelog-????????????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !) .hg/store/00changelog-????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !)
Check that removing content does not confuse the nodemap Check that removing content does not confuse the nodemap
-------------------------------------------------------- --------------------------------------------------------
removing data with rollback removing data with rollback
$ echo aso > a $ echo aso > a
$ hg ci -m a4 $ hg ci -m a4
The persistent nodemap should exist after a streaming clone The persistent nodemap should exist after a streaming clone
Simple case Simple case
----------- -----------
No race condition No race condition
$ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)' $ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)'
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (62 bytes)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (62 bytes)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00manifest.d (491 KB) (zstd !) adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
$ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)' $ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
Check and record file state beforehand Check and record file state beforehand
$ f --size test-repo/.hg/store/00changelog* $ f --size test-repo/.hg/store/00changelog*
test-repo/.hg/store/00changelog-*.nd: size=121088 (glob) test-repo/.hg/store/00changelog-*.nd: size=121088 (glob)
test-repo/.hg/store/00changelog.d: size=376891 (zstd !) test-repo/.hg/store/00changelog.d: size=376891 (zstd !)
test-repo/.hg/store/00changelog.d: size=368890 (no-zstd !) test-repo/.hg/store/00changelog.d: size=368890 (no-zstd !)
test-repo/.hg/store/00changelog.i: size=320384 test-repo/.hg/store/00changelog.i: size=320384
test-repo/.hg/store/00changelog.n: size=70 test-repo/.hg/store/00changelog.n: size=62
$ hg -R test-repo debugnodemap --metadata | tee server-metadata.txt $ hg -R test-repo debugnodemap --metadata | tee server-metadata.txt
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
Prepare a commit Prepare a commit
$ echo foo >> test-repo/foo $ echo foo >> test-repo/foo
$ hg -R test-repo/ add test-repo/foo $ hg -R test-repo/ add test-repo/foo
Do a mix of clone and commit at the same time so that the file listed on disk differ at actual transfer time. Do a mix of clone and commit at the same time so that the file listed on disk differ at actual transfer time.
$ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | egrep '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) & $ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | egrep '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) &
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1
$ hg -R test-repo/ commit -m foo $ hg -R test-repo/ commit -m foo
$ touch $HG_TEST_STREAM_WALKED_FILE_2 $ touch $HG_TEST_STREAM_WALKED_FILE_2
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3
$ cat clone-output $ cat clone-output
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (62 bytes)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (62 bytes)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00manifest.d (491 KB) (zstd !) adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
Check the result state Check the result state
$ f --size stream-clone-race-1/.hg/store/00changelog* $ f --size stream-clone-race-1/.hg/store/00changelog*
stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob) stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob)
stream-clone-race-1/.hg/store/00changelog.d: size=368890 (no-zstd !) stream-clone-race-1/.hg/store/00changelog.d: size=368890 (no-zstd !)
stream-clone-race-1/.hg/store/00changelog.d: size=376891 (zstd !) stream-clone-race-1/.hg/store/00changelog.d: size=376891 (zstd !)
stream-clone-race-1/.hg/store/00changelog.i: size=320384 stream-clone-race-1/.hg/store/00changelog.i: size=320384
stream-clone-race-1/.hg/store/00changelog.n: size=70 stream-clone-race-1/.hg/store/00changelog.n: size=62
$ hg -R stream-clone-race-1 debugnodemap --metadata | tee client-metadata.txt $ hg -R stream-clone-race-1 debugnodemap --metadata | tee client-metadata.txt
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ f --size test-repo/.hg/store/00changelog* $ f --size test-repo/.hg/store/00changelog*
test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !) test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !)
test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !) test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !)
test-repo/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !) test-repo/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !)
test-repo/.hg/store/00changelog.d: size=376950 (zstd !) test-repo/.hg/store/00changelog.d: size=376950 (zstd !)
test-repo/.hg/store/00changelog.d: size=368949 (no-zstd !) test-repo/.hg/store/00changelog.d: size=368949 (no-zstd !)
test-repo/.hg/store/00changelog.i: size=320448 test-repo/.hg/store/00changelog.i: size=320448
test-repo/.hg/store/00changelog.n: size=70 test-repo/.hg/store/00changelog.n: size=62
$ hg -R test-repo debugnodemap --metadata | tee server-metadata-2.txt $ hg -R test-repo debugnodemap --metadata | tee server-metadata-2.txt
uid: * (glob) uid: * (glob)
tip-rev: 5006 tip-rev: 5006
tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b
data-length: 121344 (rust !) data-length: 121344 (rust !)
data-length: 121344 (pure !) data-length: 121344 (pure !)
data-length: 121152 (no-rust no-pure !) data-length: 121152 (no-rust no-pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
$ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-2 --debug 2>> clone-output-2 | egrep '00(changelog|manifest)' >> clone-output-2; touch $HG_TEST_STREAM_WALKED_FILE_3) & $ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-2 --debug 2>> clone-output-2 | egrep '00(changelog|manifest)' >> clone-output-2; touch $HG_TEST_STREAM_WALKED_FILE_3) &
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1
$ rm test-repo/.hg/store/00changelog.n $ rm test-repo/.hg/store/00changelog.n
$ rm test-repo/.hg/store/00changelog-*.nd $ rm test-repo/.hg/store/00changelog-*.nd
$ hg -R test-repo/ debugupdatecache $ hg -R test-repo/ debugupdatecache
$ touch $HG_TEST_STREAM_WALKED_FILE_2 $ touch $HG_TEST_STREAM_WALKED_FILE_2
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3
$ cat clone-output-2 $ cat clone-output-2
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (62 bytes)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (62 bytes)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.d (492 KB) (zstd !) adding [s] 00manifest.d (492 KB) (zstd !)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
Check the result. Check the result.
$ f --size stream-clone-race-2/.hg/store/00changelog* $ f --size stream-clone-race-2/.hg/store/00changelog*
stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !) stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !)
stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !) stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !)
stream-clone-race-2/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !) stream-clone-race-2/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !)
stream-clone-race-2/.hg/store/00changelog.d: size=376950 (zstd !) stream-clone-race-2/.hg/store/00changelog.d: size=376950 (zstd !)
stream-clone-race-2/.hg/store/00changelog.d: size=368949 (no-zstd !) stream-clone-race-2/.hg/store/00changelog.d: size=368949 (no-zstd !)
stream-clone-race-2/.hg/store/00changelog.i: size=320448 stream-clone-race-2/.hg/store/00changelog.i: size=320448
stream-clone-race-2/.hg/store/00changelog.n: size=70 stream-clone-race-2/.hg/store/00changelog.n: size=62
$ hg -R stream-clone-race-2 debugnodemap --metadata | tee client-metadata-2.txt $ hg -R stream-clone-race-2 debugnodemap --metadata | tee client-metadata-2.txt
uid: * (glob) uid: * (glob)
tip-rev: 5006 tip-rev: 5006
tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b
data-length: 121344 (rust !) data-length: 121344 (rust !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)