This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10754

persistent-nodemap: use the intended uuid size
ClosedPublic

Authored by marmoute on May 20 2021, 10:18 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG5bc6d2fc1cfc: persistent-nodemap: use the intended uuid size
Summary

I overlooked this in the initial implementation. The format already supporting
any uuid size so, nothing horrible should happens.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.May 20 2021, 10:18 AM
pulkit accepted this revision.May 23 2021, 3:38 PM
This revision is now accepted and ready to land.May 23 2021, 3:38 PM
marmoute added a commit: rHG5bc6d2fc1cfc: persistent-nodemap: use the intended uuid size.May 23 2021, 6:06 PM
Closed by commit rHG5bc6d2fc1cfc: persistent-nodemap: use the intended uuid size (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 28188

mercurial/revlogutils/nodemap.py

ID_SIZE = 8 ID_SIZE = 8
def _make_uid(): def _make_uid():
"""return a new unique identifier. """return a new unique identifier.
The identifier is random and composed of ascii characters.""" The identifier is random and composed of ascii characters."""
return hex(os.urandom(ID_SIZE)) # size we "hex" the result we need half the number of bits to have a final
# uuid of size ID_SIZE
return hex(os.urandom(ID_SIZE // 2))
# some special test logic to avoid anoying random output in the test # some special test logic to avoid anoying random output in the test
stable_docket_file = encoding.environ.get(b'HGTEST_DOCKETIDFILE') stable_docket_file = encoding.environ.get(b'HGTEST_DOCKETIDFILE')
if stable_docket_file: if stable_docket_file:
def _make_uid(): def _make_uid():
low_part = (int_seed & low_mask) << 28 low_part = (int_seed & low_mask) << 28
int_seed = high_part + low_part + i int_seed = high_part + low_part + i
r = random.Random() r = random.Random()
if pycompat.ispy3: if pycompat.ispy3:
r.seed(int_seed, version=1) r.seed(int_seed, version=1)
else: else:
r.seed(int_seed) r.seed(int_seed)
# once we drop python 3.8 support we can simply use r.randbytes # once we drop python 3.8 support we can simply use r.randbytes
raw = r.getrandbits(ID_SIZE * 8) raw = r.getrandbits(ID_SIZE * 4)
assert ID_SIZE == 8 assert ID_SIZE == 8
p = struct.pack('>Q', raw) p = struct.pack('>L', raw)
new = hex(p) new = hex(p)
with open(stable_docket_file, 'wb') as f: with open(stable_docket_file, 'wb') as f:
f.write(new) f.write(new)
return new return new
class NodeMapDocket(object): class NodeMapDocket(object):
"""metadata associated with persistent nodemap data """metadata associated with persistent nodemap data

tests/test-persistent-nodemap.t

changelog-v2: no changelog-v2: no
plain-cl-delta: yes plain-cl-delta: yes
compression: zlib (no-zstd !) compression: zlib (no-zstd !)
compression: zstd (zstd !) compression: zstd (zstd !)
compression-level: default compression-level: default
$ hg debugbuilddag .+5000 --new-file $ hg debugbuilddag .+5000 --new-file
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5000 tip-rev: 5000
tip-node: 6b02b8c7b96654c25e86ba69eda198d7e6ad8b3c tip-node: 6b02b8c7b96654c25e86ba69eda198d7e6ad8b3c
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ f --size .hg/store/00changelog.n $ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=70 .hg/store/00changelog.n: size=62
Simple lookup works Simple lookup works
$ ANYNODE=`hg log --template '{node|short}\n' --rev tip` $ ANYNODE=`hg log --template '{node|short}\n' --rev tip`
$ hg log -r "$ANYNODE" --template '{rev}\n' $ hg log -r "$ANYNODE" --template '{rev}\n'
5000 5000
#if rust #if rust
$ f --sha256 .hg/store/00changelog-*.nd $ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob) .hg/store/00changelog-????????.nd: sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd (glob)
$ f --sha256 .hg/store/00manifest-*.nd $ f --sha256 .hg/store/00manifest-*.nd
.hg/store/00manifest-????????????????.nd: sha256=97117b1c064ea2f86664a124589e47db0e254e8d34739b5c5cc5bf31c9da2b51 (glob) .hg/store/00manifest-????????.nd: sha256=97117b1c064ea2f86664a124589e47db0e254e8d34739b5c5cc5bf31c9da2b51 (glob)
$ hg debugnodemap --dump-new | f --sha256 --size $ hg debugnodemap --dump-new | f --sha256 --size
size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd size=121088, sha256=2e029d3200bd1a986b32784fc2ef1a3bd60dc331f025718bcf5ff44d93f026fd
0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 |....... ........| 0000: 00 00 00 91 00 00 00 20 00 00 00 bb 00 00 00 e7 |....... ........|
0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 |...f..........."| 0010: 00 00 00 66 00 00 00 a1 00 00 01 13 00 00 01 22 |...f..........."|
0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e |...#...........^| 0020: 00 00 00 23 00 00 00 fc 00 00 00 ba 00 00 00 5e |...#...........^|
0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab |.......N...e....| 0030: 00 00 00 df 00 00 01 4e 00 00 01 65 00 00 00 ab |.......N...e....|
00d0: 00 00 00 b8 00 00 01 31 00 00 00 2c 00 00 00 55 |.......1...,...U| 00d0: 00 00 00 b8 00 00 01 31 00 00 00 2c 00 00 00 55 |.......1...,...U|
00e0: 00 00 00 8a 00 00 00 9a 00 00 00 0c 00 00 01 1e |................| 00e0: 00 00 00 8a 00 00 00 9a 00 00 00 0c 00 00 01 1e |................|
00f0: 00 00 00 a4 00 00 00 83 00 00 00 c9 00 00 00 8c |................| 00f0: 00 00 00 a4 00 00 00 83 00 00 00 c9 00 00 00 8c |................|
#else #else
$ f --sha256 .hg/store/00changelog-*.nd $ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 (glob) .hg/store/00changelog-????????.nd: sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 (glob)
$ hg debugnodemap --dump-new | f --sha256 --size $ hg debugnodemap --dump-new | f --sha256 --size
size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79 size=121088, sha256=f544f5462ff46097432caf6d764091f6d8c46d6121be315ead8576d548c9dd79
0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0020: ff ff ff ff ff ff f5 06 ff ff ff ff ff ff f3 e7 |................| 0020: ff ff ff ff ff ff f5 06 ff ff ff ff ff ff f3 e7 |................|
0030: ff ff ef ca ff ff ff ff ff ff ff ff ff ff ff ff |................| 0030: ff ff ef ca ff ff ff ff ff ff ff ff ff ff ff ff |................|
6b02b8c7b966+ tip 6b02b8c7b966+ tip
#endif #endif
$ hg ci -m 'foo' $ hg ci -m 'foo'
#if no-pure no-rust #if no-pure no-rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5001 tip-rev: 5001
tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
#else #else
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5001 tip-rev: 5001
tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c tip-node: 16395c3cf7e231394735e6b1717823ada303fb0c
data-length: 121344 data-length: 121344
data-unused: 256 data-unused: 256
data-unused: 0.211% data-unused: 0.211%
#endif #endif
$ f --size .hg/store/00changelog.n $ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=70 .hg/store/00changelog.n: size=62
(The pure code use the debug code that perform incremental update, the C code reencode from scratch) (The pure code use the debug code that perform incremental update, the C code reencode from scratch)
#if pure #if pure
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121344, sha256=cce54c5da5bde3ad72a4938673ed4064c86231b9c64376b082b163fdb20f8f66 (glob) .hg/store/00changelog-????????.nd: size=121344, sha256=cce54c5da5bde3ad72a4938673ed4064c86231b9c64376b082b163fdb20f8f66 (glob)
#endif #endif
#if rust #if rust
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121344, sha256=952b042fcf614ceb37b542b1b723e04f18f83efe99bee4e0f5ccd232ef470e58 (glob) .hg/store/00changelog-????????.nd: size=121344, sha256=952b042fcf614ceb37b542b1b723e04f18f83efe99bee4e0f5ccd232ef470e58 (glob)
#endif #endif
#if no-pure no-rust #if no-pure no-rust
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121088, sha256=df7c06a035b96cb28c7287d349d603baef43240be7736fe34eea419a49702e17 (glob) .hg/store/00changelog-????????.nd: size=121088, sha256=df7c06a035b96cb28c7287d349d603baef43240be7736fe34eea419a49702e17 (glob)
#endif #endif
$ hg debugnodemap --check $ hg debugnodemap --check
revision in index: 5002 revision in index: 5002
revision in nodemap: 5002 revision in nodemap: 5002
Test code path without mmap Test code path without mmap
--------------------------- ---------------------------
$ echo bar > bar $ echo bar > bar
$ hg add bar $ hg add bar
$ hg ci -m 'bar' --config storage.revlog.persistent-nodemap.mmap=no $ hg ci -m 'bar' --config storage.revlog.persistent-nodemap.mmap=no
$ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=yes $ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=yes
revision in index: 5003 revision in index: 5003
revision in nodemap: 5003 revision in nodemap: 5003
$ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=no $ hg debugnodemap --check --config storage.revlog.persistent-nodemap.mmap=no
revision in index: 5003 revision in index: 5003
revision in nodemap: 5003 revision in nodemap: 5003
#if pure #if pure
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121600 data-length: 121600
data-unused: 512 data-unused: 512
data-unused: 0.421% data-unused: 0.421%
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121600, sha256=def52503d049ccb823974af313a98a935319ba61f40f3aa06a8be4d35c215054 (glob) .hg/store/00changelog-????????.nd: size=121600, sha256=def52503d049ccb823974af313a98a935319ba61f40f3aa06a8be4d35c215054 (glob)
#endif #endif
#if rust #if rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121600 data-length: 121600
data-unused: 512 data-unused: 512
data-unused: 0.421% data-unused: 0.421%
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121600, sha256=dacf5b5f1d4585fee7527d0e67cad5b1ba0930e6a0928f650f779aefb04ce3fb (glob) .hg/store/00changelog-????????.nd: size=121600, sha256=dacf5b5f1d4585fee7527d0e67cad5b1ba0930e6a0928f650f779aefb04ce3fb (glob)
#endif #endif
#if no-pure no-rust #if no-pure no-rust
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ f --sha256 .hg/store/00changelog-*.nd --size $ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=121088, sha256=59fcede3e3cc587755916ceed29e3c33748cd1aa7d2f91828ac83e7979d935e8 (glob) .hg/store/00changelog-????????.nd: size=121088, sha256=59fcede3e3cc587755916ceed29e3c33748cd1aa7d2f91828ac83e7979d935e8 (glob)
#endif #endif
Test force warming the cache Test force warming the cache
$ rm .hg/store/00changelog.n $ rm .hg/store/00changelog.n
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
$ hg debugupdatecache $ hg debugupdatecache
#if pure #if pure
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
#else #else
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
#endif #endif
Check out of sync nodemap Check out of sync nodemap
========================= =========================
First copy old data on the side. First copy old data on the side.
$ mkdir ../tmp-copies $ mkdir ../tmp-copies
$ cp .hg/store/00changelog-????????????????.nd .hg/store/00changelog.n ../tmp-copies $ cp .hg/store/00changelog-????????.nd .hg/store/00changelog.n ../tmp-copies
Nodemap lagging behind Nodemap lagging behind
---------------------- ----------------------
make a new commit make a new commit
$ echo bar2 > bar $ echo bar2 > bar
$ hg ci -m 'bar2' $ hg ci -m 'bar2'
$ NODE=`hg log -r tip -T '{node}\n'` $ NODE=`hg log -r tip -T '{node}\n'`
$ hg log -r "$NODE" -T '{rev}\n' $ hg log -r "$NODE" -T '{rev}\n'
5003 5003
If the nodemap is lagging behind, it can catch up fine If the nodemap is lagging behind, it can catch up fine
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5003 tip-rev: 5003
tip-node: c9329770f979ade2d16912267c38ba5f82fd37b3 tip-node: c9329770f979ade2d16912267c38ba5f82fd37b3
data-length: 121344 (pure !) data-length: 121344 (pure !)
data-length: 121344 (rust !) data-length: 121344 (rust !)
data-length: 121152 (no-rust no-pure !) data-length: 121152 (no-rust no-pure !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-rust no-pure !) data-unused: 0 (no-rust no-pure !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-rust no-pure !) data-unused: 0.000% (no-rust no-pure !)
$ cp -f ../tmp-copies/* .hg/store/ $ cp -f ../tmp-copies/* .hg/store/
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ hg log -r "$NODE" -T '{rev}\n' $ hg log -r "$NODE" -T '{rev}\n'
5003 5003
$ hg log -r "$OTHERNODE" -T '{rev}\n' $ hg log -r "$OTHERNODE" -T '{rev}\n'
5004 5004
$ hg --config extensions.strip= strip --rev "$NODE~1" --no-backup $ hg --config extensions.strip= strip --rev "$NODE~1" --no-backup
the nodemap should detect the changelog have been tampered with and recover. the nodemap should detect the changelog have been tampered with and recover.
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: b355ef8adce0949b8bdf6afc72ca853740d65944 tip-node: b355ef8adce0949b8bdf6afc72ca853740d65944
data-length: 121536 (pure !) data-length: 121536 (pure !)
data-length: 121088 (rust !) data-length: 121088 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 448 (pure !) data-unused: 448 (pure !)
data-unused: 0 (rust !) data-unused: 0 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.000% (rust !) data-unused: 0.000% (rust !)
data-unused: 0.369% (pure !) data-unused: 0.369% (pure !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ cp -f ../tmp-copies/* .hg/store/ $ cp -f ../tmp-copies/* .hg/store/
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd tip-node: 880b18d239dfa9f632413a2071bfdbcc4806a4fd
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ hg log -r "$OTHERNODE" -T '{rev}\n' $ hg log -r "$OTHERNODE" -T '{rev}\n'
5002 5002
================================== ==================================
An up to date nodemap should be available to shell hooks, An up to date nodemap should be available to shell hooks,
$ echo dsljfl > a $ echo dsljfl > a
$ hg add a $ hg add a
$ hg ci -m a $ hg ci -m a
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5003 tip-rev: 5003
tip-node: a52c5079765b5865d97b993b303a18740113bbb2 tip-node: a52c5079765b5865d97b993b303a18740113bbb2
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ echo babar2 > babar $ echo babar2 > babar
$ hg ci -m 'babar2' --config "hooks.pretxnclose.nodemap-test=hg debugnodemap --metadata" $ hg ci -m 'babar2' --config "hooks.pretxnclose.nodemap-test=hg debugnodemap --metadata"
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5004 tip-rev: 5004
tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984 tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984
data-length: 121280 (pure !) data-length: 121280 (pure !)
data-length: 121280 (rust !) data-length: 121280 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5004 tip-rev: 5004
tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984 tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984
data-length: 121280 (pure !) data-length: 121280 (pure !)
data-length: 121280 (rust !) data-length: 121280 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
Another process does not see the pending nodemap content during run. Another process does not see the pending nodemap content during run.
$ echo qpoasp > a $ echo qpoasp > a
$ hg ci -m a2 \ $ hg ci -m a2 \
> --config "hooks.pretxnclose=sh \"$RUNTESTDIR/testlib/wait-on-file\" 20 sync-repo-read sync-txn-pending" \ > --config "hooks.pretxnclose=sh \"$RUNTESTDIR/testlib/wait-on-file\" 20 sync-repo-read sync-txn-pending" \
> --config "hooks.txnclose=touch sync-txn-close" > output.txt 2>&1 & > --config "hooks.txnclose=touch sync-txn-close" > output.txt 2>&1 &
(read the repository while the commit transaction is pending) (read the repository while the commit transaction is pending)
$ sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-pending && \ $ sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-pending && \
> hg debugnodemap --metadata && \ > hg debugnodemap --metadata && \
> sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-close sync-repo-read > sh "$RUNTESTDIR/testlib/wait-on-file" 20 sync-txn-close sync-repo-read
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5004 tip-rev: 5004
tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984 tip-node: 2f5fb1c06a16834c5679d672e90da7c5f3b1a984
data-length: 121280 (pure !) data-length: 121280 (pure !)
data-length: 121280 (rust !) data-length: 121280 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 192 (pure !) data-unused: 192 (pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.158% (pure !) data-unused: 0.158% (pure !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121536 (pure !) data-length: 121536 (pure !)
data-length: 121536 (rust !) data-length: 121536 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 448 (pure !) data-unused: 448 (pure !)
data-unused: 448 (rust !) data-unused: 448 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.369% (pure !) data-unused: 0.369% (pure !)
data-unused: 0.369% (rust !) data-unused: 0.369% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ cat output.txt $ cat output.txt
Check that a failing transaction will properly revert the data Check that a failing transaction will properly revert the data
$ echo plakfe > a $ echo plakfe > a
$ f --size --sha256 .hg/store/00changelog-*.nd $ f --size --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !) .hg/store/00changelog-????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !)
.hg/store/00changelog-????????????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !) .hg/store/00changelog-????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !)
.hg/store/00changelog-????????????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !) .hg/store/00changelog-????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !)
$ hg ci -m a3 --config "extensions.abort=$RUNTESTDIR/testlib/crash_transaction_late.py" $ hg ci -m a3 --config "extensions.abort=$RUNTESTDIR/testlib/crash_transaction_late.py"
transaction abort! transaction abort!
rollback completed rollback completed
abort: This is a late abort abort: This is a late abort
[255] [255]
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????? (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121536 (pure !) data-length: 121536 (pure !)
data-length: 121536 (rust !) data-length: 121536 (rust !)
data-length: 121088 (no-pure no-rust !) data-length: 121088 (no-pure no-rust !)
data-unused: 448 (pure !) data-unused: 448 (pure !)
data-unused: 448 (rust !) data-unused: 448 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.369% (pure !) data-unused: 0.369% (pure !)
data-unused: 0.369% (rust !) data-unused: 0.369% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ f --size --sha256 .hg/store/00changelog-*.nd $ f --size --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !) .hg/store/00changelog-????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !)
.hg/store/00changelog-????????????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !) .hg/store/00changelog-????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !)
.hg/store/00changelog-????????????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !) .hg/store/00changelog-????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !)
Check that removing content does not confuse the nodemap Check that removing content does not confuse the nodemap
-------------------------------------------------------- --------------------------------------------------------
removing data with rollback removing data with rollback
$ echo aso > a $ echo aso > a
$ hg ci -m a4 $ hg ci -m a4
The persistent nodemap should exist after a streaming clone The persistent nodemap should exist after a streaming clone
Simple case Simple case
----------- -----------
No race condition No race condition
$ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)' $ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)'
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (62 bytes)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (62 bytes)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00manifest.d (491 KB) (zstd !) adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
$ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)' $ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
Check and record file state beforehand Check and record file state beforehand
$ f --size test-repo/.hg/store/00changelog* $ f --size test-repo/.hg/store/00changelog*
test-repo/.hg/store/00changelog-*.nd: size=121088 (glob) test-repo/.hg/store/00changelog-*.nd: size=121088 (glob)
test-repo/.hg/store/00changelog.d: size=376891 (zstd !) test-repo/.hg/store/00changelog.d: size=376891 (zstd !)
test-repo/.hg/store/00changelog.d: size=368890 (no-zstd !) test-repo/.hg/store/00changelog.d: size=368890 (no-zstd !)
test-repo/.hg/store/00changelog.i: size=320384 test-repo/.hg/store/00changelog.i: size=320384
test-repo/.hg/store/00changelog.n: size=70 test-repo/.hg/store/00changelog.n: size=62
$ hg -R test-repo debugnodemap --metadata | tee server-metadata.txt $ hg -R test-repo debugnodemap --metadata | tee server-metadata.txt
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
Prepare a commit Prepare a commit
$ echo foo >> test-repo/foo $ echo foo >> test-repo/foo
$ hg -R test-repo/ add test-repo/foo $ hg -R test-repo/ add test-repo/foo
Do a mix of clone and commit at the same time so that the file listed on disk differ at actual transfer time. Do a mix of clone and commit at the same time so that the file listed on disk differ at actual transfer time.
$ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | egrep '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) & $ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | egrep '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) &
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1
$ hg -R test-repo/ commit -m foo $ hg -R test-repo/ commit -m foo
$ touch $HG_TEST_STREAM_WALKED_FILE_2 $ touch $HG_TEST_STREAM_WALKED_FILE_2
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3
$ cat clone-output $ cat clone-output
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (62 bytes)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (62 bytes)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00manifest.d (491 KB) (zstd !) adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
Check the result state Check the result state
$ f --size stream-clone-race-1/.hg/store/00changelog* $ f --size stream-clone-race-1/.hg/store/00changelog*
stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob) stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob)
stream-clone-race-1/.hg/store/00changelog.d: size=368890 (no-zstd !) stream-clone-race-1/.hg/store/00changelog.d: size=368890 (no-zstd !)
stream-clone-race-1/.hg/store/00changelog.d: size=376891 (zstd !) stream-clone-race-1/.hg/store/00changelog.d: size=376891 (zstd !)
stream-clone-race-1/.hg/store/00changelog.i: size=320384 stream-clone-race-1/.hg/store/00changelog.i: size=320384
stream-clone-race-1/.hg/store/00changelog.n: size=70 stream-clone-race-1/.hg/store/00changelog.n: size=62
$ hg -R stream-clone-race-1 debugnodemap --metadata | tee client-metadata.txt $ hg -R stream-clone-race-1 debugnodemap --metadata | tee client-metadata.txt
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
$ f --size test-repo/.hg/store/00changelog* $ f --size test-repo/.hg/store/00changelog*
test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !) test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !)
test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !) test-repo/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !)
test-repo/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !) test-repo/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !)
test-repo/.hg/store/00changelog.d: size=376950 (zstd !) test-repo/.hg/store/00changelog.d: size=376950 (zstd !)
test-repo/.hg/store/00changelog.d: size=368949 (no-zstd !) test-repo/.hg/store/00changelog.d: size=368949 (no-zstd !)
test-repo/.hg/store/00changelog.i: size=320448 test-repo/.hg/store/00changelog.i: size=320448
test-repo/.hg/store/00changelog.n: size=70 test-repo/.hg/store/00changelog.n: size=62
$ hg -R test-repo debugnodemap --metadata | tee server-metadata-2.txt $ hg -R test-repo debugnodemap --metadata | tee server-metadata-2.txt
uid: * (glob) uid: * (glob)
tip-rev: 5006 tip-rev: 5006
tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b
data-length: 121344 (rust !) data-length: 121344 (rust !)
data-length: 121344 (pure !) data-length: 121344 (pure !)
data-length: 121152 (no-rust no-pure !) data-length: 121152 (no-rust no-pure !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
$ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-2 --debug 2>> clone-output-2 | egrep '00(changelog|manifest)' >> clone-output-2; touch $HG_TEST_STREAM_WALKED_FILE_3) & $ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-2 --debug 2>> clone-output-2 | egrep '00(changelog|manifest)' >> clone-output-2; touch $HG_TEST_STREAM_WALKED_FILE_3) &
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1
$ rm test-repo/.hg/store/00changelog.n $ rm test-repo/.hg/store/00changelog.n
$ rm test-repo/.hg/store/00changelog-*.nd $ rm test-repo/.hg/store/00changelog-*.nd
$ hg -R test-repo/ debugupdatecache $ hg -R test-repo/ debugupdatecache
$ touch $HG_TEST_STREAM_WALKED_FILE_2 $ touch $HG_TEST_STREAM_WALKED_FILE_2
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3
$ cat clone-output-2 $ cat clone-output-2
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (62 bytes)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (62 bytes)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.d (492 KB) (zstd !) adding [s] 00manifest.d (492 KB) (zstd !)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
Check the result. Check the result.
$ f --size stream-clone-race-2/.hg/store/00changelog* $ f --size stream-clone-race-2/.hg/store/00changelog*
stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !) stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (rust !)
stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !) stream-clone-race-2/.hg/store/00changelog-*.nd: size=121344 (glob) (pure !)
stream-clone-race-2/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !) stream-clone-race-2/.hg/store/00changelog-*.nd: size=121152 (glob) (no-rust no-pure !)
stream-clone-race-2/.hg/store/00changelog.d: size=376950 (zstd !) stream-clone-race-2/.hg/store/00changelog.d: size=376950 (zstd !)
stream-clone-race-2/.hg/store/00changelog.d: size=368949 (no-zstd !) stream-clone-race-2/.hg/store/00changelog.d: size=368949 (no-zstd !)
stream-clone-race-2/.hg/store/00changelog.i: size=320448 stream-clone-race-2/.hg/store/00changelog.i: size=320448
stream-clone-race-2/.hg/store/00changelog.n: size=70 stream-clone-race-2/.hg/store/00changelog.n: size=62
$ hg -R stream-clone-race-2 debugnodemap --metadata | tee client-metadata-2.txt $ hg -R stream-clone-race-2 debugnodemap --metadata | tee client-metadata-2.txt
uid: * (glob) uid: * (glob)
tip-rev: 5006 tip-rev: 5006
tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b tip-node: ed2ec1eef9aa2a0ec5057c51483bc148d03e810b
data-length: 121344 (rust !) data-length: 121344 (rust !)
data-unused: 192 (rust !) data-unused: 192 (rust !)
data-unused: 0.158% (rust !) data-unused: 0.158% (rust !)