This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10479

nodemap: add a test about racy commit during stream clone
ClosedPublic

Authored by marmoute on Apr 20 2021, 12:09 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGd70319c3ca14: nodemap: add a test about racy commit during stream clone
Summary

That test show that the resulting client nodemap is different from the server
one. This happens because the server one transferred a corrupted node map. The
data file match the pre-commit content while the docket has post commit content.
As the result the nodemap was detected invalid and recomputed.

When running without the rust implementation, the code is also generating a new
datafile unconditionally, This mean the older file is no longer there are
transfer time, resulting in a crash.

We will fix this issue later, but we start with writing tests highlighting the
issue.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Apr 20 2021, 12:09 AM
Alphare accepted this revision.Apr 20 2021, 4:33 AM
marmoute added a commit: rHGd70319c3ca14: nodemap: add a test about racy commit during stream clone.Apr 20 2021, 10:19 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGd70319c3ca14: nodemap: add a test about racy commit during stream clone (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mtests/test-persistent-nodemap.t (132 lines)

Diff 27120

tests/test-persistent-nodemap.t

tag: tip tag: tip
user: test user: test
date: Thu Jan 01 00:00:00 1970 +0000 date: Thu Jan 01 00:00:00 1970 +0000
summary: a2 summary: a2
stream clone stream clone
------------ ============
The persistent nodemap should exist after a streaming clone The persistent nodemap should exist after a streaming clone
Simple case
-----------
No race condition
$ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)' $ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)'
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (70 bytes)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00manifest.d (491 KB) (zstd !) adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (70 bytes)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
$ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)' $ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
00changelog-*.nd (glob) 00changelog-*.nd (glob)
00changelog.n 00changelog.n
00manifest-*.nd (glob) 00manifest-*.nd (glob)
00manifest.n 00manifest.n
$ hg -R stream-clone debugnodemap --metadata $ hg -R stream-clone debugnodemap --metadata
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
new data appened
-----------------
Other commit happening on the server during the stream clone
setup the step-by-step stream cloning
$ HG_TEST_STREAM_WALKED_FILE_1="$TESTTMP/sync_file_walked_1"
$ export HG_TEST_STREAM_WALKED_FILE_1
$ HG_TEST_STREAM_WALKED_FILE_2="$TESTTMP/sync_file_walked_2"
$ export HG_TEST_STREAM_WALKED_FILE_2
$ HG_TEST_STREAM_WALKED_FILE_3="$TESTTMP/sync_file_walked_3"
$ export HG_TEST_STREAM_WALKED_FILE_3
$ cat << EOF >> test-repo/.hg/hgrc
> [extensions]
> steps=$RUNTESTDIR/testlib/ext-stream-clone-steps.py
> EOF
Check and record file state beforehand
$ f --size test-repo/.hg/store/00changelog*
test-repo/.hg/store/00changelog-*.nd: size=121088 (glob)
test-repo/.hg/store/00changelog.d: size=376891 (zstd !)
test-repo/.hg/store/00changelog.d: size=368890 (no-zstd !)
test-repo/.hg/store/00changelog.i: size=320384
test-repo/.hg/store/00changelog.n: size=70
$ hg -R test-repo debugnodemap --metadata | tee server-metadata.txt
uid: * (glob)
tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088
data-unused: 0
data-unused: 0.000%
Prepare a commit
$ echo foo >> test-repo/foo
$ hg -R test-repo/ add test-repo/foo
Do a mix of clone and commit at the same time so that the file listed on disk differ at actual transfer time.
$ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | egrep '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) &
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1
$ hg -R test-repo/ commit -m foo
$ touch $HG_TEST_STREAM_WALKED_FILE_2
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3
$ cat clone-output
remote: abort: unexpected error: [Errno 2] $ENOENT$: *'$TESTTMP/test-repo/.hg/store/00manifest-*.nd' (glob) (known-bad-output no-rust no-pure !)
abort: pull failed on remote (known-bad-output no-rust no-pure !)
adding [s] 00manifest.n (70 bytes)
adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00manifest.d (452 KB) (no-zstd !)
remote: abort: $ENOENT$: '$TESTTMP/test-repo/.hg/store/00manifest-*.nd' (glob) (known-bad-output no-rust no-pure !)
adding [s] 00manifest-*.nd (118 KB) (glob) (rust !)
adding [s] 00changelog.n (70 bytes) (rust !)
adding [s] 00changelog.d (368 KB) (zstd rust !)
adding [s] 00changelog-*.nd (118 KB) (glob) (rust !)
adding [s] 00manifest.i (313 KB) (rust !)
adding [s] 00changelog.i (313 KB) (rust !)
adding [s] 00manifest-*.nd (118 KB) (glob) (pure !)
adding [s] 00changelog.n (70 bytes) (pure !)
adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog-*.nd (118 KB) (glob) (pure !)
adding [s] 00manifest.i (313 KB) (pure !)
adding [s] 00changelog.i (313 KB) (pure !)
Check the result state
$ f --size stream-clone-race-1/.hg/store/00changelog*
stream-clone-race-1/.hg/store/00changelog*: file not found (known-bad-output no-rust no-pure !)
stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob) (rust !)
stream-clone-race-1/.hg/store/00changelog.d: size=376891 (zstd rust !)
stream-clone-race-1/.hg/store/00changelog.i: size=320384 (rust !)
stream-clone-race-1/.hg/store/00changelog.n: size=70 (rust !)
stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob) (pure !)
stream-clone-race-1/.hg/store/00changelog.d: size=368890 (no-zstd pure !)
stream-clone-race-1/.hg/store/00changelog.i: size=320384 (pure !)
stream-clone-race-1/.hg/store/00changelog.n: size=70 (pure !)
$ hg -R stream-clone-race-1 debugnodemap --metadata | tee client-metadata.txt
abort: repository stream-clone-race-1 not found (known-bad-output no-rust no-pure !)
uid: * (glob) (rust !)
tip-rev: 5005 (rust !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (rust !)
data-length: 121088 (rust !)
data-unused: 0 (rust !)
data-unused: 0.000% (rust !)
uid: * (glob) (pure !)
tip-rev: 5005 (pure !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (pure !)
data-length: 121088 (pure !)
data-unused: 0 (pure !)
data-unused: 0.000% (pure !)
We get a usable nodemap, so no rewrite would be needed and the metadata should be identical
(ie: the following diff should be empty)
$ diff -u server-metadata.txt client-metadata.txt
--- server-metadata.txt * (glob) (known-bad-output !)
+++ client-metadata.txt * (glob) (known-bad-output !)
@@ -1,4 +1,4 @@ (known-bad-output rust !)
@@ -1,4 +1,4 @@ (known-bad-output pure !)
@@ -1,6 +0,0 @@ (known-bad-output no-rust no-pure !)
-uid: * (glob) (known-bad-output !)
+uid: * (glob) (known-bad-output rust !)
tip-rev: 5005 (known-bad-output rust !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (known-bad-output rust !)
data-length: 121088 (known-bad-output rust !)
+uid: * (glob) (known-bad-output pure !)
tip-rev: 5005 (known-bad-output pure !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (known-bad-output pure !)
data-length: 121088 (known-bad-output pure !)
-tip-rev: 5005 (known-bad-output no-rust no-pure !)
-tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (known-bad-output no-rust no-pure !)
-data-length: 121088 (known-bad-output no-rust no-pure !)
-data-unused: 0 (known-bad-output no-rust no-pure !)
-data-unused: 0.000% (known-bad-output no-rust no-pure !)
[1]
Clean up after the test.
$ rm -f "$HG_TEST_STREAM_WALKED_FILE_1"
$ rm -f "$HG_TEST_STREAM_WALKED_FILE_2"
$ rm -f "$HG_TEST_STREAM_WALKED_FILE_3"