This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10479

nodemap: add a test about racy commit during stream clone
ClosedPublic

Authored by marmoute on Apr 20 2021, 12:09 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGd70319c3ca14: nodemap: add a test about racy commit during stream clone
Summary

That test show that the resulting client nodemap is different from the server
one. This happens because the server one transferred a corrupted node map. The
data file match the pre-commit content while the docket has post commit content.
As the result the nodemap was detected invalid and recomputed.

When running without the rust implementation, the code is also generating a new
datafile unconditionally, This mean the older file is no longer there are
transfer time, resulting in a crash.

We will fix this issue later, but we start with writing tests highlighting the
issue.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

marmoute created this revision.Apr 20 2021, 12:09 AM
Alphare accepted this revision.Apr 20 2021, 4:33 AM
marmoute added a commit: rHGd70319c3ca14: nodemap: add a test about racy commit during stream clone.Apr 20 2021, 10:19 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGd70319c3ca14: nodemap: add a test about racy commit during stream clone (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mtests/test-persistent-nodemap.t (132 lines)

Diff 27075

tests/test-persistent-nodemap.t

tag: tip tag: tip
user: test user: test
date: Thu Jan 01 00:00:00 1970 +0000 date: Thu Jan 01 00:00:00 1970 +0000
summary: a2 summary: a2
stream clone stream clone
------------ ============
The persistent nodemap should exist after a streaming clone The persistent nodemap should exist after a streaming clone
Simple case
-----------
No race condition
$ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)' $ hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone --debug | egrep '00(changelog|manifest)'
adding [s] 00manifest.n (70 bytes) adding [s] 00manifest.n (70 bytes)
adding [s] 00manifest.d (452 KB) (no-zstd !) adding [s] 00manifest.d (452 KB) (no-zstd !)
adding [s] 00manifest.d (491 KB) (zstd !) adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00manifest-*.nd (118 KB) (glob) adding [s] 00manifest-*.nd (118 KB) (glob)
adding [s] 00changelog.n (70 bytes) adding [s] 00changelog.n (70 bytes)
adding [s] 00changelog.d (360 KB) (no-zstd !) adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog.d (368 KB) (zstd !) adding [s] 00changelog.d (368 KB) (zstd !)
adding [s] 00changelog-*.nd (118 KB) (glob) adding [s] 00changelog-*.nd (118 KB) (glob)
adding [s] 00manifest.i (313 KB) adding [s] 00manifest.i (313 KB)
adding [s] 00changelog.i (313 KB) adding [s] 00changelog.i (313 KB)
$ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)' $ ls -1 stream-clone/.hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
00changelog-*.nd (glob) 00changelog-*.nd (glob)
00changelog.n 00changelog.n
00manifest-*.nd (glob) 00manifest-*.nd (glob)
00manifest.n 00manifest.n
$ hg -R stream-clone debugnodemap --metadata $ hg -R stream-clone debugnodemap --metadata
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%
new data appened
-----------------
Other commit happening on the server during the stream clone
setup the step-by-step stream cloning
$ HG_TEST_STREAM_WALKED_FILE_1="$TESTTMP/sync_file_walked_1"
$ export HG_TEST_STREAM_WALKED_FILE_1
$ HG_TEST_STREAM_WALKED_FILE_2="$TESTTMP/sync_file_walked_2"
$ export HG_TEST_STREAM_WALKED_FILE_2
$ HG_TEST_STREAM_WALKED_FILE_3="$TESTTMP/sync_file_walked_3"
$ export HG_TEST_STREAM_WALKED_FILE_3
$ cat << EOF >> test-repo/.hg/hgrc
> [extensions]
> steps=$RUNTESTDIR/testlib/ext-stream-clone-steps.py
> EOF
Check and record file state beforehand
$ f --size test-repo/.hg/store/00changelog*
test-repo/.hg/store/00changelog-*.nd: size=121088 (glob)
test-repo/.hg/store/00changelog.d: size=376891 (zstd !)
test-repo/.hg/store/00changelog.d: size=368890 (no-zstd !)
test-repo/.hg/store/00changelog.i: size=320384
test-repo/.hg/store/00changelog.n: size=70
$ hg -R test-repo debugnodemap --metadata | tee server-metadata.txt
uid: * (glob)
tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088
data-unused: 0
data-unused: 0.000%
Prepare a commit
$ echo foo >> test-repo/foo
$ hg -R test-repo/ add test-repo/foo
Do a mix of clone and commit at the same time so that the file listed on disk differ at actual transfer time.
$ (hg clone -U --stream --config ui.ssh="\"$PYTHON\" \"$TESTDIR/dummyssh\"" ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | egrep '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) &
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1
$ hg -R test-repo/ commit -m foo
$ touch $HG_TEST_STREAM_WALKED_FILE_2
$ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3
$ cat clone-output
remote: abort: unexpected error: [Errno 2] $ENOENT$: *'$TESTTMP/test-repo/.hg/store/00manifest-*.nd' (glob) (known-bad-output no-rust no-pure !)
abort: pull failed on remote (known-bad-output no-rust no-pure !)
adding [s] 00manifest.n (70 bytes)
adding [s] 00manifest.d (491 KB) (zstd !)
adding [s] 00manifest.d (452 KB) (no-zstd !)
remote: abort: $ENOENT$: '$TESTTMP/test-repo/.hg/store/00manifest-*.nd' (glob) (known-bad-output no-rust no-pure !)
adding [s] 00manifest-*.nd (118 KB) (glob) (rust !)
adding [s] 00changelog.n (70 bytes) (rust !)
adding [s] 00changelog.d (368 KB) (zstd rust !)
adding [s] 00changelog-*.nd (118 KB) (glob) (rust !)
adding [s] 00manifest.i (313 KB) (rust !)
adding [s] 00changelog.i (313 KB) (rust !)
adding [s] 00manifest-*.nd (118 KB) (glob) (pure !)
adding [s] 00changelog.n (70 bytes) (pure !)
adding [s] 00changelog.d (360 KB) (no-zstd !)
adding [s] 00changelog-*.nd (118 KB) (glob) (pure !)
adding [s] 00manifest.i (313 KB) (pure !)
adding [s] 00changelog.i (313 KB) (pure !)
Check the result state
$ f --size stream-clone-race-1/.hg/store/00changelog*
stream-clone-race-1/.hg/store/00changelog*: file not found (known-bad-output no-rust no-pure !)
stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob) (rust !)
stream-clone-race-1/.hg/store/00changelog.d: size=376891 (zstd rust !)
stream-clone-race-1/.hg/store/00changelog.i: size=320384 (rust !)
stream-clone-race-1/.hg/store/00changelog.n: size=70 (rust !)
stream-clone-race-1/.hg/store/00changelog-*.nd: size=121088 (glob) (pure !)
stream-clone-race-1/.hg/store/00changelog.d: size=368890 (no-zstd pure !)
stream-clone-race-1/.hg/store/00changelog.i: size=320384 (pure !)
stream-clone-race-1/.hg/store/00changelog.n: size=70 (pure !)
$ hg -R stream-clone-race-1 debugnodemap --metadata | tee client-metadata.txt
abort: repository stream-clone-race-1 not found (known-bad-output no-rust no-pure !)
uid: * (glob) (rust !)
tip-rev: 5005 (rust !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (rust !)
data-length: 121088 (rust !)
data-unused: 0 (rust !)
data-unused: 0.000% (rust !)
uid: * (glob) (pure !)
tip-rev: 5005 (pure !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (pure !)
data-length: 121088 (pure !)
data-unused: 0 (pure !)
data-unused: 0.000% (pure !)
We get a usable nodemap, so no rewrite would be needed and the metadata should be identical
(ie: the following diff should be empty)
$ diff -u server-metadata.txt client-metadata.txt
--- server-metadata.txt * (glob) (known-bad-output !)
+++ client-metadata.txt * (glob) (known-bad-output !)
@@ -1,4 +1,4 @@ (known-bad-output rust !)
@@ -1,4 +1,4 @@ (known-bad-output pure !)
@@ -1,6 +0,0 @@ (known-bad-output no-rust no-pure !)
-uid: * (glob) (known-bad-output !)
+uid: * (glob) (known-bad-output rust !)
tip-rev: 5005 (known-bad-output rust !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (known-bad-output rust !)
data-length: 121088 (known-bad-output rust !)
+uid: * (glob) (known-bad-output pure !)
tip-rev: 5005 (known-bad-output pure !)
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (known-bad-output pure !)
data-length: 121088 (known-bad-output pure !)
-tip-rev: 5005 (known-bad-output no-rust no-pure !)
-tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe (known-bad-output no-rust no-pure !)
-data-length: 121088 (known-bad-output no-rust no-pure !)
-data-unused: 0 (known-bad-output no-rust no-pure !)
-data-unused: 0.000% (known-bad-output no-rust no-pure !)
[1]
Clean up after the test.
$ rm -f "$HG_TEST_STREAM_WALKED_FILE_1"
$ rm -f "$HG_TEST_STREAM_WALKED_FILE_2"
$ rm -f "$HG_TEST_STREAM_WALKED_FILE_3"