This is an archive of the discontinued Mercurial Phabricator instance.

Differential D12029

stream-clone: add a explicit test for format change during stream clone
ClosedPublic

Authored by marmoute on Jan 24 2022, 6:34 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGa6f16ec07ed7: stream-clone: add a explicit test for format change during stream clone
Summary

They are different kind of requirements, the one which impact the data storage
and are relevant to the files being streamed and the one which does not. For
example some requirements are only relevant to the working copy, like sparse, or
dirstate-v2.

Since they are irrelevant to the content being streamed, they do not prevent the
receiving side to use streaming clone and mercurial skip adverting them over
the wire and, ideally, within the bundle.

In addition, this let the client decide to use whichever format it desire for
the part that does not affect the store itself. So the configuration related to
these format are used as normal when doing a streaming clone.

In practice, the feature was not really tested and is badly broken with bundle-2,
since the requirements are not filtered out from the stream bundle.

So we start with adding simple tests as a good base before the fix and adjust
the feature.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Revision Contents
Changeset List

Diff 31905

tests/test-clone-stream-format.t

  • This file was added.
This file contains tests case that deal with format change accross stream clone
#require serve no-reposimplestore no-chg
#testcases stream-legacy
(the #stream-bundle2 variant is actually buggy for the moment)
#if stream-legacy
$ cat << EOF >> $HGRCPATH
> [server]
> bundle2.stream = no
> EOF
#endif
Initialize repository
$ hg init server
$ cd server
$ sh $TESTDIR/testlib/stream_clone_setup.sh
adding 00changelog-ab349180a0405010.nd
adding 00changelog.d
adding 00changelog.i
adding 00changelog.n
adding 00manifest.d
adding 00manifest.i
adding container/isam-build-centos7/bazel-coverage-generator-sandboxfs-compatibility-0758e3e4f6057904d44399bd666faba9e7f40686.patch
adding data/foo.d
adding data/foo.i
adding data/foo.n
adding data/undo.babar
adding data/undo.d
adding data/undo.foo.d
adding data/undo.foo.i
adding data/undo.foo.n
adding data/undo.i
adding data/undo.n
adding data/undo.py
adding foo.d
adding foo.i
adding foo.n
adding meta/foo.d
adding meta/foo.i
adding meta/foo.n
adding meta/undo.babar
adding meta/undo.d
adding meta/undo.foo.d
adding meta/undo.foo.i
adding meta/undo.foo.n
adding meta/undo.i
adding meta/undo.n
adding meta/undo.py
adding savanah/foo.d
adding savanah/foo.i
adding savanah/foo.n
adding savanah/undo.babar
adding savanah/undo.d
adding savanah/undo.foo.d
adding savanah/undo.foo.i
adding savanah/undo.foo.n
adding savanah/undo.i
adding savanah/undo.n
adding savanah/undo.py
adding store/C\xc3\xa9lesteVille_is_a_Capital_City (esc)
adding store/foo.d
adding store/foo.i
adding store/foo.n
adding store/undo.babar
adding store/undo.d
adding store/undo.foo.d
adding store/undo.foo.i
adding store/undo.foo.n
adding store/undo.i
adding store/undo.n
adding store/undo.py
adding undo.babar
adding undo.d
adding undo.foo.d
adding undo.foo.i
adding undo.foo.n
adding undo.i
adding undo.n
adding undo.py
$ cd ..
Test streaming from/to repository without a store:
==================================================
$ hg clone --pull --config format.usestore=no server server-no-store
requesting all changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 1088 changes to 1088 files
new changesets 96ee1d7354c4:5223b5e3265f
updating to branch default
1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg verify -R server-no-store
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg -R server serve -p $HGPORT -d --pid-file=hg-1.pid --error errors-1.txt
$ cat hg-1.pid > $DAEMON_PIDS
$ hg -R server-no-store serve -p $HGPORT2 -d --pid-file=hg-2.pid --error errors-2.txt
$ cat hg-2.pid >> $DAEMON_PIDS
$ hg debugrequires -R server | grep store
store
$ hg debugrequires -R server-no-store | grep store
[1]
store → no-store cloning
$ hg clone --quiet --stream -U http://localhost:$HGPORT clone-remove-store --config format.usestore=no
$ cat errors-1.txt
$ hg -R clone-remove-store verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg debugrequires -R clone-remove-store | grep store
[1]
no-store → store cloning
$ hg clone --quiet --stream -U http://localhost:$HGPORT2 clone-add-store --config format.usestore=yes
$ cat errors-2.txt
$ hg -R clone-add-store verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg debugrequires -R clone-add-store | grep store
store
$ killdaemons.py
Test streaming from/to repository without a fncache
===================================================
$ rm hg-*.pid errors-*.txt
$ hg clone --pull --config format.usefncache=no server server-no-fncache
requesting all changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 1088 changes to 1088 files
new changesets 96ee1d7354c4:5223b5e3265f
updating to branch default
1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg verify -R server-no-fncache
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg -R server serve -p $HGPORT -d --pid-file=hg-1.pid --error errors-1.txt
$ cat hg-1.pid > $DAEMON_PIDS
$ hg -R server-no-fncache serve -p $HGPORT2 -d --pid-file=hg-2.pid --error errors-2.txt
$ cat hg-2.pid >> $DAEMON_PIDS
$ hg debugrequires -R server | grep fncache
fncache
$ hg debugrequires -R server-no-fncache | grep fncache
[1]
fncache → no-fncache cloning
$ hg clone --quiet --stream -U http://localhost:$HGPORT clone-remove-fncache --config format.usefncache=no
$ cat errors-1.txt
$ hg -R clone-remove-fncache verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg debugrequires -R clone-remove-fncache | grep fncache
[1]
no-fncache → fncache cloning
$ hg clone --quiet --stream -U http://localhost:$HGPORT2 clone-add-fncache --config format.usefncache=yes
$ cat errors-2.txt
$ hg -R clone-add-fncache verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg debugrequires -R clone-add-fncache | grep fncache
fncache
$ killdaemons.py
Test streaming from/to repository without a dotencode
===================================================
$ rm hg-*.pid errors-*.txt
$ hg clone --pull --config format.dotencode=no server server-no-dotencode
requesting all changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 1088 changes to 1088 files
new changesets 96ee1d7354c4:5223b5e3265f
updating to branch default
1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg verify -R server-no-dotencode
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg -R server serve -p $HGPORT -d --pid-file=hg-1.pid --error errors-1.txt
$ cat hg-1.pid > $DAEMON_PIDS
$ hg -R server-no-dotencode serve -p $HGPORT2 -d --pid-file=hg-2.pid --error errors-2.txt
$ cat hg-2.pid >> $DAEMON_PIDS
$ hg debugrequires -R server | grep dotencode
dotencode
$ hg debugrequires -R server-no-dotencode | grep dotencode
[1]
dotencode → no-dotencode cloning
$ hg clone --quiet --stream -U http://localhost:$HGPORT clone-remove-dotencode --config format.dotencode=no
$ cat errors-1.txt
$ hg -R clone-remove-dotencode verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg debugrequires -R clone-remove-dotencode | grep dotencode
[1]
no-dotencode → dotencode cloning
$ hg clone --quiet --stream -U http://localhost:$HGPORT2 clone-add-dotencode --config format.dotencode=yes
$ cat errors-2.txt
$ hg -R clone-add-dotencode verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg debugrequires -R clone-add-dotencode | grep dotencode
dotencode
$ killdaemons.py
Cloning from a share
--------------------
We should be able to clone from a "share" repository, it will use the source store for streaming.
The resulting clone should not use share.
$ rm hg-*.pid errors-*.txt
$ hg share --config extensions.share= server server-share -U
$ hg -R server-share serve -p $HGPORT -d --pid-file=hg-1.pid --error errors-1.txt
$ cat hg-1.pid > $DAEMON_PIDS
$ hg clone --quiet --stream -U http://localhost:$HGPORT clone-from-share
$ hg -R clone-from-share verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 3 changesets with 1088 changes to 1088 files
$ hg debugrequires -R clone-from-share | grep share
[1]
$ killdaemons.py

tests/test-clone-stream.t

#require serve no-reposimplestore no-chg #require serve no-reposimplestore no-chg
#testcases stream-legacy stream-bundle2 #testcases stream-legacy stream-bundle2
#if stream-legacy #if stream-legacy
$ cat << EOF >> $HGRCPATH $ cat << EOF >> $HGRCPATH
> [server] > [server]
> bundle2.stream = no > bundle2.stream = no
> EOF > EOF
#endif #endif
Initialize repository Initialize repository
the status call is to check for issue5130
$ hg init server $ hg init server
$ cd server $ cd server
$ touch foo $ sh $TESTDIR/testlib/stream_clone_setup.sh
$ hg -q commit -A -m initial
>>> for i in range(1024):
... with open(str(i), 'wb') as fh:
... fh.write(b"%d" % i) and None
$ hg -q commit -A -m 'add a lot of files'
$ hg st
add files with "tricky" name:
$ echo foo > 00changelog.i
$ echo foo > 00changelog.d
$ echo foo > 00changelog.n
$ echo foo > 00changelog-ab349180a0405010.nd
$ echo foo > 00manifest.i
$ echo foo > 00manifest.d
$ echo foo > foo.i
$ echo foo > foo.d
$ echo foo > foo.n
$ echo foo > undo.py
$ echo foo > undo.i
$ echo foo > undo.d
$ echo foo > undo.n
$ echo foo > undo.foo.i
$ echo foo > undo.foo.d
$ echo foo > undo.foo.n
$ echo foo > undo.babar
$ mkdir savanah
$ echo foo > savanah/foo.i
$ echo foo > savanah/foo.d
$ echo foo > savanah/foo.n
$ echo foo > savanah/undo.py
$ echo foo > savanah/undo.i
$ echo foo > savanah/undo.d
$ echo foo > savanah/undo.n
$ echo foo > savanah/undo.foo.i
$ echo foo > savanah/undo.foo.d
$ echo foo > savanah/undo.foo.n
$ echo foo > savanah/undo.babar
$ mkdir data
$ echo foo > data/foo.i
$ echo foo > data/foo.d
$ echo foo > data/foo.n
$ echo foo > data/undo.py
$ echo foo > data/undo.i
$ echo foo > data/undo.d
$ echo foo > data/undo.n
$ echo foo > data/undo.foo.i
$ echo foo > data/undo.foo.d
$ echo foo > data/undo.foo.n
$ echo foo > data/undo.babar
$ mkdir meta
$ echo foo > meta/foo.i
$ echo foo > meta/foo.d
$ echo foo > meta/foo.n
$ echo foo > meta/undo.py
$ echo foo > meta/undo.i
$ echo foo > meta/undo.d
$ echo foo > meta/undo.n
$ echo foo > meta/undo.foo.i
$ echo foo > meta/undo.foo.d
$ echo foo > meta/undo.foo.n
$ echo foo > meta/undo.babar
$ mkdir store
$ echo foo > store/foo.i
$ echo foo > store/foo.d
$ echo foo > store/foo.n
$ echo foo > store/undo.py
$ echo foo > store/undo.i
$ echo foo > store/undo.d
$ echo foo > store/undo.n
$ echo foo > store/undo.foo.i
$ echo foo > store/undo.foo.d
$ echo foo > store/undo.foo.n
$ echo foo > store/undo.babar
Name with special characters
$ echo foo > store/CélesteVille_is_a_Capital_City
name causing issue6581
$ mkdir -p container/isam-build-centos7/
$ touch container/isam-build-centos7/bazel-coverage-generator-sandboxfs-compatibility-0758e3e4f6057904d44399bd666faba9e7f40686.patch
Add all that
$ hg add .
adding 00changelog-ab349180a0405010.nd adding 00changelog-ab349180a0405010.nd
adding 00changelog.d adding 00changelog.d
adding 00changelog.i adding 00changelog.i
adding 00changelog.n adding 00changelog.n
adding 00manifest.d adding 00manifest.d
adding 00manifest.i adding 00manifest.i
adding container/isam-build-centos7/bazel-coverage-generator-sandboxfs-compatibility-0758e3e4f6057904d44399bd666faba9e7f40686.patch adding container/isam-build-centos7/bazel-coverage-generator-sandboxfs-compatibility-0758e3e4f6057904d44399bd666faba9e7f40686.patch
adding data/foo.d adding data/foo.d
adding undo.babar adding undo.babar
adding undo.d adding undo.d
adding undo.foo.d adding undo.foo.d
adding undo.foo.i adding undo.foo.i
adding undo.foo.n adding undo.foo.n
adding undo.i adding undo.i
adding undo.n adding undo.n
adding undo.py adding undo.py
$ hg ci -m 'add files with "tricky" name'
$ hg --config server.uncompressed=false serve -p $HGPORT -d --pid-file=hg.pid $ hg --config server.uncompressed=false serve -p $HGPORT -d --pid-file=hg.pid
$ cat hg.pid > $DAEMON_PIDS $ cat hg.pid > $DAEMON_PIDS
$ cd .. $ cd ..
Check local clone Check local clone
================== ==================
The logic is close enough of uncompressed. The logic is close enough of uncompressed.

tests/testlib/stream_clone_setup.sh

  • This file was added.
# setup some files and commit for a good stream clone testing.
touch foo
hg -q commit -A -m initial
python3 << EOF
for i in range(1024):
with open(str(i), 'wb') as fh:
fh.write(b"%d" % i) and None
EOF
hg -q commit -A -m 'add a lot of files'
# (the status call is to check for issue5130)
hg st
# add files with "tricky" name:
echo foo > 00changelog.i
echo foo > 00changelog.d
echo foo > 00changelog.n
echo foo > 00changelog-ab349180a0405010.nd
echo foo > 00manifest.i
echo foo > 00manifest.d
echo foo > foo.i
echo foo > foo.d
echo foo > foo.n
echo foo > undo.py
echo foo > undo.i
echo foo > undo.d
echo foo > undo.n
echo foo > undo.foo.i
echo foo > undo.foo.d
echo foo > undo.foo.n
echo foo > undo.babar
mkdir savanah
echo foo > savanah/foo.i
echo foo > savanah/foo.d
echo foo > savanah/foo.n
echo foo > savanah/undo.py
echo foo > savanah/undo.i
echo foo > savanah/undo.d
echo foo > savanah/undo.n
echo foo > savanah/undo.foo.i
echo foo > savanah/undo.foo.d
echo foo > savanah/undo.foo.n
echo foo > savanah/undo.babar
mkdir data
echo foo > data/foo.i
echo foo > data/foo.d
echo foo > data/foo.n
echo foo > data/undo.py
echo foo > data/undo.i
echo foo > data/undo.d
echo foo > data/undo.n
echo foo > data/undo.foo.i
echo foo > data/undo.foo.d
echo foo > data/undo.foo.n
echo foo > data/undo.babar
mkdir meta
echo foo > meta/foo.i
echo foo > meta/foo.d
echo foo > meta/foo.n
echo foo > meta/undo.py
echo foo > meta/undo.i
echo foo > meta/undo.d
echo foo > meta/undo.n
echo foo > meta/undo.foo.i
echo foo > meta/undo.foo.d
echo foo > meta/undo.foo.n
echo foo > meta/undo.babar
mkdir store
echo foo > store/foo.i
echo foo > store/foo.d
echo foo > store/foo.n
echo foo > store/undo.py
echo foo > store/undo.i
echo foo > store/undo.d
echo foo > store/undo.n
echo foo > store/undo.foo.i
echo foo > store/undo.foo.d
echo foo > store/undo.foo.n
echo foo > store/undo.babar
# Name with special characters
echo foo > store/CélesteVille_is_a_Capital_City
# name causing issue6581
mkdir -p container/isam-build-centos7/
touch container/isam-build-centos7/bazel-coverage-generator-sandboxfs-compatibility-0758e3e4f6057904d44399bd666faba9e7f40686.patch
# Add all that
hg add .
hg ci -m 'add files with "tricky" name'