( )⚙ D12084 stream-clone: also filter the requirement we put in the bundle 2

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D12084

stream-clone: also filter the requirement we put in the bundle 2
ClosedPublic

Authored by marmoute on Jan 24 2022, 9:51 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGa3cf460a6b1b: stream-clone: also filter the requirement we put in the bundle 2
Summary

We were wrongly putting irrelevant requirements in the bundle and the receiving
side was getting confused, treating them as being missing while still putting
them in the requires file. Leading do corrupted repositories.

This changes fix stream-clone behavior regarding format when bundle-2 is
involved, so we now also test this cases.

Behavior with older version of Mercurial will be fine as they filter the
requirements they get from the bundle on their side anyway.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Jan 24 2022, 9:51 AM
Alphare accepted this revision.Jan 24 2022, 11:19 AM
This revision is now accepted and ready to land.Jan 24 2022, 11:19 AM
baymax updated this revision to Diff 31849.Jan 24 2022, 1:11 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Revision Contents
Changeset List

Diff 31960

mercurial/bundle2.py

) )
) )
elif repo.obsstore._version in remoteversions: elif repo.obsstore._version in remoteversions:
includeobsmarkers = True includeobsmarkers = True
filecount, bytecount, it = streamclone.generatev2( filecount, bytecount, it = streamclone.generatev2(
repo, includepats, excludepats, includeobsmarkers repo, includepats, excludepats, includeobsmarkers
) )
requirements = _formatrequirementsspec(repo.requirements) requirements = streamclone.streamed_requirements(repo)
requirements = _formatrequirementsspec(requirements)
part = bundler.newpart(b'stream2', data=it) part = bundler.newpart(b'stream2', data=it)
part.addparam(b'bytecount', b'%d' % bytecount, mandatory=True) part.addparam(b'bytecount', b'%d' % bytecount, mandatory=True)
part.addparam(b'filecount', b'%d' % filecount, mandatory=True) part.addparam(b'filecount', b'%d' % filecount, mandatory=True)
part.addparam(b'requirements', requirements, mandatory=True) part.addparam(b'requirements', requirements, mandatory=True)
def buildobsmarkerspart(bundler, markers, mandatory=True): def buildobsmarkerspart(bundler, markers, mandatory=True):
"""add an obsmarker part to the bundler with <markers> """add an obsmarker part to the bundler with <markers>

tests/test-clone-stream-format.t

This file contains tests case that deal with format change accross stream clone This file contains tests case that deal with format change accross stream clone
#require serve no-reposimplestore no-chg #require serve no-reposimplestore no-chg
#testcases stream-legacy #testcases stream-legacy stream-bundle2
(the #stream-bundle2 variant is actually buggy for the moment)
#if stream-legacy #if stream-legacy
$ cat << EOF >> $HGRCPATH $ cat << EOF >> $HGRCPATH
> [server] > [server]
> bundle2.stream = no > bundle2.stream = no
> EOF > EOF
#endif #endif

tests/test-clone-stream.t

$ get-with-headers.py $LOCALIP:$HGPORT '?cmd=getbundle' content-type --bodyfile body --hgproto '0.1 0.2 comp=zlib,none' --requestheader "x-hgarg-1=bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=0&common=0000000000000000000000000000000000000000&heads=c17445101a72edac06facd130d14808dfbd5c7c2&stream=1" $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=getbundle' content-type --bodyfile body --hgproto '0.1 0.2 comp=zlib,none' --requestheader "x-hgarg-1=bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=0&common=0000000000000000000000000000000000000000&heads=c17445101a72edac06facd130d14808dfbd5c7c2&stream=1"
200 Script output follows 200 Script output follows
content-type: application/mercurial-0.2 content-type: application/mercurial-0.2
#if no-zstd no-rust #if no-zstd no-rust
$ f --size --hex --bytes 256 body $ f --size --hex --bytes 256 body
body: size=119153 body: size=119123
0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......| 0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
0010: 80 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2.......| 0010: 62 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |b.STREAM2.......|
0020: 06 09 04 0c 44 62 79 74 65 63 6f 75 6e 74 31 30 |....Dbytecount10| 0020: 06 09 04 0c 26 62 79 74 65 63 6f 75 6e 74 31 30 |....&bytecount10|
0030: 34 31 31 35 66 69 6c 65 63 6f 75 6e 74 31 30 39 |4115filecount109| 0030: 34 31 31 35 66 69 6c 65 63 6f 75 6e 74 31 30 39 |4115filecount109|
0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 64 6f 74 |3requirementsdot| 0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 67 65 6e |3requirementsgen|
0050: 65 6e 63 6f 64 65 25 32 43 66 6e 63 61 63 68 65 |encode%2Cfncache| 0050: 65 72 61 6c 64 65 6c 74 61 25 32 43 72 65 76 6c |eraldelta%2Crevl|
0060: 25 32 43 67 65 6e 65 72 61 6c 64 65 6c 74 61 25 |%2Cgeneraldelta%| 0060: 6f 67 76 31 25 32 43 73 70 61 72 73 65 72 65 76 |ogv1%2Csparserev|
0070: 32 43 72 65 76 6c 6f 67 76 31 25 32 43 73 70 61 |2Crevlogv1%2Cspa| 0070: 6c 6f 67 00 00 80 00 73 08 42 64 61 74 61 2f 30 |log....s.Bdata/0|
0080: 72 73 65 72 65 76 6c 6f 67 25 32 43 73 74 6f 72 |rserevlog%2Cstor| 0080: 2e 69 00 03 00 01 00 00 00 00 00 00 00 02 00 00 |.i..............|
0090: 65 00 00 80 00 73 08 42 64 61 74 61 2f 30 2e 69 |e....s.Bdata/0.i| 0090: 00 01 00 00 00 00 00 00 00 01 ff ff ff ff ff ff |................|
00a0: 00 03 00 01 00 00 00 00 00 00 00 02 00 00 00 01 |................| 00a0: ff ff 80 29 63 a0 49 d3 23 87 bf ce fe 56 67 92 |...)c.I.#....Vg.|
00b0: 00 00 00 00 00 00 00 01 ff ff ff ff ff ff ff ff |................| 00b0: 67 2c 69 d1 ec 39 00 00 00 00 00 00 00 00 00 00 |g,i..9..........|
00c0: 80 29 63 a0 49 d3 23 87 bf ce fe 56 67 92 67 2c |.)c.I.#....Vg.g,| 00c0: 00 00 75 30 73 26 45 64 61 74 61 2f 30 30 63 68 |..u0s&Edata/00ch|
00d0: 69 d1 ec 39 00 00 00 00 00 00 00 00 00 00 00 00 |i..9............| 00d0: 61 6e 67 65 6c 6f 67 2d 61 62 33 34 39 31 38 30 |angelog-ab349180|
00e0: 75 30 73 26 45 64 61 74 61 2f 30 30 63 68 61 6e |u0s&Edata/00chan| 00e0: 61 30 34 30 35 30 31 30 2e 6e 64 2e 69 00 03 00 |a0405010.nd.i...|
00f0: 67 65 6c 6f 67 2d 61 62 33 34 39 31 38 30 61 30 |gelog-ab349180a0| 00f0: 01 00 00 00 00 00 00 00 05 00 00 00 04 00 00 00 |................|
#endif #endif
#if zstd no-rust #if zstd no-rust
$ f --size --hex --bytes 256 body $ f --size --hex --bytes 256 body
body: size=116340 (no-bigendian !) body: size=116310
body: size=116335 (bigendian !) body: size=116335 (bigendian !)
0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......| 0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
0010: 9a 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2.......| 0010: 7c 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 ||.STREAM2.......|
0020: 06 09 04 0c 5e 62 79 74 65 63 6f 75 6e 74 31 30 |....^bytecount10| 0020: 06 09 04 0c 40 62 79 74 65 63 6f 75 6e 74 31 30 |....@bytecount10|
0030: 31 32 37 36 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1276filecount109| (no-bigendian !) 0030: 31 32 37 36 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1276filecount109| (no-bigendian !)
0030: 31 32 37 31 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1271filecount109| (bigendian !) 0030: 31 32 37 31 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1271filecount109| (bigendian !)
0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 64 6f 74 |3requirementsdot| 0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 67 65 6e |3requirementsgen|
0050: 65 6e 63 6f 64 65 25 32 43 66 6e 63 61 63 68 65 |encode%2Cfncache| 0050: 65 72 61 6c 64 65 6c 74 61 25 32 43 72 65 76 6c |eraldelta%2Crevl|
0060: 25 32 43 67 65 6e 65 72 61 6c 64 65 6c 74 61 25 |%2Cgeneraldelta%| 0060: 6f 67 2d 63 6f 6d 70 72 65 73 73 69 6f 6e 2d 7a |og-compression-z|
0070: 32 43 72 65 76 6c 6f 67 2d 63 6f 6d 70 72 65 73 |2Crevlog-compres| 0070: 73 74 64 25 32 43 72 65 76 6c 6f 67 76 31 25 32 |std%2Crevlogv1%2|
0080: 73 69 6f 6e 2d 7a 73 74 64 25 32 43 72 65 76 6c |sion-zstd%2Crevl| 0080: 43 73 70 61 72 73 65 72 65 76 6c 6f 67 00 00 80 |Csparserevlog...|
0090: 6f 67 76 31 25 32 43 73 70 61 72 73 65 72 65 76 |ogv1%2Csparserev| 0090: 00 73 08 42 64 61 74 61 2f 30 2e 69 00 03 00 01 |.s.Bdata/0.i....|
00a0: 6c 6f 67 25 32 43 73 74 6f 72 65 00 00 80 00 73 |log%2Cstore....s| 00a0: 00 00 00 00 00 00 00 02 00 00 00 01 00 00 00 00 |................|
00b0: 08 42 64 61 74 61 2f 30 2e 69 00 03 00 01 00 00 |.Bdata/0.i......| 00b0: 00 00 00 01 ff ff ff ff ff ff ff ff 80 29 63 a0 |.............)c.|
00c0: 00 00 00 00 00 02 00 00 00 01 00 00 00 00 00 00 |................| 00c0: 49 d3 23 87 bf ce fe 56 67 92 67 2c 69 d1 ec 39 |I.#....Vg.g,i..9|
00d0: 00 01 ff ff ff ff ff ff ff ff 80 29 63 a0 49 d3 |...........)c.I.| 00d0: 00 00 00 00 00 00 00 00 00 00 00 00 75 30 73 26 |............u0s&|
00e0: 23 87 bf ce fe 56 67 92 67 2c 69 d1 ec 39 00 00 |#....Vg.g,i..9..| 00e0: 45 64 61 74 61 2f 30 30 63 68 61 6e 67 65 6c 6f |Edata/00changelo|
00f0: 00 00 00 00 00 00 00 00 00 00 75 30 73 26 45 64 |..........u0s&Ed| 00f0: 67 2d 61 62 33 34 39 31 38 30 61 30 34 30 35 30 |g-ab349180a04050|
#endif #endif
#if zstd rust no-dirstate-v2 #if zstd rust no-dirstate-v2
$ f --size --hex --bytes 256 body $ f --size --hex --bytes 256 body
body: size=116361 body: size=116331
0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......| 0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
0010: af 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2.......| 0010: 91 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2.......|
0020: 06 09 04 0c 73 62 79 74 65 63 6f 75 6e 74 31 30 |....sbytecount10| 0020: 06 09 04 0c 55 62 79 74 65 63 6f 75 6e 74 31 30 |....Ubytecount10|
0030: 31 32 37 36 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1276filecount109| 0030: 31 32 37 36 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1276filecount109|
0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 64 6f 74 |3requirementsdot| 0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 67 65 6e |3requirementsgen|
0050: 65 6e 63 6f 64 65 25 32 43 66 6e 63 61 63 68 65 |encode%2Cfncache| 0050: 65 72 61 6c 64 65 6c 74 61 25 32 43 70 65 72 73 |eraldelta%2Cpers|
0060: 25 32 43 67 65 6e 65 72 61 6c 64 65 6c 74 61 25 |%2Cgeneraldelta%| 0060: 69 73 74 65 6e 74 2d 6e 6f 64 65 6d 61 70 25 32 |istent-nodemap%2|
0070: 32 43 70 65 72 73 69 73 74 65 6e 74 2d 6e 6f 64 |2Cpersistent-nod| 0070: 43 72 65 76 6c 6f 67 2d 63 6f 6d 70 72 65 73 73 |Crevlog-compress|
0080: 65 6d 61 70 25 32 43 72 65 76 6c 6f 67 2d 63 6f |emap%2Crevlog-co| 0080: 69 6f 6e 2d 7a 73 74 64 25 32 43 72 65 76 6c 6f |ion-zstd%2Crevlo|
0090: 6d 70 72 65 73 73 69 6f 6e 2d 7a 73 74 64 25 32 |mpression-zstd%2| 0090: 67 76 31 25 32 43 73 70 61 72 73 65 72 65 76 6c |gv1%2Csparserevl|
00a0: 43 72 65 76 6c 6f 67 76 31 25 32 43 73 70 61 72 |Crevlogv1%2Cspar| 00a0: 6f 67 00 00 80 00 73 08 42 64 61 74 61 2f 30 2e |og....s.Bdata/0.|
00b0: 73 65 72 65 76 6c 6f 67 25 32 43 73 74 6f 72 65 |serevlog%2Cstore| 00b0: 69 00 03 00 01 00 00 00 00 00 00 00 02 00 00 00 |i...............|
00c0: 00 00 80 00 73 08 42 64 61 74 61 2f 30 2e 69 00 |....s.Bdata/0.i.| 00c0: 01 00 00 00 00 00 00 00 01 ff ff ff ff ff ff ff |................|
00d0: 03 00 01 00 00 00 00 00 00 00 02 00 00 00 01 00 |................| 00d0: ff 80 29 63 a0 49 d3 23 87 bf ce fe 56 67 92 67 |..)c.I.#....Vg.g|
00e0: 00 00 00 00 00 00 01 ff ff ff ff ff ff ff ff 80 |................| 00e0: 2c 69 d1 ec 39 00 00 00 00 00 00 00 00 00 00 00 |,i..9...........|
00f0: 29 63 a0 49 d3 23 87 bf ce fe 56 67 92 67 2c 69 |)c.I.#....Vg.g,i| 00f0: 00 75 30 73 26 45 64 61 74 61 2f 30 30 63 68 61 |.u0s&Edata/00cha|
#endif #endif
#if zstd dirstate-v2 #if zstd dirstate-v2
$ f --size --hex --bytes 256 body $ f --size --hex --bytes 256 body
body: size=109549 body: size=109549
0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......| 0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
0010: c0 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2.......| 0010: c0 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2.......|
0020: 05 09 04 0c 85 62 79 74 65 63 6f 75 6e 74 39 35 |.....bytecount95| 0020: 05 09 04 0c 85 62 79 74 65 63 6f 75 6e 74 39 35 |.....bytecount95|
0030: 38 39 37 66 69 6c 65 63 6f 75 6e 74 31 30 33 30 |897filecount1030| 0030: 38 39 37 66 69 6c 65 63 6f 75 6e 74 31 30 33 30 |897filecount1030|

tests/test-stream-bundle-v2.t

> B > B
> | > |
> A > A
> EOF > EOF
$ hg bundle -a --type="none-v2;stream=v2" bundle.hg $ hg bundle -a --type="none-v2;stream=v2" bundle.hg
$ hg debugbundle bundle.hg $ hg debugbundle bundle.hg
Stream params: {} Stream params: {}
stream2 -- {bytecount: 1693, filecount: 11, requirements: dotencode%2Cfncache%2Cgeneraldelta%2Crevlogv1%2Csparserevlog%2Cstore} (mandatory: True) (no-zstd !) stream2 -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlogv1%2Csparserevlog} (mandatory: True) (no-zstd !)
stream2 -- {bytecount: 1693, filecount: 11, requirements: dotencode%2Cfncache%2Cgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog%2Cstore} (mandatory: True) (zstd no-rust !) stream2 -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: True) (zstd no-rust !)
stream2 -- {bytecount: 1693, filecount: 11, requirements: dotencode%2Cfncache%2Cgeneraldelta%2Cpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog%2Cstore} (mandatory: True) (rust !) stream2 -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Cpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: True) (rust !)
$ hg debugbundle --spec bundle.hg $ hg debugbundle --spec bundle.hg
none-v2;stream=v2;requirements%3Ddotencode%2Cfncache%2Cgeneraldelta%2Crevlogv1%2Csparserevlog%2Cstore (no-zstd !) none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlogv1%2Csparserevlog (no-zstd !)
none-v2;stream=v2;requirements%3Ddotencode%2Cfncache%2Cgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog%2Cstore (zstd no-rust !) none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog (zstd no-rust !)
none-v2;stream=v2;requirements%3Ddotencode%2Cfncache%2Cgeneraldelta%2Cpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog%2Cstore (rust !) none-v2;stream=v2;requirements%3Dgeneraldelta%2Cpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog (rust !)
Test that we can apply the bundle as a stream clone bundle Test that we can apply the bundle as a stream clone bundle
$ cat > .hg/clonebundles.manifest << EOF $ cat > .hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/bundle.hg BUNDLESPEC=`hg debugbundle --spec bundle.hg` > http://localhost:$HGPORT1/bundle.hg BUNDLESPEC=`hg debugbundle --spec bundle.hg`
> EOF > EOF
$ hg serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log $ hg serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log