( )⚙ D12083 requirements: add an official `REVLOG_COMPRESSION_ZSTD` const

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D12083

requirements: add an official `REVLOG_COMPRESSION_ZSTD` const
ClosedPublic

Authored by marmoute on Jan 24 2022, 9:51 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG6fd9a17c32ab: requirements: add an official `REVLOG_COMPRESSION_ZSTD` const
Summary

Such constant was missing and its value was missing from the set of requirements
that needs to be preserved through stream clone. This did not had any
consequence yet as the "bundle 2 does not filter its requirements" is shadowing
the issue.

However we are now in a situation where we can fix this issue. So lets do it
next.

With the preparation work on test, changing the streamreqs value only impact two
tests, where checking the full value seems to remains relevant.

Important note: Since older version of Mercurial used the old supportedformat
class attribute to check for stream requirement they supported, older version
will consider this requirements to prevent them from using streaming clone. Even
as they support this requirements for years. Pack for stable will be send to fix
it, but they will have to be backported to older version if needed.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

marmoute created this revision.Jan 24 2022, 9:51 AM
Alphare accepted this revision.Jan 24 2022, 11:17 AM
Alphare added a subscriber: Alphare.

When you send the patch for stable we can think about telling the packaging list about this issue?

This revision is now accepted and ready to land.Jan 24 2022, 11:17 AM
marmoute added a comment.Jan 24 2022, 12:02 PM
In D12083#184933, @Alphare wrote:

When you send the patch for stable we can think about telling the packaging list about this issue?

This is 110% my plan.

baymax updated this revision to Diff 31848.Jan 24 2022, 1:11 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Revision Contents
Changeset List

Diff 31848

mercurial/localrepo.py

fn(ui, supported) fn(ui, supported)
# Add derived requirements from registered compression engines. # Add derived requirements from registered compression engines.
for name in util.compengines: for name in util.compengines:
engine = util.compengines[name] engine = util.compengines[name]
if engine.available() and engine.revlogheader(): if engine.available() and engine.revlogheader():
supported.add(b'exp-compression-%s' % name) supported.add(b'exp-compression-%s' % name)
if engine.name() == b'zstd': if engine.name() == b'zstd':
supported.add(b'revlog-compression-zstd') supported.add(requirementsmod.REVLOG_COMPRESSION_ZSTD)
return supported return supported
def ensurerequirementsrecognized(requirements, supported): def ensurerequirementsrecognized(requirements, supported):
"""Validate that a set of local requirements is recognized. """Validate that a set of local requirements is recognized.
Receives a set of requirements. Raises an ``error.RepoError`` if there Receives a set of requirements. Raises an ``error.RepoError`` if there

mercurial/requirements.py

# of stripping them # of stripping them
INTERNAL_PHASE_REQUIREMENT = b'internal-phase' INTERNAL_PHASE_REQUIREMENT = b'internal-phase'
# Stores manifest in Tree structure # Stores manifest in Tree structure
TREEMANIFEST_REQUIREMENT = b'treemanifest' TREEMANIFEST_REQUIREMENT = b'treemanifest'
REVLOGV1_REQUIREMENT = b'revlogv1' REVLOGV1_REQUIREMENT = b'revlogv1'
# allow using ZSTD as compression engine for revlog content
REVLOG_COMPRESSION_ZSTD = b'revlog-compression-zstd'
# Increment the sub-version when the revlog v2 format changes to lock out old # Increment the sub-version when the revlog v2 format changes to lock out old
# clients. # clients.
CHANGELOGV2_REQUIREMENT = b'exp-changelog-v2' CHANGELOGV2_REQUIREMENT = b'exp-changelog-v2'
# Increment the sub-version when the revlog v2 format changes to lock out old # Increment the sub-version when the revlog v2 format changes to lock out old
# clients. # clients.
REVLOGV2_REQUIREMENT = b'exp-revlogv2.2' REVLOGV2_REQUIREMENT = b'exp-revlogv2.2'
# note: the list is currently inherited from previous code and miss some relevant requirement while containing some irrelevant ones. # note: the list is currently inherited from previous code and miss some relevant requirement while containing some irrelevant ones.
STREAM_FIXED_REQUIREMENTS = { STREAM_FIXED_REQUIREMENTS = {
BOOKMARKS_IN_STORE_REQUIREMENT, BOOKMARKS_IN_STORE_REQUIREMENT,
CHANGELOGV2_REQUIREMENT, CHANGELOGV2_REQUIREMENT,
COPIESSDC_REQUIREMENT, COPIESSDC_REQUIREMENT,
DIRSTATE_V2_REQUIREMENT, DIRSTATE_V2_REQUIREMENT,
GENERALDELTA_REQUIREMENT, GENERALDELTA_REQUIREMENT,
NODEMAP_REQUIREMENT, NODEMAP_REQUIREMENT,
REVLOG_COMPRESSION_ZSTD,
REVLOGV1_REQUIREMENT, REVLOGV1_REQUIREMENT,
REVLOGV2_REQUIREMENT, REVLOGV2_REQUIREMENT,
SHARESAFE_REQUIREMENT, SHARESAFE_REQUIREMENT,
SPARSEREVLOG_REQUIREMENT, SPARSEREVLOG_REQUIREMENT,
TREEMANIFEST_REQUIREMENT, TREEMANIFEST_REQUIREMENT,
} }

tests/test-bundle.t

packed1 is produced properly packed1 is produced properly
#if reporevlogstore rust #if reporevlogstore rust
$ hg -R test debugcreatestreamclonebundle packed.hg $ hg -R test debugcreatestreamclonebundle packed.hg
writing 2665 bytes for 6 files writing 2665 bytes for 6 files
bundle requirements: generaldelta, persistent-nodemap, revlogv1, sparserevlog bundle requirements: generaldelta, persistent-nodemap, revlog-compression-zstd, revlogv1, sparserevlog
$ f -B 64 --size --sha1 --hexdump packed.hg $ f -B 64 --size --sha1 --hexdump packed.hg
packed.hg: size=2860, sha1=81d7a2e535892cda51e82c200f818de2cca828d3 packed.hg: size=2884, sha1=b0c868701f8a9fe44daf094b2f5bf661cf90c789
0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 06 00 00 |HGS1UN..........| 0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 06 00 00 |HGS1UN..........|
0010: 00 00 00 00 0a 69 00 36 67 65 6e 65 72 61 6c 64 |.....i.6generald| 0010: 00 00 00 00 0a 69 00 4e 67 65 6e 65 72 61 6c 64 |.....i.Ngenerald|
0020: 65 6c 74 61 2c 70 65 72 73 69 73 74 65 6e 74 2d |elta,persistent-| 0020: 65 6c 74 61 2c 70 65 72 73 69 73 74 65 6e 74 2d |elta,persistent-|
0030: 6e 6f 64 65 6d 61 70 2c 72 65 76 6c 6f 67 76 31 |nodemap,revlogv1| 0030: 6e 6f 64 65 6d 61 70 2c 72 65 76 6c 6f 67 2d 63 |nodemap,revlog-c|
$ hg debugbundle --spec packed.hg $ hg debugbundle --spec packed.hg
none-packed1;requirements%3Dgeneraldelta%2Cpersistent-nodemap%2Crevlogv1%2Csparserevlog none-packed1;requirements%3Dgeneraldelta%2Cpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog
#endif #endif
#if reporevlogstore no-rust zstd #if reporevlogstore no-rust zstd
$ hg -R test debugcreatestreamclonebundle packed.hg $ hg -R test debugcreatestreamclonebundle packed.hg
writing 2665 bytes for 6 files writing 2665 bytes for 6 files
bundle requirements: generaldelta, revlogv1, sparserevlog bundle requirements: generaldelta, revlog-compression-zstd, revlogv1, sparserevlog
$ f -B 64 --size --sha1 --hexdump packed.hg $ f -B 64 --size --sha1 --hexdump packed.hg
packed.hg: size=2841, sha1=8b645a65f49b0ae43042a9f3da56d4bfdf1c7f99 packed.hg: size=2865, sha1=353d10311f4befa195d9a1ca4b8e26518115c702
0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 06 00 00 |HGS1UN..........| 0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 06 00 00 |HGS1UN..........|
0010: 00 00 00 00 0a 69 00 23 67 65 6e 65 72 61 6c 64 |.....i.#generald| 0010: 00 00 00 00 0a 69 00 3b 67 65 6e 65 72 61 6c 64 |.....i.;generald|
0020: 65 6c 74 61 2c 72 65 76 6c 6f 67 76 31 2c 73 70 |elta,revlogv1,sp| 0020: 65 6c 74 61 2c 72 65 76 6c 6f 67 2d 63 6f 6d 70 |elta,revlog-comp|
0030: 61 72 73 65 72 65 76 6c 6f 67 00 64 61 74 61 2f |arserevlog.data/| 0030: 72 65 73 73 69 6f 6e 2d 7a 73 74 64 2c 72 65 76 |ression-zstd,rev|
$ hg debugbundle --spec packed.hg $ hg debugbundle --spec packed.hg
none-packed1;requirements%3Dgeneraldelta%2Crevlogv1%2Csparserevlog none-packed1;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog
#endif #endif
#if reporevlogstore no-rust no-zstd #if reporevlogstore no-rust no-zstd
$ hg -R test debugcreatestreamclonebundle packed.hg $ hg -R test debugcreatestreamclonebundle packed.hg
writing 2664 bytes for 6 files writing 2664 bytes for 6 files
bundle requirements: generaldelta, revlogv1, sparserevlog bundle requirements: generaldelta, revlogv1, sparserevlog
$ cd .. $ cd ..
#endif #endif
#if reporevlogstore rust #if reporevlogstore rust
$ hg -R testnongd debugcreatestreamclonebundle packednongd.hg $ hg -R testnongd debugcreatestreamclonebundle packednongd.hg
writing 301 bytes for 3 files writing 301 bytes for 3 files
bundle requirements: persistent-nodemap, revlogv1 bundle requirements: persistent-nodemap, revlog-compression-zstd, revlogv1
$ f -B 64 --size --sha1 --hexdump packednongd.hg $ f -B 64 --size --sha1 --hexdump packednongd.hg
packednongd.hg: size=402, sha1=d3cc1417f0e8142cf9340aaaa520b660ad3ec3ea packednongd.hg: size=426, sha1=79563ccd6ef779bcfe62a4da64f89a1b308e92e0
0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 03 00 00 |HGS1UN..........| 0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 03 00 00 |HGS1UN..........|
0010: 00 00 00 00 01 2d 00 1c 70 65 72 73 69 73 74 65 |.....-..persiste| 0010: 00 00 00 00 01 2d 00 34 70 65 72 73 69 73 74 65 |.....-.4persiste|
0020: 6e 74 2d 6e 6f 64 65 6d 61 70 2c 72 65 76 6c 6f |nt-nodemap,revlo| 0020: 6e 74 2d 6e 6f 64 65 6d 61 70 2c 72 65 76 6c 6f |nt-nodemap,revlo|
0030: 67 76 31 00 64 61 74 61 2f 66 6f 6f 2e 69 00 36 |gv1.data/foo.i.6| 0030: 67 2d 63 6f 6d 70 72 65 73 73 69 6f 6e 2d 7a 73 |g-compression-zs|
$ hg debugbundle --spec packednongd.hg $ hg debugbundle --spec packednongd.hg
none-packed1;requirements%3Dpersistent-nodemap%2Crevlogv1 none-packed1;requirements%3Dpersistent-nodemap%2Crevlog-compression-zstd%2Crevlogv1
#endif #endif
#if reporevlogstore no-rust zstd #if reporevlogstore no-rust zstd
$ hg -R testnongd debugcreatestreamclonebundle packednongd.hg $ hg -R testnongd debugcreatestreamclonebundle packednongd.hg
writing 301 bytes for 3 files writing 301 bytes for 3 files
bundle requirements: revlogv1 bundle requirements: revlog-compression-zstd, revlogv1
$ f -B 64 --size --sha1 --hexdump packednongd.hg $ f -B 64 --size --sha1 --hexdump packednongd.hg
packednongd.hg: size=383, sha1=1d9c230238edd5d38907100b729ba72b1831fe6f packednongd.hg: size=407, sha1=0b8714422b785ba8eb98c916b41ffd5fb994c9b5
0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 03 00 00 |HGS1UN..........| 0000: 48 47 53 31 55 4e 00 00 00 00 00 00 00 03 00 00 |HGS1UN..........|
0010: 00 00 00 00 01 2d 00 09 72 65 76 6c 6f 67 76 31 |.....-..revlogv1| 0010: 00 00 00 00 01 2d 00 21 72 65 76 6c 6f 67 2d 63 |.....-.!revlog-c|
0020: 00 64 61 74 61 2f 66 6f 6f 2e 69 00 36 34 0a 00 |.data/foo.i.64..| 0020: 6f 6d 70 72 65 73 73 69 6f 6e 2d 7a 73 74 64 2c |ompression-zstd,|
0030: 01 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 0030: 72 65 76 6c 6f 67 76 31 00 64 61 74 61 2f 66 6f |revlogv1.data/fo|
$ hg debugbundle --spec packednongd.hg $ hg debugbundle --spec packednongd.hg
none-packed1;requirements%3Drevlogv1 none-packed1;requirements%3Drevlog-compression-zstd%2Crevlogv1
#endif #endif
#if reporevlogstore no-rust no-zstd #if reporevlogstore no-rust no-zstd
$ hg -R testnongd debugcreatestreamclonebundle packednongd.hg $ hg -R testnongd debugcreatestreamclonebundle packednongd.hg
writing 301 bytes for 3 files writing 301 bytes for 3 files
#endif #endif
#if reporevlogstore rust #if reporevlogstore rust
$ hg -R testsecret debugcreatestreamclonebundle packedsecret.hg $ hg -R testsecret debugcreatestreamclonebundle packedsecret.hg
(warning: stream clone bundle will contain secret revisions) (warning: stream clone bundle will contain secret revisions)
writing 301 bytes for 3 files writing 301 bytes for 3 files
bundle requirements: generaldelta, persistent-nodemap, revlogv1, sparserevlog bundle requirements: generaldelta, persistent-nodemap, revlog-compression-zstd, revlogv1, sparserevlog
#endif #endif
#if reporevlogstore no-rust zstd #if reporevlogstore no-rust zstd
$ hg -R testsecret debugcreatestreamclonebundle packedsecret.hg $ hg -R testsecret debugcreatestreamclonebundle packedsecret.hg
(warning: stream clone bundle will contain secret revisions) (warning: stream clone bundle will contain secret revisions)
writing 301 bytes for 3 files writing 301 bytes for 3 files
bundle requirements: generaldelta, revlogv1, sparserevlog bundle requirements: generaldelta, revlog-compression-zstd, revlogv1, sparserevlog
#endif #endif
#if reporevlogstore no-rust no-zstd #if reporevlogstore no-rust no-zstd
$ hg -R testsecret debugcreatestreamclonebundle packedsecret.hg $ hg -R testsecret debugcreatestreamclonebundle packedsecret.hg
(warning: stream clone bundle will contain secret revisions) (warning: stream clone bundle will contain secret revisions)
writing 301 bytes for 3 files writing 301 bytes for 3 files

tests/test-clonebundles.t

searching for changes searching for changes
no changes found no changes found
2 local changesets published 2 local changesets published
Stream clone bundles are supported Stream clone bundles are supported
$ hg -R server debugcreatestreamclonebundle packed.hg $ hg -R server debugcreatestreamclonebundle packed.hg
writing 613 bytes for 4 files writing 613 bytes for 4 files
bundle requirements: generaldelta, revlogv1, sparserevlog (no-rust !) bundle requirements: generaldelta, revlogv1, sparserevlog (no-rust no-zstd !)
bundle requirements: generaldelta, persistent-nodemap, revlogv1, sparserevlog (rust !) bundle requirements: generaldelta, revlog-compression-zstd, revlogv1, sparserevlog (no-rust zstd !)
bundle requirements: generaldelta, persistent-nodemap, revlog-compression-zstd, revlogv1, sparserevlog (rust !)
No bundle spec should work No bundle spec should work
$ cat > server/.hg/clonebundles.manifest << EOF $ cat > server/.hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/packed.hg > http://localhost:$HGPORT1/packed.hg
> EOF > EOF
$ hg clone -U http://localhost:$HGPORT stream-clone-no-spec $ hg clone -U http://localhost:$HGPORT stream-clone-no-spec

tests/test-debugcommands.t

#if rust #if rust
$ hg --debug debugpeer ssh://user@dummy/debugrevlog $ hg --debug debugpeer ssh://user@dummy/debugrevlog
running .* ".*[/\\]dummyssh" ['"]user@dummy['"] ['"]hg -R debugrevlog serve --stdio['"] (re) running .* ".*[/\\]dummyssh" ['"]user@dummy['"] ['"]hg -R debugrevlog serve --stdio['"] (re)
devel-peer-request: hello+between devel-peer-request: hello+between
devel-peer-request: pairs: 81 bytes devel-peer-request: pairs: 81 bytes
sending hello command sending hello command
sending between command sending between command
remote: 463 remote: 487
remote: capabilities: batch branchmap $USUAL_BUNDLE2_CAPS$ changegroupsubset getbundle known lookup protocaps pushkey streamreqs=generaldelta,persistent-nodemap,revlogv1,sparserevlog unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash remote: capabilities: batch branchmap $USUAL_BUNDLE2_CAPS$ changegroupsubset getbundle known lookup protocaps pushkey streamreqs=generaldelta,persistent-nodemap,revlog-compression-zstd,revlogv1,sparserevlog unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
remote: 1 remote: 1
devel-peer-request: protocaps devel-peer-request: protocaps
devel-peer-request: caps: * bytes (glob) devel-peer-request: caps: * bytes (glob)
sending protocaps command sending protocaps command
url: ssh://user@dummy/debugrevlog url: ssh://user@dummy/debugrevlog
local: no local: no
pushable: yes pushable: yes
#endif #endif
#if no-rust zstd #if no-rust zstd
$ hg --debug debugpeer ssh://user@dummy/debugrevlog $ hg --debug debugpeer ssh://user@dummy/debugrevlog
running .* ".*[/\\]dummyssh" ['"]user@dummy['"] ['"]hg -R debugrevlog serve --stdio['"] (re) running .* ".*[/\\]dummyssh" ['"]user@dummy['"] ['"]hg -R debugrevlog serve --stdio['"] (re)
devel-peer-request: hello+between devel-peer-request: hello+between
devel-peer-request: pairs: 81 bytes devel-peer-request: pairs: 81 bytes
sending hello command sending hello command
sending between command sending between command
remote: 444 remote: 468
remote: capabilities: batch branchmap $USUAL_BUNDLE2_CAPS$ changegroupsubset getbundle known lookup protocaps pushkey streamreqs=generaldelta,revlogv1,sparserevlog unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash remote: capabilities: batch branchmap $USUAL_BUNDLE2_CAPS$ changegroupsubset getbundle known lookup protocaps pushkey streamreqs=generaldelta,revlog-compression-zstd,revlogv1,sparserevlog unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
remote: 1 remote: 1
devel-peer-request: protocaps devel-peer-request: protocaps
devel-peer-request: caps: * bytes (glob) devel-peer-request: caps: * bytes (glob)
sending protocaps command sending protocaps command
url: ssh://user@dummy/debugrevlog url: ssh://user@dummy/debugrevlog
local: no local: no
pushable: yes pushable: yes