This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8431

upgrade: support upgrade and downgrade from persistent nodemap
ClosedPublic

Authored by marmoute on Apr 15 2020, 4:16 PM.

Details

Reviewers
mharbison72
Alphare
durin42
Group Reviewers
hg-reviewers
Commits
rHG526d69eeea31: upgrade: support upgrade and downgrade from persistent nodemap
Summary

The requirements is now recognised and dealt with and the associated files
properly handled.

The persistent nodemap should be ready for usage in the field now.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

marmoute created this revision.Apr 15 2020, 4:16 PM
mharbison72 accepted this revision.Apr 16 2020, 9:51 AM
mharbison72 added a subscriber: mharbison72.

s/requirements is not/requirement is now/ in the summary, but otherwise LGTM

Alphare accepted this revision.May 7 2020, 9:26 AM
Alphare added a subscriber: Alphare.

Same remark as Matt.

marmoute edited the summary of this revision. (Show Details)May 7 2020, 10:13 AM
marmoute added a comment.May 7 2020, 10:13 AM
In D8431#125976, @mharbison72 wrote:

s/requirements is not/requirement is now/ in the summary, but otherwise LGTM

Good catch, fixing it now.

durin42 requested changes to this revision.May 7 2020, 11:17 AM
durin42 added a subscriber: durin42.

This requirement needs documentation in mercurial/helptext/internals/requirements.txt. I've taken the rest of the stack, but am blocking this patch on that change.

This revision now requires changes to proceed.May 7 2020, 11:17 AM
marmoute updated this revision to Diff 21315.May 7 2020, 5:01 PM
durin42 added inline comments.May 11 2020, 5:08 PM
mercurial/helptext/internals/requirements.txt
150

The parenthetical on this line isn't closed. Should we just drop the opening paren?

155

nit: installations (should be plural, also has typo)

156

"Other installations ..."

marmoute updated this revision to Diff 21349.May 12 2020, 4:36 AM
marmoute added a commit: rHG526d69eeea31: upgrade: support upgrade and downgrade from persistent nodemap.May 14 2020, 4:31 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG526d69eeea31: upgrade: support upgrade and downgrade from persistent nodemap (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 21349

mercurial/helptext/internals/requirements.txt

where they used to be stored. The active bookmark is still stored where they used to be stored. The active bookmark is still stored
directly in ``.hg/``. This makes them always shared by ``hg share``, directly in ``.hg/``. This makes them always shared by ``hg share``,
whether or not ``-B`` was passed. whether or not ``-B`` was passed.
Support for this requirement was added in Mercurial 5.1 (released Support for this requirement was added in Mercurial 5.1 (released
August 2019). The requirement will only be present on repositories August 2019). The requirement will only be present on repositories
that have opted in to this format (by having that have opted in to this format (by having
``format.bookmarks-in-store=true`` set when they were created). ``format.bookmarks-in-store=true`` set when they were created).
persistent-nodemap
==================
The `nodemap` index (mapping nodeid to local revision number) is persisted on
disk. This provides speed benefit (if the associated native code is used). The
durin42Unsubmitted
Not Done

The parenthetical on this line isn't closed. Should we just drop the opening paren?

durin42: The parenthetical on this line isn't closed. Should we just drop the opening paren?
persistent nodemap is only used for two revlogs: the changelog and the
manifestlog.
Support for this requirement was added in Mercurial 5.5 (released August 2020).
Note that as of 5.5, only installations compiled with the Rust extension will
durin42Unsubmitted
Not Done

nit: installations (should be plural, also has typo)

durin42: nit: installations (should be plural, also has typo)
benefit from a speedup. The other installations will do the necessary work to
durin42Unsubmitted
Not Done

"Other installations ..."

durin42: "Other installations ..."
keep the index up to date, but will suffer a slowdown.

mercurial/upgrade.py

If an upgrade were to create a repository that dropped a requirement, If an upgrade were to create a repository that dropped a requirement,
the dropped requirement must appear in the returned set for the upgrade the dropped requirement must appear in the returned set for the upgrade
to be allowed. to be allowed.
""" """
supported = { supported = {
localrepo.SPARSEREVLOG_REQUIREMENT, localrepo.SPARSEREVLOG_REQUIREMENT,
localrepo.SIDEDATA_REQUIREMENT, localrepo.SIDEDATA_REQUIREMENT,
localrepo.COPIESSDC_REQUIREMENT, localrepo.COPIESSDC_REQUIREMENT,
localrepo.NODEMAP_REQUIREMENT,
} }
for name in compression.compengines: for name in compression.compengines:
engine = compression.compengines[name] engine = compression.compengines[name]
if engine.available() and engine.revlogheader(): if engine.available() and engine.revlogheader():
supported.add(b'exp-compression-%s' % name) supported.add(b'exp-compression-%s' % name)
if engine.name() == b'zstd': if engine.name() == b'zstd':
supported.add(b'revlog-compression-zstd') supported.add(b'revlog-compression-zstd')
return supported return supported
b'dotencode', b'dotencode',
b'fncache', b'fncache',
b'generaldelta', b'generaldelta',
b'revlogv1', b'revlogv1',
b'store', b'store',
localrepo.SPARSEREVLOG_REQUIREMENT, localrepo.SPARSEREVLOG_REQUIREMENT,
localrepo.SIDEDATA_REQUIREMENT, localrepo.SIDEDATA_REQUIREMENT,
localrepo.COPIESSDC_REQUIREMENT, localrepo.COPIESSDC_REQUIREMENT,
localrepo.NODEMAP_REQUIREMENT,
} }
for name in compression.compengines: for name in compression.compengines:
engine = compression.compengines[name] engine = compression.compengines[name]
if engine.available() and engine.revlogheader(): if engine.available() and engine.revlogheader():
supported.add(b'exp-compression-%s' % name) supported.add(b'exp-compression-%s' % name)
if engine.name() == b'zstd': if engine.name() == b'zstd':
supported.add(b'revlog-compression-zstd') supported.add(b'revlog-compression-zstd')
return supported return supported
""" """
supported = { supported = {
b'dotencode', b'dotencode',
b'fncache', b'fncache',
b'generaldelta', b'generaldelta',
localrepo.SPARSEREVLOG_REQUIREMENT, localrepo.SPARSEREVLOG_REQUIREMENT,
localrepo.SIDEDATA_REQUIREMENT, localrepo.SIDEDATA_REQUIREMENT,
localrepo.COPIESSDC_REQUIREMENT, localrepo.COPIESSDC_REQUIREMENT,
localrepo.NODEMAP_REQUIREMENT,
} }
for name in compression.compengines: for name in compression.compengines:
engine = compression.compengines[name] engine = compression.compengines[name]
if engine.available() and engine.revlogheader(): if engine.available() and engine.revlogheader():
supported.add(b'exp-compression-%s' % name) supported.add(b'exp-compression-%s' % name)
if engine.name() == b'zstd': if engine.name() == b'zstd':
supported.add(b'revlog-compression-zstd') supported.add(b'revlog-compression-zstd')
return supported return supported
requirements: set of requirements for ``dstrepo`` requirements: set of requirements for ``dstrepo``
path: store file being examined path: store file being examined
mode: the ``ST_MODE`` file type of ``path`` mode: the ``ST_MODE`` file type of ``path``
st: ``stat`` data structure for ``path`` st: ``stat`` data structure for ``path``
Function should return ``True`` if the file is to be copied. Function should return ``True`` if the file is to be copied.
""" """
# Skip revlogs. # Skip revlogs.
if path.endswith((b'.i', b'.d')): if path.endswith((b'.i', b'.d', b'.n', b'.nd')):
return False return False
# Skip transaction related files. # Skip transaction related files.
if path.startswith(b'undo'): if path.startswith(b'undo'):
return False return False
# Only copy regular files. # Only copy regular files.
if mode != stat.S_IFREG: if mode != stat.S_IFREG:
return False return False
# Skip other skipped files. # Skip other skipped files.

tests/test-persistent-nodemap.t

data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 0.369% (pure !) data-unused: 0.369% (pure !)
data-unused: 0.369% (rust !) data-unused: 0.369% (rust !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ f --size --sha256 .hg/store/00changelog-*.nd $ f --size --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !) .hg/store/00changelog-????????????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !)
.hg/store/00changelog-????????????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !) .hg/store/00changelog-????????????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !)
.hg/store/00changelog-????????????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !) .hg/store/00changelog-????????????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !)
Test upgrade / downgrade
========================
downgrading
$ cat << EOF >> .hg/hgrc
> [format]
> use-persistent-nodemap=no
> EOF
$ hg debugformat -v
format-variant repo config default
fncache: yes yes yes
dotencode: yes yes yes
generaldelta: yes yes yes
sparserevlog: yes yes yes
sidedata: no no no
persistent-nodemap: yes no no
copies-sdc: no no no
plain-cl-delta: yes yes yes
compression: zlib zlib zlib
compression-level: default default default
$ hg debugupgraderepo --run --no-backup --quiet
upgrade will perform the following actions:
requirements
preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store
removed: persistent-nodemap
$ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
[1]
$ hg debugnodemap --metadata
upgrading
$ cat << EOF >> .hg/hgrc
> [format]
> use-persistent-nodemap=yes
> EOF
$ hg debugformat -v
format-variant repo config default
fncache: yes yes yes
dotencode: yes yes yes
generaldelta: yes yes yes
sparserevlog: yes yes yes
sidedata: no no no
persistent-nodemap: no yes no
copies-sdc: no no no
plain-cl-delta: yes yes yes
compression: zlib zlib zlib
compression-level: default default default
$ hg debugupgraderepo --run --no-backup --quiet
upgrade will perform the following actions:
requirements
preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store
added: persistent-nodemap
$ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
00changelog-*.nd (glob)
00changelog.n
00manifest-*.nd (glob)
00manifest.n
$ hg debugnodemap --metadata
uid: * (glob)
tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088
data-unused: 0
data-unused: 0.000%
Running unrelated upgrade
$ hg debugupgraderepo --run --no-backup --quiet --optimize re-delta-all
upgrade will perform the following actions:
requirements
preserved: dotencode, fncache, generaldelta, persistent-nodemap, revlogv1, sparserevlog, store
optimisations: re-delta-all
$ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
00changelog-*.nd (glob)
00changelog.n
00manifest-*.nd (glob)
00manifest.n
$ hg debugnodemap --metadata
uid: * (glob)
tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088
data-unused: 0
data-unused: 0.000%