( )⚙ D9992 upgrade: speed up when we have only nodemap to downgrade

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9992

upgrade: speed up when we have only nodemap to downgrade
ClosedPublic

Authored by pulkit on Feb 15 2021, 5:28 AM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG67b5fafd3a46: upgrade: speed up when we have only nodemap to downgrade
Summary

Similar to what we do on upgrade, if we have only persistent-nodemap to
downgrade we will just delete the nodemap files and update repository
requirements instead of processing all the revlogs.

After downgrade, we are left with unrequired docket and transaction files which
seems fine but can work on deleting them if someone feels we should.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

pulkit created this revision.Feb 15 2021, 5:28 AM
Alphare added a subscriber: Alphare.Feb 23 2021, 4:11 AM

I think I'd rather we also delete the docket file(s) and transaction files if we can, but this would not be a blocker for this patch in particular to land.

mercurial/revlogutils/nodemap.py
134

nit: accessing rather than calling

mercurial/upgrade_utils/engine.py
481

That seems overly complicated since you already check len(upgrade_op.removes_actions) == 1 right before

pulkit added a commit: rHG67b5fafd3a46: upgrade: speed up when we have only nodemap to downgrade.Feb 24 2021, 10:54 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG67b5fafd3a46: upgrade: speed up when we have only nodemap to downgrade (authored by pulkit). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 25628

mercurial/revlogutils/nodemap.py

return # we do not use persistent_nodemap on this revlog return # we do not use persistent_nodemap on this revlog
notr = _NoTransaction() notr = _NoTransaction()
persist_nodemap(notr, revlog) persist_nodemap(notr, revlog)
for k in sorted(notr._postclose): for k in sorted(notr._postclose):
notr._postclose[k](None) notr._postclose[k](None)
def delete_nodemap(tr, repo, revlog):
""" Delete nodemap data on disk for a given revlog"""
if revlog.nodemap_file is None:
msg = "calling persist nodemap on a revlog without the feature enabled"
AlphareUnsubmitted
Not Done

nit: accessing rather than calling

Alphare: nit: `accessing` rather than `calling`
raise error.ProgrammingError(msg)
repo.svfs.unlink(revlog.nodemap_file)
def persist_nodemap(tr, revlog, pending=False, force=False): def persist_nodemap(tr, revlog, pending=False, force=False):
"""Write nodemap data on disk for a given revlog""" """Write nodemap data on disk for a given revlog"""
if getattr(revlog, 'filteredrevs', ()): if getattr(revlog, 'filteredrevs', ()):
raise error.ProgrammingError( raise error.ProgrammingError(
"cannot persist nodemap of a filtered changelog" "cannot persist nodemap of a filtered changelog"
) )
if revlog.nodemap_file is None: if revlog.nodemap_file is None:
if force: if force:

mercurial/upgrade_utils/engine.py

# we want to directly operate on the underlying revlog to force # we want to directly operate on the underlying revlog to force
# create a nodemap file. This is fine since this is upgrade code # create a nodemap file. This is fine since this is upgrade code
# and it heavily relies on repository being revlog based # and it heavily relies on repository being revlog based
# hence accessing private attributes can be justified # hence accessing private attributes can be justified
nodemap.persist_nodemap( nodemap.persist_nodemap(
tr, unfi.manifestlog._rootstore._revlog, force=True tr, unfi.manifestlog._rootstore._revlog, force=True
) )
scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements) scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements)
elif (
len(upgrade_op.removed_actions) == 1
and [
AlphareUnsubmitted
Not Done

That seems overly complicated since you already check len(upgrade_op.removes_actions) == 1 right before

Alphare: That seems overly complicated since you already check `len(upgrade_op.removes_actions) == 1`…
x
for x in upgrade_op.removed_actions
if x.name == b'persistent-nodemap'
]
and not upgrade_op.upgrade_actions
):
ui.status(
_(b'downgrading repository to not use persistent nodemap feature\n')
)
with srcrepo.transaction(b'upgrade') as tr:
unfi = srcrepo.unfiltered()
cl = unfi.changelog
nodemap.delete_nodemap(tr, srcrepo, cl)
# check comment 20 lines above for accessing private attributes
nodemap.delete_nodemap(
tr, srcrepo, unfi.manifestlog._rootstore._revlog
)
scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements)
else: else:
with dstrepo.transaction(b'upgrade') as tr: with dstrepo.transaction(b'upgrade') as tr:
_clonerevlogs( _clonerevlogs(
ui, ui,
srcrepo, srcrepo,
dstrepo, dstrepo,
tr, tr,
upgrade_op, upgrade_op,

tests/test-persistent-nodemap.t

share-safe: no no no share-safe: no no no
sparserevlog: yes yes yes sparserevlog: yes yes yes
sidedata: no no no sidedata: no no no
persistent-nodemap: yes no no persistent-nodemap: yes no no
copies-sdc: no no no copies-sdc: no no no
plain-cl-delta: yes yes yes plain-cl-delta: yes yes yes
compression: zlib zlib zlib compression: zlib zlib zlib
compression-level: default default default compression-level: default default default
$ hg debugupgraderepo --run --no-backup --quiet $ hg debugupgraderepo --run --no-backup
upgrade will perform the following actions: upgrade will perform the following actions:
requirements requirements
preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store
removed: persistent-nodemap removed: persistent-nodemap
processed revlogs: processed revlogs:
- all-filelogs - all-filelogs
- changelog - changelog
- manifest - manifest
beginning upgrade...
repository locked and read-only
creating temporary repository to stage upgraded data: $TESTTMP/test-repo/.hg/upgrade.* (glob)
(it is safe to interrupt this process any time before data migration completes)
downgrading repository to not use persistent nodemap feature
removing temporary repository $TESTTMP/test-repo/.hg/upgrade.* (glob)
$ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)' $ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
[1] 00changelog-*.nd (glob)
00manifest-*.nd (glob)
undo.backup.00changelog.n
undo.backup.00manifest.n
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
upgrading upgrading
$ cat << EOF >> .hg/hgrc $ cat << EOF >> .hg/hgrc
> [format] > [format]
> use-persistent-nodemap=yes > use-persistent-nodemap=yes
(it is safe to interrupt this process any time before data migration completes) (it is safe to interrupt this process any time before data migration completes)
upgrading repository to use persistent nodemap feature upgrading repository to use persistent nodemap feature
removing temporary repository $TESTTMP/test-repo/.hg/upgrade.* (glob) removing temporary repository $TESTTMP/test-repo/.hg/upgrade.* (glob)
$ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)' $ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
00changelog-*.nd (glob) 00changelog-*.nd (glob)
00changelog.n 00changelog.n
00manifest-*.nd (glob) 00manifest-*.nd (glob)
00manifest.n 00manifest.n
undo.backup.00changelog.n
undo.backup.00manifest.n
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%