This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9936

upgrade: implement partial upgrade for upgrading persistent-nodemap
ClosedPublic

Authored by pulkit on Jan 31 2021, 2:35 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG98e39f04d60e: upgrade: implement partial upgrade for upgrading persistent-nodemap
Summary

Upgrading repositories to use persistent nodemap should be fast and easy as it
requires only two things:

  1. Updating the requirements
  2. Writing a persistent-nodemap on disk

For both of the steps above, we don't need to edit existing revlogs.

This patch makes upgrade only do the above mentioned two steps if we are
only upgarding to use persistent-nodemap feature.

Since nodemap.persist_nodemap() assumes that there exists a nodemap file for
the given revlog if we are trying to call it, this patch adds force argument
to create a file if does not exist which is true in our upgrade case.

The test changes demonstrate that we no longer write nodemap files for manifest
after upgrade which I think is desirable.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

pulkit created this revision.Jan 31 2021, 2:35 PM
pulkit added a commit: rHG98e39f04d60e: upgrade: implement partial upgrade for upgrading persistent-nodemap.Feb 2 2021, 1:33 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG98e39f04d60e: upgrade: implement partial upgrade for upgrading persistent-nodemap (authored by pulkit). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 25474

mercurial/revlogutils/nodemap.py

return # we do not use persistent_nodemap on this revlog return # we do not use persistent_nodemap on this revlog
notr = _NoTransaction() notr = _NoTransaction()
persist_nodemap(notr, revlog) persist_nodemap(notr, revlog)
for k in sorted(notr._postclose): for k in sorted(notr._postclose):
notr._postclose[k](None) notr._postclose[k](None)
def persist_nodemap(tr, revlog, pending=False): def persist_nodemap(tr, revlog, pending=False, force=False):
"""Write nodemap data on disk for a given revlog""" """Write nodemap data on disk for a given revlog"""
if getattr(revlog, 'filteredrevs', ()): if getattr(revlog, 'filteredrevs', ()):
raise error.ProgrammingError( raise error.ProgrammingError(
"cannot persist nodemap of a filtered changelog" "cannot persist nodemap of a filtered changelog"
) )
if revlog.nodemap_file is None: if revlog.nodemap_file is None:
if force:
revlog.nodemap_file = get_nodemap_file(
revlog.opener, revlog.indexfile
)
else:
msg = "calling persist nodemap on a revlog without the feature enabled" msg = "calling persist nodemap on a revlog without the feature enabled"
raise error.ProgrammingError(msg) raise error.ProgrammingError(msg)
can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental") can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
ondisk_docket = revlog._nodemap_docket ondisk_docket = revlog._nodemap_docket
feed_data = util.safehasattr(revlog.index, "update_nodemap_data") feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap") use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
data = None data = None
# first attemp an incremental update of the data # first attemp an incremental update of the data

mercurial/upgrade_utils/engine.py

metadata, metadata,
pycompat, pycompat,
requirements, requirements,
revlog, revlog,
scmutil, scmutil,
util, util,
vfs as vfsmod, vfs as vfsmod,
) )
from ..revlogutils import nodemap
def _revlogfrompath(repo, path): def _revlogfrompath(repo, path):
"""Obtain a revlog from a repo path. """Obtain a revlog from a repo path.
An instance of the appropriate class is returned. An instance of the appropriate class is returned.
""" """
if path == b'00changelog.i': if path == b'00changelog.i':
b'(it is safe to interrupt this process any time before ' b'(it is safe to interrupt this process any time before '
b'data migration completes)\n' b'data migration completes)\n'
) )
) )
if upgrade_op.requirements_only: if upgrade_op.requirements_only:
ui.status(_(b'upgrading repository requirements\n')) ui.status(_(b'upgrading repository requirements\n'))
scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements) scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements)
# if there is only one action and that is persistent nodemap upgrade
# directly write the nodemap file and update requirements instead of going
# through the whole cloning process
elif (
len(upgrade_op.upgrade_actions) == 1
and b'persistent-nodemap' in upgrade_op._upgrade_actions_names
and not upgrade_op.removed_actions
):
ui.status(
_(b'upgrading repository to use persistent nodemap feature\n')
)
with srcrepo.transaction(b'upgrade') as tr:
unfi = srcrepo.unfiltered()
cl = unfi.changelog
nodemap.persist_nodemap(tr, cl, force=True)
scmutil.writereporequirements(srcrepo, upgrade_op.new_requirements)
else: else:
with dstrepo.transaction(b'upgrade') as tr: with dstrepo.transaction(b'upgrade') as tr:
_clonerevlogs( _clonerevlogs(
ui, ui,
srcrepo, srcrepo,
dstrepo, dstrepo,
tr, tr,
upgrade_op, upgrade_op,

tests/test-persistent-nodemap.t

- all-filelogs - all-filelogs
- changelog - changelog
- manifest - manifest
beginning upgrade... beginning upgrade...
repository locked and read-only repository locked and read-only
creating temporary repository to stage upgraded data: $TESTTMP/test-repo/.hg/upgrade.* (glob) creating temporary repository to stage upgraded data: $TESTTMP/test-repo/.hg/upgrade.* (glob)
(it is safe to interrupt this process any time before data migration completes) (it is safe to interrupt this process any time before data migration completes)
migrating 15018 total revisions (5006 in filelogs, 5006 in manifests, 5006 in changelog) upgrading repository to use persistent nodemap feature
migrating 1.74 MB in store; 569 MB tracked data
migrating 5004 filelogs containing 5006 revisions (346 KB in store; 28.2 KB tracked data)
finished migrating 5006 filelog revisions across 5004 filelogs; change in size: 0 bytes
migrating 1 manifests containing 5006 revisions (765 KB in store; 569 MB tracked data)
finished migrating 5006 manifest revisions across 1 manifests; change in size: 0 bytes
migrating changelog containing 5006 revisions (673 KB in store; 363 KB tracked data)
finished migrating 5006 changelog revisions; change in size: 0 bytes
finished migrating 15018 total revisions; total change in store size: 0 bytes
copying phaseroots
data fully upgraded in a temporary repository
marking source repository as being upgraded; clients will be unable to read from repository
starting in-place swap of repository data
replacing store...
store replacement complete; repository was inconsistent for *s (glob)
finalizing requirements file and making repository readable again
removing temporary repository $TESTTMP/test-repo/.hg/upgrade.* (glob) removing temporary repository $TESTTMP/test-repo/.hg/upgrade.* (glob)
$ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)' $ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
00changelog-*.nd (glob) 00changelog-*.nd (glob)
00changelog.n 00changelog.n
00manifest-*.nd (glob)
00manifest.n
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: * (glob) uid: * (glob)
tip-rev: 5005 tip-rev: 5005
tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
data-length: 121088 data-length: 121088
data-unused: 0 data-unused: 0
data-unused: 0.000% data-unused: 0.000%