This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8187

nodemap: make sure hooks have access to an up-to-date version
ClosedPublic

Authored by marmoute on Feb 28 2020, 1:55 PM.

Details

Reviewers
indygreg
durin42
Group Reviewers
hg-reviewers
Commits
rHG64e2f603de9d: nodemap: make sure hooks have access to an up-to-date version
Summary

We make sure hooks can read persistent nodemap data and that they access
something up-to-date with the pending transaction.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Feb 28 2020, 1:55 PM
marmoute updated this revision to Diff 20582.Mar 6 2020, 6:37 PM

eventless rebase past D8255

durin42 requested changes to this revision.Mar 10 2020, 4:42 PM
durin42 added a subscriber: durin42.

No longer applies on top of @.

This revision now requires changes to proceed.Mar 10 2020, 4:42 PM
marmoute updated this revision to Diff 20684.Mar 10 2020, 9:35 PM

rebase above latest default up to landed-D8182

marmoute added a comment.Mar 10 2020, 9:36 PM
In D8187#123258, @durin42 wrote:

No longer applies on top of @.

I don't understand this feedback. There are about 15 other changesets before it in this stack, so this is not meant to be directly applied on the head of default.

durin42 added a comment.Mar 11 2020, 11:20 AM
In D8187#123306, @marmoute wrote:
In D8187#123258, @durin42 wrote:

No longer applies on top of @.

I don't understand this feedback. There are about 15 other changesets before it in this stack, so this is not meant to be directly applied on the head of default.

I mentioned this in IRC, but I'll put it here for completeness: the parent pointers in this series are borked, and yadda was showing them /extremely/ out of order, to the point that this was about the 5th change in the series. I think I've managed to piece together a correct order, but if it doesn't apply I'll drop the whole stack until the parent pointers are contiguous again.

marmoute added a commit: rHG64e2f603de9d: nodemap: make sure hooks have access to an up-to-date version.Mar 11 2020, 11:43 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG64e2f603de9d: nodemap: make sure hooks have access to an up-to-date version (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 20704

mercurial/configitems.py

b'devel', b'warn-empty-changegroup', default=False, b'devel', b'warn-empty-changegroup', default=False,
) )
coreconfigitem( coreconfigitem(
b'devel', b'legacy.exchange', default=list, b'devel', b'legacy.exchange', default=list,
) )
# TODO before getting `persistent-nodemap` out of experimental # TODO before getting `persistent-nodemap` out of experimental
# #
# * code/tests around aborted transaction # * code/tests around aborted transaction
# * code/tests around pending data for hooks
# * regenerate a new nodemap when the unused/total ration is to high # * regenerate a new nodemap when the unused/total ration is to high
# * decide for a "status" of the persistent nodemap and associated location # * decide for a "status" of the persistent nodemap and associated location
# - part of the store next the revlog itself (new requirements) # - part of the store next the revlog itself (new requirements)
# - part of the cache directory # - part of the cache directory
# - part of an `index` directory # - part of an `index` directory
# (https://www.mercurial-scm.org/wiki/ComputedIndexPlan) # (https://www.mercurial-scm.org/wiki/ComputedIndexPlan)
# * do we want to use this for more than just changelog? if so we need: # * do we want to use this for more than just changelog? if so we need:
# - simpler "pending" logic for them # - simpler "pending" logic for them

mercurial/revlog.py

and can be used to implement COW semantics or the like. and can be used to implement COW semantics or the like.
""" """
self.upperboundcomp = upperboundcomp self.upperboundcomp = upperboundcomp
self.indexfile = indexfile self.indexfile = indexfile
self.datafile = datafile or (indexfile[:-2] + b".d") self.datafile = datafile or (indexfile[:-2] + b".d")
self.nodemap_file = None self.nodemap_file = None
if persistentnodemap: if persistentnodemap:
if indexfile.endswith(b'.a'):
pending_path = indexfile[:-4] + b".n.a"
if opener.exists(pending_path):
self.nodemap_file = pending_path
else:
self.nodemap_file = indexfile[:-4] + b".n"
else:
self.nodemap_file = indexfile[:-2] + b".n" self.nodemap_file = indexfile[:-2] + b".n"
self.opener = opener self.opener = opener
# When True, indexfile is opened with checkambig=True at writing, to # When True, indexfile is opened with checkambig=True at writing, to
# avoid file stat ambiguity. # avoid file stat ambiguity.
self._checkambig = checkambig self._checkambig = checkambig
self._mmaplargeindex = mmaplargeindex self._mmaplargeindex = mmaplargeindex
self._censorable = censorable self._censorable = censorable
# 3-tuple of (node, rev, text) for a raw revision. # 3-tuple of (node, rev, text) for a raw revision.

mercurial/revlogutils/nodemap.py

""" """
if revlog._inline: if revlog._inline:
return # inlined revlog are too small for this to be relevant return # inlined revlog are too small for this to be relevant
if revlog.nodemap_file is None: if revlog.nodemap_file is None:
return # we do not use persistent_nodemap on this revlog return # we do not use persistent_nodemap on this revlog
callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file
if tr.hasfinalize(callback_id): if tr.hasfinalize(callback_id):
return # no need to register again return # no need to register again
tr.addpending(
callback_id, lambda tr: _persist_nodemap(tr, revlog, pending=True)
)
tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog)) tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
class _NoTransaction(object): class _NoTransaction(object):
"""transaction like object to update the nodemap outside a transaction """transaction like object to update the nodemap outside a transaction
""" """
def __init__(self): def __init__(self):
setup (eg, `debugupdatecache`). setup (eg, `debugupdatecache`).
""" """
notr = _NoTransaction() notr = _NoTransaction()
_persist_nodemap(notr, revlog) _persist_nodemap(notr, revlog)
for k in sorted(notr._postclose): for k in sorted(notr._postclose):
notr._postclose[k](None) notr._postclose[k](None)
def _persist_nodemap(tr, revlog): def _persist_nodemap(tr, revlog, pending=False):
"""Write nodemap data on disk for a given revlog """Write nodemap data on disk for a given revlog
""" """
if getattr(revlog, 'filteredrevs', ()): if getattr(revlog, 'filteredrevs', ()):
raise error.ProgrammingError( raise error.ProgrammingError(
"cannot persist nodemap of a filtered changelog" "cannot persist nodemap of a filtered changelog"
) )
if revlog.nodemap_file is None: if revlog.nodemap_file is None:
msg = "calling persist nodemap on a revlog without the feature enableb" msg = "calling persist nodemap on a revlog without the feature enableb"
else: else:
fd.flush() fd.flush()
new_data = util.buffer(util.mmapread(fd, len(data))) new_data = util.buffer(util.mmapread(fd, len(data)))
target_docket.data_length = len(data) target_docket.data_length = len(data)
target_docket.tip_rev = revlog.tiprev() target_docket.tip_rev = revlog.tiprev()
target_docket.tip_node = revlog.node(target_docket.tip_rev) target_docket.tip_node = revlog.node(target_docket.tip_rev)
# EXP-TODO: if this is a cache, this should use a cache vfs, not a # EXP-TODO: if this is a cache, this should use a cache vfs, not a
# store vfs # store vfs
with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp: file_path = revlog.nodemap_file
if pending:
file_path += b'.a'
with revlog.opener(file_path, b'w', atomictemp=True) as fp:
fp.write(target_docket.serialize()) fp.write(target_docket.serialize())
revlog._nodemap_docket = target_docket revlog._nodemap_docket = target_docket
if feed_data: if feed_data:
revlog.index.update_nodemap_data(target_docket, new_data) revlog.index.update_nodemap_data(target_docket, new_data)
# EXP-TODO: if the transaction abort, we should remove the new data and # EXP-TODO: if the transaction abort, we should remove the new data and
# reinstall the old one. # reinstall the old one.
data.append(S_HEADER.pack(*headers)) data.append(S_HEADER.pack(*headers))
data.append(self.uid) data.append(self.uid)
data.append(self.tip_node) data.append(self.tip_node)
return b''.join(data) return b''.join(data)
def _rawdata_filepath(revlog, docket): def _rawdata_filepath(revlog, docket):
"""The (vfs relative) nodemap's rawdata file for a given uid""" """The (vfs relative) nodemap's rawdata file for a given uid"""
if revlog.nodemap_file.endswith(b'.n.a'):
prefix = revlog.nodemap_file[:-4]
else:
prefix = revlog.nodemap_file[:-2] prefix = revlog.nodemap_file[:-2]
return b"%s-%s.nd" % (prefix, docket.uid) return b"%s-%s.nd" % (prefix, docket.uid)
def _other_rawdata_filepath(revlog, docket): def _other_rawdata_filepath(revlog, docket):
prefix = revlog.nodemap_file[:-2] prefix = revlog.nodemap_file[:-2]
pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix) pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
new_file_path = _rawdata_filepath(revlog, docket) new_file_path = _rawdata_filepath(revlog, docket)
new_file_name = revlog.opener.basename(new_file_path) new_file_name = revlog.opener.basename(new_file_path)

tests/test-persistent-nodemap.t

$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 6ce944fafcee85af91f29ea5b51654cc6101ad7e tip-node: 6ce944fafcee85af91f29ea5b51654cc6101ad7e
data-length: 122944 data-length: 122944
data-unused: 0 data-unused: 0
$ hg log -r "$OTHERNODE" -T '{rev}\n' $ hg log -r "$OTHERNODE" -T '{rev}\n'
5002 5002
Check transaction related property
==================================
An up to date nodemap should be available to shell hooks,
$ echo dsljfl > a
$ hg add a
$ hg ci -m a
$ hg debugnodemap --metadata
uid: ???????????????? (glob)
tip-rev: 5003
tip-node: c91af76d172f1053cca41b83f7c2e4e514fe2bcf
data-length: 123008
data-unused: 0
$ echo babar2 > babar
$ hg ci -m 'babar2' --config "hooks.pretxnclose.nodemap-test=hg debugnodemap --metadata"
uid: ???????????????? (glob)
tip-rev: 5004
tip-node: ba87cd9559559e4b91b28cb140d003985315e031
data-length: 123328 (pure !)
data-length: 123328 (rust !)
data-length: 123136 (no-pure no-rust !)
data-unused: 192 (pure !)
data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !)
$ hg debugnodemap --metadata
uid: ???????????????? (glob)
tip-rev: 5004
tip-node: ba87cd9559559e4b91b28cb140d003985315e031
data-length: 123328 (pure !)
data-length: 123328 (rust !)
data-length: 123136 (no-pure no-rust !)
data-unused: 192 (pure !)
data-unused: 192 (rust !)
data-unused: 0 (no-pure no-rust !)