This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8193

nodemap: automatically "vacuum" the persistent nodemap when too sparse
ClosedPublic

Authored by marmoute on Feb 28 2020, 1:55 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGc70bcaf7927b: nodemap: automatically "vacuum" the persistent nodemap when too sparse
Summary

We arbitrarily pick "10%" as the threshold.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Feb 28 2020, 1:55 PM
marmoute updated this revision to Diff 20585.Mar 6 2020, 6:38 PM

eventless rebase past D8255

marmoute updated this revision to Diff 20688.Mar 10 2020, 9:36 PM

rebase above latest default up to landed-D8182

marmoute updated this revision to Diff 20744.Mar 11 2020, 2:20 PM
marmoute updated this revision to Diff 20897.Mar 26 2020, 9:13 AM
marmoute added a commit: rHGc70bcaf7927b: nodemap: automatically "vacuum" the persistent nodemap when too sparse.Apr 1 2020, 11:21 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGc70bcaf7927b: nodemap: automatically "vacuum" the persistent nodemap when too sparse (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 20935

mercurial/configitems.py

coreconfigitem( coreconfigitem(
b'devel', b'warn-empty-changegroup', default=False, b'devel', b'warn-empty-changegroup', default=False,
) )
coreconfigitem( coreconfigitem(
b'devel', b'legacy.exchange', default=list, b'devel', b'legacy.exchange', default=list,
) )
# TODO before getting `persistent-nodemap` out of experimental # TODO before getting `persistent-nodemap` out of experimental
# #
# * regenerate a new nodemap when the unused/total ration is to high
# * decide for a "status" of the persistent nodemap and associated location # * decide for a "status" of the persistent nodemap and associated location
# - part of the store next the revlog itself (new requirements) # - part of the store next the revlog itself (new requirements)
# - part of the cache directory # - part of the cache directory
# - part of an `index` directory # - part of an `index` directory
# (https://www.mercurial-scm.org/wiki/ComputedIndexPlan) # (https://www.mercurial-scm.org/wiki/ComputedIndexPlan)
# * do we want to use this for more than just changelog? if so we need: # * do we want to use this for more than just changelog? if so we need:
# - simpler "pending" logic for them # - simpler "pending" logic for them
# - double check the memory story (we dont want to keep all revlog in memory) # - double check the memory story (we dont want to keep all revlog in memory)

mercurial/revlogutils/nodemap.py

# first attemp an incremental update of the data # first attemp an incremental update of the data
if can_incremental and ondisk_docket is not None: if can_incremental and ondisk_docket is not None:
target_docket = revlog._nodemap_docket.copy() target_docket = revlog._nodemap_docket.copy()
( (
src_docket, src_docket,
data_changed_count, data_changed_count,
data, data,
) = revlog.index.nodemap_data_incremental() ) = revlog.index.nodemap_data_incremental()
new_length = target_docket.data_length + len(data)
new_unused = target_docket.data_unused + data_changed_count
if src_docket != target_docket: if src_docket != target_docket:
data = None data = None
elif new_length <= (new_unused * 10): # under 10% of unused data
data = None
else: else:
datafile = _rawdata_filepath(revlog, target_docket) datafile = _rawdata_filepath(revlog, target_docket)
# EXP-TODO: if this is a cache, this should use a cache vfs, not a # EXP-TODO: if this is a cache, this should use a cache vfs, not a
# store vfs # store vfs
new_length = target_docket.data_length + len(data)
tr.add(datafile, target_docket.data_length) tr.add(datafile, target_docket.data_length)
with revlog.opener(datafile, b'r+') as fd: with revlog.opener(datafile, b'r+') as fd:
fd.seek(target_docket.data_length) fd.seek(target_docket.data_length)
fd.write(data) fd.write(data)
if feed_data: if feed_data:
if use_mmap: if use_mmap:
fd.seek(0) fd.seek(0)
new_data = fd.read(new_length) new_data = fd.read(new_length)
else: else:
fd.flush() fd.flush()
new_data = util.buffer(util.mmapread(fd, new_length)) new_data = util.buffer(util.mmapread(fd, new_length))
target_docket.data_length = new_length target_docket.data_length = new_length
target_docket.data_unused += data_changed_count target_docket.data_unused = new_unused
if data is None: if data is None:
# otherwise fallback to a full new export # otherwise fallback to a full new export
target_docket = NodeMapDocket() target_docket = NodeMapDocket()
datafile = _rawdata_filepath(revlog, target_docket) datafile = _rawdata_filepath(revlog, target_docket)
if util.safehasattr(revlog.index, "nodemap_data_all"): if util.safehasattr(revlog.index, "nodemap_data_all"):
data = revlog.index.nodemap_data_all() data = revlog.index.nodemap_data_all()
else: else:

tests/test-persistent-nodemap.t

the nodemap should detect the changelog have been tampered with and recover. the nodemap should detect the changelog have been tampered with and recover.
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 42bf3068c7ddfdfded53c4eb11d02266faeebfee tip-node: 42bf3068c7ddfdfded53c4eb11d02266faeebfee
data-length: 123456 (pure !) data-length: 123456 (pure !)
data-length: 246464 (rust !) data-length: 123008 (rust !)
data-length: 123008 (no-pure no-rust !) data-length: 123008 (no-pure no-rust !)
data-unused: 448 (pure !) data-unused: 448 (pure !)
data-unused: 123904 (rust !) data-unused: 0 (rust !)
data-unused: 0 (no-pure no-rust !) data-unused: 0 (no-pure no-rust !)
data-unused: 50.273% (rust !) data-unused: 0.000% (rust !)
data-unused: 0.363% (pure !) data-unused: 0.363% (pure !)
data-unused: 0.000% (no-pure no-rust !) data-unused: 0.000% (no-pure no-rust !)
$ cp -f ../tmp-copies/* .hg/store/ $ cp -f ../tmp-copies/* .hg/store/
$ hg debugnodemap --metadata $ hg debugnodemap --metadata
uid: ???????????????? (glob) uid: ???????????????? (glob)
tip-rev: 5002 tip-rev: 5002
tip-node: 6ce944fafcee85af91f29ea5b51654cc6101ad7e tip-node: 6ce944fafcee85af91f29ea5b51654cc6101ad7e