This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7839

nodemap: delete older raw data file when creating a new ones
ClosedPublic

Authored by marmoute on Jan 11 2020, 12:03 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG563dfdfd01a4: nodemap: delete older raw data file when creating a new ones
Summary

When we write new full files, it replace an older one with a different name. We
add the associated cleanup for the older file to be removed after the
transaction.

We delete all file matching the expected pattern to give use extra chance to
delete orphan files we might have failed to delete earlier.

Note: eventually we won't rewrite all data for each transaction. This is coming
in later changesets.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

marmoute created this revision.Jan 11 2020, 12:03 PM
marmoute updated this revision to Diff 19294.Jan 15 2020, 9:54 AM
martinvonz added a subscriber: martinvonz.Jan 23 2020, 4:32 PM
martinvonz added inline comments.
mercurial/revlogutils/nodemap.py
151

meant to escape the "." in the regex?

marmoute added inline comments.Jan 31 2020, 4:21 AM
mercurial/revlogutils/nodemap.py
151

yes… good catch ☺

marmoute retitled this revision from nodemap: deleted older raw data file when creating a new ones to nodemap: delete older raw data file when creating a new ones.Jan 31 2020, 10:28 AM
marmoute updated this revision to Diff 19755.
marmoute updated this revision to Diff 19785.Jan 31 2020, 5:58 PM
marmoute updated this revision to Diff 19829.Feb 2 2020, 3:17 AM

small doc update on .#s[1]

marmoute updated this revision to Diff 19887.Feb 4 2020, 7:24 PM

rebase to latest default

marmoute added a commit: rHG563dfdfd01a4: nodemap: delete older raw data file when creating a new ones.Feb 10 2020, 8:24 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG563dfdfd01a4: nodemap: delete older raw data file when creating a new ones (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 19755

mercurial/revlogutils/nodemap.py

# nodemap.py - nodemap related code and utilities # nodemap.py - nodemap related code and utilities
# #
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net> # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
# Copyright 2019 George Racinet <georges.racinet@octobus.net> # Copyright 2019 George Racinet <georges.racinet@octobus.net>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import os import os
import re
import struct import struct
from .. import ( from .. import (
error, error,
node as nodemod, node as nodemod,
) )
"cannot persist nodemap of a filtered changelog" "cannot persist nodemap of a filtered changelog"
) )
if revlog.nodemap_file is None: if revlog.nodemap_file is None:
msg = "calling persist nodemap on a revlog without the feature enableb" msg = "calling persist nodemap on a revlog without the feature enableb"
raise error.ProgrammingError(msg) raise error.ProgrammingError(msg)
data = persistent_data(revlog.index) data = persistent_data(revlog.index)
uid = _make_uid() uid = _make_uid()
datafile = _rawdata_filepath(revlog, uid) datafile = _rawdata_filepath(revlog, uid)
olds = _other_rawdata_filepath(revlog, uid)
if olds:
realvfs = getattr(revlog, '_realopener', revlog.opener)
def cleanup(tr):
for oldfile in olds:
realvfs.tryunlink(oldfile)
callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
tr.addpostclose(callback_id, cleanup)
# EXP-TODO: if this is a cache, this should use a cache vfs, not a # EXP-TODO: if this is a cache, this should use a cache vfs, not a
# store vfs # store vfs
with revlog.opener(datafile, 'w') as fd: with revlog.opener(datafile, 'w') as fd:
fd.write(data) fd.write(data)
# EXP-TODO: if this is a cache, this should use a cache vfs, not a # EXP-TODO: if this is a cache, this should use a cache vfs, not a
# store vfs # store vfs
with revlog.opener(revlog.nodemap_file, 'w', atomictemp=True) as fp: with revlog.opener(revlog.nodemap_file, 'w', atomictemp=True) as fp:
fp.write(_serialize_docket(uid)) fp.write(_serialize_docket(uid))
def _rawdata_filepath(revlog, uid): def _rawdata_filepath(revlog, uid):
"""The (vfs relative) nodemap's rawdata file for a given uid""" """The (vfs relative) nodemap's rawdata file for a given uid"""
prefix = revlog.nodemap_file[:-2] prefix = revlog.nodemap_file[:-2]
return b"%s-%s.nd" % (prefix, uid) return b"%s-%s.nd" % (prefix, uid)
def _other_rawdata_filepath(revlog, uid):
prefix = revlog.nodemap_file[:-2]
pattern = re.compile(b"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
martinvonzUnsubmitted
Not Done

meant to escape the "." in the regex?

martinvonz: meant to escape the "." in the regex?
marmouteAuthorUnsubmitted
Done

yes… good catch ☺

marmoute: yes… good catch ☺
new_file_path = _rawdata_filepath(revlog, uid)
new_file_name = revlog.opener.basename(new_file_path)
dirpath = revlog.opener.dirname(new_file_path)
others = []
for f in revlog.opener.listdir(dirpath):
if pattern.match(f) and f != new_file_name:
others.append(f)
return others
### Nodemap Trie ### Nodemap Trie
# #
# This is a simple reference implementation to compute and persist a nodemap # This is a simple reference implementation to compute and persist a nodemap
# trie. This reference implementation is write only. The python version of this # trie. This reference implementation is write only. The python version of this
# is not expected to be actually used, since it wont provide performance # is not expected to be actually used, since it wont provide performance
# improvement over existing non-persistent C implementation. # improvement over existing non-persistent C implementation.
# #
# The nodemap is persisted as Trie using 4bits-address/16-entries block. each # The nodemap is persisted as Trie using 4bits-address/16-entries block. each

tests/test-persistent-nodemap.t

=================================== ===================================
Test the persistent on-disk nodemap Test the persistent on-disk nodemap
=================================== ===================================
$ hg init test-repo $ hg init test-repo
$ cd test-repo $ cd test-repo
$ cat << EOF >> .hg/hgrc $ cat << EOF >> .hg/hgrc
> [experimental] > [experimental]
> exp-persistent-nodemap=yes > exp-persistent-nodemap=yes
> EOF > EOF
$ hg debugbuilddag .+5000 $ hg debugbuilddag .+5000
$ f --size .hg/store/00changelog.n $ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=18 .hg/store/00changelog.n: size=18
$ f --sha256 .hg/store/00changelog-*.nd
.hg/store/00changelog-????????????????.nd: sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c (glob)
$ hg debugnodemap --dump-new | f --sha256 --size $ hg debugnodemap --dump-new | f --sha256 --size
size=245760, sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c size=245760, sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c
$ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size $ hg debugnodemap --dump-disk | f --sha256 --bytes=256 --hexdump --size
size=245760, sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c size=245760, sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c
0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0030: ff ff ff ff ff ff fa c2 ff ff ff ff ff ff ff ff |................| 0030: ff ff ff ff ff ff fa c2 ff ff ff ff ff ff ff ff |................|
0040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ed b3 |................| 0060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ed b3 |................|
0070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ee 34 |...............4| 0080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ee 34 |...............4|
0090: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff |................| 0090: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff |................|
00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
add a new commit
$ hg up
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo foo > foo
$ hg add foo
$ hg ci -m 'foo'
$ f --size .hg/store/00changelog.n
.hg/store/00changelog.n: size=18
$ f --sha256 .hg/store/00changelog-*.nd --size
.hg/store/00changelog-????????????????.nd: size=245760, sha256=e6ee5d59afaab2cb1afae1077715be280578d29df508bd3dd9d74a994bc555e7 (glob)