( )⚙ D10624 revlogv2: introduce a very basic docket file

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10624

revlogv2: introduce a very basic docket file
ClosedPublic

Authored by marmoute on May 3 2021, 7:54 AM.

Details

Reviewers
indygreg
Alphare
Group Reviewers
hg-reviewers
Commits
rHG616b8f412676: revlogv2: introduce a very basic docket file
Summary

This is the first stone toward using a docket file in revlogv2. Right now the
docket is very basic and only store the version number (which is -also- stored
into the index file…) and the other files have fixed name. This new
implementation break transactionally… but they are no test checking
transactionally for revlogv2… So I take this as an opportunity to start small.
They are no usage of revlogv2 outside of tests anyway.

The docket keeps the .i naming used by previous version index to preserve a
unique entry point. We could decide to use a different name and look it up
first, or to fully rework this in a future "store" version. However that does
not seems necessary right now.

We will re-introduces transactionality (and associated testing…) in a later
changesets.

A long list of TODOs have been added to the relevant comment.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

marmoute created this revision.May 3 2021, 7:54 AM
baymax updated this revision to Diff 27467.May 3 2021, 9:32 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

marmoute updated this revision to Diff 27542.May 4 2021, 10:13 AM
joerg.sonnenberger added a subscriber: joerg.sonnenberger.May 4 2021, 1:06 PM

Adding more files to the revlog layer sounds like a move in the wrong direction and the motivation here seems quite weak too. Please discuss this with an actual plan on the mailing list.

marmoute added a comment.May 5 2021, 5:58 PM
In D10624#161854, @joerg.sonnenberger wrote:

Adding more files to the revlog layer sounds like a move in the wrong direction and the motivation here seems quite weak too. Please discuss this with an actual plan on the mailing list.

The main goal here is to have a more extensive/flexible revlog format (for example using dictionnary for compression, or children cache, etc) and (eventually) strict mmap safety.

Reducing the overall number of file for filelogs is listed as an explicite TODO before freezing the format, so I don't think that extra file will be an issue by the time we freeze the format. On the short terms my plan are only about releasing a narrow "changelogv2" format with different index field as the current revlogv2 one. In since there is only one changelog and it already has a docket. I don't expect this to be an issue.

Various aspect of this were discussed at 5.7 sprint (in VC) and have some details about it here https://www.mercurial-scm.org/wiki/MMapPlan

baymax updated this revision to Diff 27870.May 11 2021, 1:44 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare accepted this revision.May 17 2021, 9:58 AM
Alphare added a subscriber: Alphare.

Just to clarify: this is a move in the right direction for mmap and rewriting purposes, but a solution for unified filelog/revlog will be needed to move all revlogs to revlogv2.

mercurial/revlogutils/docket.py
13

*insert joke about french pointers*

This revision is now accepted and ready to land.May 17 2021, 9:58 AM
marmoute added a commit: rHG616b8f412676: revlogv2: introduce a very basic docket file.May 17 2021, 5:51 PM
Closed by commit rHG616b8f412676: revlogv2: introduce a very basic docket file (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 27401

mercurial/changelog.py

def filteredrevs(self, val): def filteredrevs(self, val):
# Ensure all updates go through this function # Ensure all updates go through this function
assert isinstance(val, frozenset) assert isinstance(val, frozenset)
self._filteredrevs = val self._filteredrevs = val
self._filteredrevs_hashcache = {} self._filteredrevs_hashcache = {}
def delayupdate(self, tr): def delayupdate(self, tr):
"""delay visibility of index updates to other readers""" """delay visibility of index updates to other readers"""
if self._docket is not None:
return
if not self._delayed: if not self._delayed:
if len(self) == 0: if len(self) == 0:
self._divert = True self._divert = True
if self._realopener.exists(self._indexfile + b'.a'): if self._realopener.exists(self._indexfile + b'.a'):
self._realopener.unlink(self._indexfile + b'.a') self._realopener.unlink(self._indexfile + b'.a')
self.opener = _divertopener(self._realopener, self._indexfile) self.opener = _divertopener(self._realopener, self._indexfile)
else: else:

mercurial/configitems.py

) )
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'revisions.prefixhexnode', b'revisions.prefixhexnode',
default=False, default=False,
) )
# "out of experimental" todo list. # "out of experimental" todo list.
# #
# * stop storing version information in the index (it is already in the docket)
# * properly hide uncommitted content to other process
# * expose transaction content hooks during pre-commit validation
# * include management of a persistent nodemap in the main docket
# * enforce a "no-truncate" policy for mmap safety
# - for censoring operation
# - for stripping operation
# - for rollback operation
# * to grow a docket file to at least store the last offset of the data # * to grow a docket file to at least store the last offset of the data
# file when rewriting sidedata. # file when rewriting sidedata.
# * need a way of dealing with garbage data if we allow rewriting # * need a way of dealing with garbage data if we allow rewriting
# *existing* sidedata. # *existing* sidedata.
# * Exchange-wise, we will also need to do something more efficient than # * Exchange-wise, we will also need to do something more efficient than
# keeping references to the affected revlogs, especially memory-wise when # keeping references to the affected revlogs, especially memory-wise when
# rewriting sidedata. # rewriting sidedata.
# * Also... compress the sidedata? (this should be coming very soon) # * Also... compress the sidedata? (this should be coming very soon)
# * introduce a proper solution to reduce the number of filelog related files.
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'revlogv2', b'revlogv2',
default=None, default=None,
) )
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'revisions.disambiguatewithin', b'revisions.disambiguatewithin',

mercurial/revlog.py

util, util,
) )
from .interfaces import ( from .interfaces import (
repository, repository,
util as interfaceutil, util as interfaceutil,
) )
from .revlogutils import ( from .revlogutils import (
deltas as deltautil, deltas as deltautil,
docket as docketutil,
flagutil, flagutil,
nodemap as nodemaputil, nodemap as nodemaputil,
revlogv0, revlogv0,
sidedata as sidedatautil, sidedata as sidedatautil,
) )
from .utils import ( from .utils import (
storageutil, storageutil,
stringutil, stringutil,
analysis. Note: that this must be reliably be set by normal code, but analysis. Note: that this must be reliably be set by normal code, but
that test, debug, or performance measurement code might not set this to that test, debug, or performance measurement code might not set this to
accurate value. accurate value.
""" """
self.upperboundcomp = upperboundcomp self.upperboundcomp = upperboundcomp
self.radix = radix self.radix = radix
self._docket_file = None
self._indexfile = None self._indexfile = None
self._datafile = None self._datafile = None
self._nodemap_file = None self._nodemap_file = None
self.postfix = postfix self.postfix = postfix
self.opener = opener self.opener = opener
if persistentnodemap: if persistentnodemap:
self._nodemap_file = nodemaputil.get_nodemap_file(self) self._nodemap_file = nodemaputil.get_nodemap_file(self)
self._chainbasecache = util.lrucachedict(100) self._chainbasecache = util.lrucachedict(100)
# 2-tuple of (offset, data) of raw data from the revlog at an offset. # 2-tuple of (offset, data) of raw data from the revlog at an offset.
self._chunkcache = (0, b'') self._chunkcache = (0, b'')
# How much data to read and cache into the raw revlog data cache. # How much data to read and cache into the raw revlog data cache.
self._chunkcachesize = 65536 self._chunkcachesize = 65536
self._maxchainlen = None self._maxchainlen = None
self._deltabothparents = True self._deltabothparents = True
self.index = None self.index = None
self._docket = None
self._nodemap_docket = None self._nodemap_docket = None
# Mapping of partial identifiers to full nodes. # Mapping of partial identifiers to full nodes.
self._pcache = {} self._pcache = {}
# Mapping of revision integer to full node. # Mapping of revision integer to full node.
self._compengine = b'zlib' self._compengine = b'zlib'
self._compengineopts = {} self._compengineopts = {}
self._maxdeltachainspan = -1 self._maxdeltachainspan = -1
self._withsparseread = False self._withsparseread = False
msg %= (display_flag, self._format_version, self.display_id) msg %= (display_flag, self._format_version, self.display_id)
raise error.RevlogError(msg) raise error.RevlogError(msg)
features = FEATURES_BY_VERSION[self._format_version] features = FEATURES_BY_VERSION[self._format_version]
self._inline = features[b'inline'](self._format_flags) self._inline = features[b'inline'](self._format_flags)
self._generaldelta = features[b'generaldelta'](self._format_flags) self._generaldelta = features[b'generaldelta'](self._format_flags)
self.hassidedata = features[b'sidedata'] self.hassidedata = features[b'sidedata']
index_data = entry_data if not features[b'docket']:
self._indexfile = entry_point self._indexfile = entry_point
index_data = entry_data
else:
self._docket_file = entry_point
if self._initempty:
self._docket = docketutil.default_docket(self, header)
else:
self._docket = docketutil.parse_docket(self, entry_data)
self._indexfile = self._docket.index_filepath()
index_data = self._get_data(self._indexfile, mmapindexthreshold)
self._inline = False
# generaldelta implied by version 2 revlogs.
self._generaldelta = True
# the logic for persistent nodemap will be dealt with within the
# main docket, so disable it for now.
self._nodemap_file = None
if self.postfix is None or self.postfix == b'a': if self.postfix is None or self.postfix == b'a':
self._datafile = b'%s.d' % self.radix self._datafile = b'%s.d' % self.radix
else: else:
self._datafile = b'%s.d.%s' % (self.radix, self.postfix) self._datafile = b'%s.d.%s' % (self.radix, self.postfix)
self.nodeconstants = sha1nodeconstants self.nodeconstants = sha1nodeconstants
self.nullid = self.nodeconstants.nullid self.nullid = self.nodeconstants.nullid
if self._inline: if self._inline:
transaction.add(self._indexfile, dsize + isize) transaction.add(self._indexfile, dsize + isize)
else: else:
transaction.add(self._indexfile, isize) transaction.add(self._indexfile, isize)
try: try:
self._writinghandles = (ifh, dfh) self._writinghandles = (ifh, dfh)
try: try:
yield yield
if self._docket is not None:
self._docket.write(transaction)
finally: finally:
self._writinghandles = None self._writinghandles = None
finally: finally:
ifh.close() ifh.close()
finally: finally:
if dfh is not None: if dfh is not None:
dfh.close() dfh.close()
self.opener.stat(path).st_size for path in self.files() self.opener.stat(path).st_size for path in self.files()
) )
return d return d
def rewrite_sidedata(self, transaction, helpers, startrev, endrev): def rewrite_sidedata(self, transaction, helpers, startrev, endrev):
if not self.hassidedata: if not self.hassidedata:
return return
# inline are not yet supported because they suffer from an issue when # revlog formats with sidedata support does not support inline
# rewriting them (since it's not an append-only operation).
# See issue6485.
assert not self._inline assert not self._inline
if not helpers[1] and not helpers[2]: if not helpers[1] and not helpers[2]:
# Nothing to generate or remove # Nothing to generate or remove
return return
# changelog implement some "delayed" writing mechanism that assume that # changelog implement some "delayed" writing mechanism that assume that
# all index data is writen in append mode and is therefor incompatible # all index data is writen in append mode and is therefor incompatible
# with the seeked write done in this method. The use of such "delayed" # with the seeked write done in this method. The use of such "delayed"

mercurial/revlogutils/constants.py

return lambda flags: bool(flags & flag) return lambda flags: bool(flags & flag)
FEATURES_BY_VERSION = { FEATURES_BY_VERSION = {
REVLOGV0: { REVLOGV0: {
b'inline': _no, b'inline': _no,
b'generaldelta': _no, b'generaldelta': _no,
b'sidedata': False, b'sidedata': False,
b'docket': False,
}, },
REVLOGV1: { REVLOGV1: {
b'inline': _from_flag(FLAG_INLINE_DATA), b'inline': _from_flag(FLAG_INLINE_DATA),
b'generaldelta': _from_flag(FLAG_GENERALDELTA), b'generaldelta': _from_flag(FLAG_GENERALDELTA),
b'sidedata': False, b'sidedata': False,
b'docket': False,
}, },
REVLOGV2: { REVLOGV2: {
# There is a bug in the transaction handling when going from an # The point of inline-revlog is to reduce the number of files used in
# inline revlog to a separate index and data file. Turn it off until # the store. Using a docket defeat this purpose. So we needs other
# it's fixed, since v2 revlogs sometimes get rewritten on exchange. # means to reduce the number of files for revlogv2.
# See issue6485
b'inline': _no, b'inline': _no,
b'generaldelta': _yes, b'generaldelta': _yes,
b'sidedata': True, b'sidedata': True,
b'docket': True,
}, },
} }
SPARSE_REVLOG_MAX_CHAIN_LENGTH = 1000 SPARSE_REVLOG_MAX_CHAIN_LENGTH = 1000

mercurial/revlogutils/docket.py

  • This file was added.
# docket - code related to revlog "docket"
#
# Copyright 2021 Pierre-Yves David <pierre-yves.david@octobus.net>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
### Revlog docket file
#
# The revlog is stored on disk using multiple files:
#
# * a small docket file, containing metadata and pointeur,
#
AlphareUnsubmitted
Not Done

*insert joke about french pointers*

Alphare: *insert joke about french pointers*
# * an index file, containing fixed width information about revisions,
#
# * a data file, containing variable width data for these revisions,
from __future__ import absolute_import
import struct
from . import (
constants,
)
# Docket format
#
# * 4 bytes: revlog version
# | This is mandatory as docket must be compatible with the previous
# | revlog index header.
S_HEADER = struct.Struct(constants.INDEX_HEADER.format)
class RevlogDocket(object):
"""metadata associated with revlog"""
def __init__(self, revlog, version_header=None):
self._version_header = version_header
self._dirty = False
self._radix = revlog.radix
self._path = revlog._docket_file
self._opener = revlog.opener
def index_filepath(self):
"""file path to the current index file associated to this docket"""
# very simplistic version at first
return b"%s.idx" % self._radix
def write(self, transaction):
"""write the modification of disk if any
This make the new content visible to all process"""
if self._dirty:
transaction.addbackup(self._path, location=b'store')
with self._opener(self._path, mode=b'w', atomictemp=True) as f:
f.write(self._serialize())
self._dirty = False
def _serialize(self):
return S_HEADER.pack(self._version_header)
def default_docket(revlog, version_header):
"""given a revlog version a new docket object for the given revlog"""
if (version_header & 0xFFFF) != constants.REVLOGV2:
return None
docket = RevlogDocket(revlog, version_header=version_header)
docket._dirty = True
return docket
def parse_docket(revlog, data):
"""given some docket data return a docket object for the given revlog"""
header = S_HEADER.unpack(data[: S_HEADER.size])
(version_header,) = header
docket = RevlogDocket(
revlog,
version_header=version_header,
)
return docket

mercurial/store.py

b'00changelog.d', b'00changelog.d',
b'00changelog.i', b'00changelog.i',
b'phaseroots', b'phaseroots',
b'obsstore', b'obsstore',
b'requires', b'requires',
] ]
REVLOG_FILES_MAIN_EXT = (b'.i', b'i.tmpcensored') REVLOG_FILES_MAIN_EXT = (b'.i', b'i.tmpcensored')
REVLOG_FILES_OTHER_EXT = (b'.d', b'.n', b'.nd', b'd.tmpcensored') REVLOG_FILES_OTHER_EXT = (b'.idx', b'.d', b'.n', b'.nd', b'd.tmpcensored')
# files that are "volatile" and might change between listing and streaming # files that are "volatile" and might change between listing and streaming
# #
# note: the ".nd" file are nodemap data and won't "change" but they might be # note: the ".nd" file are nodemap data and won't "change" but they might be
# deleted. # deleted.
REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd') REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd')
# some exception to the above matching # some exception to the above matching
EXCLUDED = re.compile(b'.*undo\.[^/]+\.nd?$') EXCLUDED = re.compile(b'.*undo\.[^/]+\.(nd?|i)$')
def is_revlog(f, kind, st): def is_revlog(f, kind, st):
if kind != stat.S_IFREG: if kind != stat.S_IFREG:
return None return None
return revlog_type(f) return revlog_type(f)
def revlog_type(f): def revlog_type(f):
if f.endswith(REVLOG_FILES_MAIN_EXT): if f.endswith(REVLOG_FILES_MAIN_EXT) and EXCLUDED.match(f) is None:
return FILEFLAGS_REVLOG_MAIN return FILEFLAGS_REVLOG_MAIN
elif f.endswith(REVLOG_FILES_OTHER_EXT) and EXCLUDED.match(f) is None: elif f.endswith(REVLOG_FILES_OTHER_EXT) and EXCLUDED.match(f) is None:
t = FILETYPE_FILELOG_OTHER t = FILETYPE_FILELOG_OTHER
if f.endswith(REVLOG_FILES_VOLATILE_EXT): if f.endswith(REVLOG_FILES_VOLATILE_EXT):
t |= FILEFLAGS_VOLATILE t |= FILEFLAGS_VOLATILE
return t return t