( )⚙ D9993 sidedata: move to new sidedata storage in revlogv2

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9993

sidedata: move to new sidedata storage in revlogv2
ClosedPublic

Authored by Alphare on Feb 15 2021, 5:29 AM.

Details

Reviewers
indygreg
marmoute
Group Reviewers
hg-reviewers
Commits
rHG3d740058b467: sidedata: move to new sidedata storage in revlogv2
Summary

The current (experimental) sidedata system uses flagprocessors to signify the
presence and store/retrieve sidedata from the raw revlog data. This proved to be
quite fragile from an exchange perspective and a lot more complex than simply
having a dedicated space in the new revlog format.

This change does not handle exchange (ironically), so the test for amend - that
uses a bundle - is broken. This functionality is split into the next patches.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

Alphare created this revision.Feb 15 2021, 5:29 AM
baymax updated this revision to Diff 25989.Mar 1 2021, 11:51 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

marmoute requested changes to this revision.Mar 2 2021, 1:45 PM
marmoute added a subscriber: marmoute.

The request for change is actually about comment that are related to earlier changeset, but that I only understood in that changesets.

mercurial/revlog.py
2362–2363

what does merging revlog means ?

2364–2365

This is premature and we should not include this in this series.

2386–2390

Ha, so this is what this is. I don't think we should include them in the format at this point. The only new feature we have are the sidedata and we should focus on that.

This revision now requires changes to proceed.Mar 2 2021, 1:45 PM
Alphare added inline comments.Mar 3 2021, 6:52 AM
mercurial/revlog.py
2386–2390

Accounting for them in the format was easy enough, I don't think it's worth any more attention by reworking several changesets to remove them just to re-do that work in a few weeks.

marmoute added inline comments.Mar 3 2021, 7:29 AM
mercurial/revlog.py
2386–2390

It is unlikley for that work to be done "in a few week" and even if so, we have no garantee the current format fit the need for these data when that work is actually done. So is is really premature.

To me, the important part of this revlogv2 it to setup a more versatile codebase where adding more field/data get simpler. So, not having them part of the initial batch should be fine, as adding these one, or others, later should be simpler.

Alphare marked 3 inline comments as done.Mar 4 2021, 10:06 AM
Alphare updated this revision to Diff 26087.Mar 4 2021, 10:18 AM
marmoute accepted this revision.Mar 9 2021, 11:14 AM

Thanks for the update.

Alphare added a commit: rHG3d740058b467: sidedata: move to new sidedata storage in revlogv2.Mar 15 2021, 2:31 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG3d740058b467: sidedata: move to new sidedata storage in revlogv2 (authored by Alphare). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 25629

hgext/lfs/wrapper.py

hgmeta = {} hgmeta = {}
for k in p.keys(): for k in p.keys():
if k.startswith(b'x-hg-'): if k.startswith(b'x-hg-'):
name = k[len(b'x-hg-') :] name = k[len(b'x-hg-') :]
hgmeta[name] = p[k] hgmeta[name] = p[k]
if hgmeta or text.startswith(b'\1\n'): if hgmeta or text.startswith(b'\1\n'):
text = storageutil.packmeta(hgmeta, text) text = storageutil.packmeta(hgmeta, text)
return (text, True, {}) return (text, True)
def writetostore(self, text, sidedata): def writetostore(self, text):
# hg filelog metadata (includes rename, etc) # hg filelog metadata (includes rename, etc)
hgmeta, offset = storageutil.parsemeta(text) hgmeta, offset = storageutil.parsemeta(text)
if offset and offset > 0: if offset and offset > 0:
# lfs blob does not contain hg filelog metadata # lfs blob does not contain hg filelog metadata
text = text[offset:] text = text[offset:]
# git-lfs only supports sha256 # git-lfs only supports sha256
oid = hex(hashlib.sha256(text).digest()) oid = hex(hashlib.sha256(text).digest())

hgext/remotefilelog/remotefilelog.py

cachedelta=None, cachedelta=None,
node=None, node=None,
flags=revlog.REVIDX_DEFAULT_FLAGS, flags=revlog.REVIDX_DEFAULT_FLAGS,
sidedata=None, sidedata=None,
): ):
# text passed to "addrevision" includes hg filelog metadata header # text passed to "addrevision" includes hg filelog metadata header
if node is None: if node is None:
node = storageutil.hashrevisionsha1(text, p1, p2) node = storageutil.hashrevisionsha1(text, p1, p2)
if sidedata is None:
sidedata = {}
meta, metaoffset = storageutil.parsemeta(text) meta, metaoffset = storageutil.parsemeta(text)
rawtext, validatehash = flagutil.processflagswrite( rawtext, validatehash = flagutil.processflagswrite(
self, text, flags, sidedata=sidedata self,
text,
flags,
) )
return self.addrawrevision( return self.addrawrevision(
rawtext, rawtext,
transaction, transaction,
linknode, linknode,
p1, p1,
p2, p2,
node, node,

mercurial/revlog.py

_zlibdecompress = zlib.decompress _zlibdecompress = zlib.decompress
# max size of revlog with inline data # max size of revlog with inline data
_maxinline = 131072 _maxinline = 131072
_chunksize = 1048576 _chunksize = 1048576
# Flag processors for REVIDX_ELLIPSIS. # Flag processors for REVIDX_ELLIPSIS.
def ellipsisreadprocessor(rl, text): def ellipsisreadprocessor(rl, text):
return text, False, {} return text, False
def ellipsiswriteprocessor(rl, text, sidedata): def ellipsiswriteprocessor(rl, text):
return text, False return text, False
def ellipsisrawprocessor(rl, text): def ellipsisrawprocessor(rl, text):
return False return False
ellipsisprocessor = ( ellipsisprocessor = (
self._compengineopts[b'zlib.level'] = opts[b'zlib.level'] self._compengineopts[b'zlib.level'] = opts[b'zlib.level']
if b'zstd.level' in opts: if b'zstd.level' in opts:
self._compengineopts[b'zstd.level'] = opts[b'zstd.level'] self._compengineopts[b'zstd.level'] = opts[b'zstd.level']
if b'maxdeltachainspan' in opts: if b'maxdeltachainspan' in opts:
self._maxdeltachainspan = opts[b'maxdeltachainspan'] self._maxdeltachainspan = opts[b'maxdeltachainspan']
if self._mmaplargeindex and b'mmapindexthreshold' in opts: if self._mmaplargeindex and b'mmapindexthreshold' in opts:
mmapindexthreshold = opts[b'mmapindexthreshold'] mmapindexthreshold = opts[b'mmapindexthreshold']
self.hassidedata = bool(opts.get(b'side-data', False)) self.hassidedata = bool(opts.get(b'side-data', False))
if self.hassidedata:
self._flagprocessors[REVIDX_SIDEDATA] = sidedatautil.processors
self._sparserevlog = bool(opts.get(b'sparse-revlog', False)) self._sparserevlog = bool(opts.get(b'sparse-revlog', False))
withsparseread = bool(opts.get(b'with-sparse-read', False)) withsparseread = bool(opts.get(b'with-sparse-read', False))
# sparse-revlog forces sparse-read # sparse-revlog forces sparse-read
self._withsparseread = self._sparserevlog or withsparseread self._withsparseread = self._sparserevlog or withsparseread
if b'sparse-read-density-threshold' in opts: if b'sparse-read-density-threshold' in opts:
self._srdensitythreshold = opts[b'sparse-read-density-threshold'] self._srdensitythreshold = opts[b'sparse-read-density-threshold']
if b'sparse-read-min-gap-size' in opts: if b'sparse-read-min-gap-size' in opts:
self._srmingapsize = opts[b'sparse-read-min-gap-size'] self._srmingapsize = opts[b'sparse-read-min-gap-size']
return int(self.index[rev][0] >> 16) return int(self.index[rev][0] >> 16)
def flags(self, rev): def flags(self, rev):
return self.index[rev][0] & 0xFFFF return self.index[rev][0] & 0xFFFF
def length(self, rev): def length(self, rev):
return self.index[rev][1] return self.index[rev][1]
def sidedata_length(self, rev):
if self.version & 0xFFFF != REVLOGV2:
return 0
return self.index[rev][11]
def rawsize(self, rev): def rawsize(self, rev):
"""return the length of the uncompressed text for a given revision""" """return the length of the uncompressed text for a given revision"""
l = self.index[rev][2] l = self.index[rev][2]
if l >= 0: if l >= 0:
return l return l
t = self.rawdata(rev) t = self.rawdata(rev)
return len(t) return len(t)
except IndexError: except IndexError:
if rev == wdirrev: if rev == wdirrev:
raise error.WdirUnsupported raise error.WdirUnsupported
raise raise
# Derived from index values. # Derived from index values.
def end(self, rev): def end(self, rev):
return self.start(rev) + self.length(rev) return self.start(rev) + self.length(rev) + self.sidedata_length(rev)
def parents(self, node): def parents(self, node):
i = self.index i = self.index
d = i[self.rev(node)] d = i[self.rev(node)]
return i[d[5]][7], i[d[6]][7] # map revisions to nodes inline return i[d[5]][7], i[d[6]][7] # map revisions to nodes inline
def chainlen(self, rev): def chainlen(self, rev):
return self._chaininfo(rev)[0] return self._chaininfo(rev)[0]
"""deprecated entry point to access flag processors""" """deprecated entry point to access flag processors"""
msg = b'_processflag(...) use the specialized variant' msg = b'_processflag(...) use the specialized variant'
util.nouideprecwarn(msg, b'5.2', stacklevel=2) util.nouideprecwarn(msg, b'5.2', stacklevel=2)
if raw: if raw:
return text, flagutil.processflagsraw(self, text, flags) return text, flagutil.processflagsraw(self, text, flags)
elif operation == b'read': elif operation == b'read':
return flagutil.processflagsread(self, text, flags) return flagutil.processflagsread(self, text, flags)
else: # write operation else: # write operation
return flagutil.processflagswrite(self, text, flags, None) return flagutil.processflagswrite(self, text, flags)
def revision(self, nodeorrev, _df=None, raw=False): def revision(self, nodeorrev, _df=None, raw=False):
"""return an uncompressed revision of a given node or revision """return an uncompressed revision of a given node or revision
number. number.
_df - an existing file handle to read from. (internal-only) _df - an existing file handle to read from. (internal-only)
raw - an optional argument specifying if the revision data is to be raw - an optional argument specifying if the revision data is to be
treated as raw data when applying flag transforms. 'raw' should be set treated as raw data when applying flag transforms. 'raw' should be set
# fast path the special `nullid` rev # fast path the special `nullid` rev
if node == nullid: if node == nullid:
return b"", {} return b"", {}
# ``rawtext`` is the text as stored inside the revlog. Might be the # ``rawtext`` is the text as stored inside the revlog. Might be the
# revision or might need to be processed to retrieve the revision. # revision or might need to be processed to retrieve the revision.
rev, rawtext, validated = self._rawtext(node, rev, _df=_df) rev, rawtext, validated = self._rawtext(node, rev, _df=_df)
if self.version & 0xFFFF == REVLOGV2:
if rev is None:
rev = self.rev(node)
sidedata = self._sidedata(rev)
else:
sidedata = {}
if raw and validated: if raw and validated:
# if we don't want to process the raw text and that raw # if we don't want to process the raw text and that raw
# text is cached, we can exit early. # text is cached, we can exit early.
return rawtext, {} return rawtext, sidedata
if rev is None: if rev is None:
rev = self.rev(node) rev = self.rev(node)
# the revlog's flag for this revision # the revlog's flag for this revision
# (usually alter its state or content) # (usually alter its state or content)
flags = self.flags(rev) flags = self.flags(rev)
if validated and flags == REVIDX_DEFAULT_FLAGS: if validated and flags == REVIDX_DEFAULT_FLAGS:
# no extra flags set, no flag processor runs, text = rawtext # no extra flags set, no flag processor runs, text = rawtext
return rawtext, {} return rawtext, sidedata
sidedata = {}
if raw: if raw:
validatehash = flagutil.processflagsraw(self, rawtext, flags) validatehash = flagutil.processflagsraw(self, rawtext, flags)
text = rawtext text = rawtext
else: else:
try:
r = flagutil.processflagsread(self, rawtext, flags) r = flagutil.processflagsread(self, rawtext, flags)
except error.SidedataHashError as exc: text, validatehash = r
msg = _(b"integrity check failed on %s:%s sidedata key %d")
msg %= (self.indexfile, pycompat.bytestr(rev), exc.sidedatakey)
raise error.RevlogError(msg)
text, validatehash, sidedata = r
if validatehash: if validatehash:
self.checkhash(text, node, rev=rev) self.checkhash(text, node, rev=rev)
if not validated: if not validated:
self._revisioncache = (node, rev, rawtext) self._revisioncache = (node, rev, rawtext)
return text, sidedata return text, sidedata
def _rawtext(self, node, rev, _df=None): def _rawtext(self, node, rev, _df=None):
if basetext is None: if basetext is None:
basetext = bytes(bins[0]) basetext = bytes(bins[0])
bins = bins[1:] bins = bins[1:]
rawtext = mdiff.patches(basetext, bins) rawtext = mdiff.patches(basetext, bins)
del basetext # let us have a chance to free memory early del basetext # let us have a chance to free memory early
return (rev, rawtext, False) return (rev, rawtext, False)
def _sidedata(self, rev):
"""Return the sidedata for a given revision number."""
index_entry = self.index[rev]
sidedata_offset = index_entry[10]
sidedata_size = index_entry[11]
if self._inline:
sidedata_offset += self._io.size * (1 + rev)
if sidedata_size == 0:
return {}
segment = self._getsegment(sidedata_offset, sidedata_size)
sidedata = sidedatautil.deserialize_sidedata(segment)
return sidedata
def rawdata(self, nodeorrev, _df=None): def rawdata(self, nodeorrev, _df=None):
"""return an uncompressed raw data of a given node or revision number. """return an uncompressed raw data of a given node or revision number.
_df - an existing file handle to read from. (internal-only) _df - an existing file handle to read from. (internal-only)
""" """
return self._revisiondata(nodeorrev, _df, raw=True)[0] return self._revisiondata(nodeorrev, _df, raw=True)[0]
def hash(self, text, p1, p2): def hash(self, text, p1, p2):
""" """
if link == nullrev: if link == nullrev:
raise error.RevlogError( raise error.RevlogError(
_(b"attempted to add linkrev -1 to %s") % self.indexfile _(b"attempted to add linkrev -1 to %s") % self.indexfile
) )
if sidedata is None: if sidedata is None:
sidedata = {} sidedata = {}
flags = flags & ~REVIDX_SIDEDATA
elif not self.hassidedata: elif not self.hassidedata:
raise error.ProgrammingError( raise error.ProgrammingError(
_(b"trying to add sidedata to a revlog who don't support them") _(b"trying to add sidedata to a revlog who don't support them")
) )
else:
flags |= REVIDX_SIDEDATA
if flags: if flags:
node = node or self.hash(text, p1, p2) node = node or self.hash(text, p1, p2)
rawtext, validatehash = flagutil.processflagswrite( rawtext, validatehash = flagutil.processflagswrite(self, text, flags)
self, text, flags, sidedata=sidedata
)
# If the flag processor modifies the revision data, ignore any provided # If the flag processor modifies the revision data, ignore any provided
# cachedelta. # cachedelta.
if rawtext != text: if rawtext != text:
cachedelta = None cachedelta = None
if len(rawtext) > _maxentrysize: if len(rawtext) > _maxentrysize:
raise error.RevlogError( raise error.RevlogError(
transaction, transaction,
link, link,
p1, p1,
p2, p2,
node, node,
flags, flags,
cachedelta=cachedelta, cachedelta=cachedelta,
deltacomputer=deltacomputer, deltacomputer=deltacomputer,
sidedata=sidedata,
) )
def addrawrevision( def addrawrevision(
self, self,
rawtext, rawtext,
transaction, transaction,
link, link,
p1, p1,
p2, p2,
node, node,
flags, flags,
cachedelta=None, cachedelta=None,
deltacomputer=None, deltacomputer=None,
sidedata=None,
): ):
"""add a raw revision with known flags, node and parents """add a raw revision with known flags, node and parents
useful when reusing a revision not stored in this revlog (ex: received useful when reusing a revision not stored in this revlog (ex: received
over wire, or read from an external bundle). over wire, or read from an external bundle).
""" """
dfh = None dfh = None
if not self._inline: if not self._inline:
dfh = self._datafp(b"a+") dfh = self._datafp(b"a+")
ifh = self._indexfp(b"a+") ifh = self._indexfp(b"a+")
try: try:
return self._addrevision( return self._addrevision(
node, node,
rawtext, rawtext,
transaction, transaction,
link, link,
p1, p1,
p2, p2,
flags, flags,
cachedelta, cachedelta,
ifh, ifh,
dfh, dfh,
deltacomputer=deltacomputer, deltacomputer=deltacomputer,
sidedata=sidedata,
) )
finally: finally:
if dfh: if dfh:
dfh.close() dfh.close()
ifh.close() ifh.close()
def compress(self, data): def compress(self, data):
"""Generate a possibly-compressed representation of data.""" """Generate a possibly-compressed representation of data."""
p1, p1,
p2, p2,
flags, flags,
cachedelta, cachedelta,
ifh, ifh,
dfh, dfh,
alwayscache=False, alwayscache=False,
deltacomputer=None, deltacomputer=None,
sidedata=None,
): ):
"""internal function to add revisions to the log """internal function to add revisions to the log
see addrevision for argument descriptions. see addrevision for argument descriptions.
note: "addrevision" takes non-raw text, "_addrevision" takes raw text. note: "addrevision" takes non-raw text, "_addrevision" takes raw text.
if "deltacomputer" is not provided or None, a defaultdeltacomputer will if "deltacomputer" is not provided or None, a defaultdeltacomputer will
if deltacomputer is None: if deltacomputer is None:
deltacomputer = deltautil.deltacomputer(self) deltacomputer = deltautil.deltacomputer(self)
revinfo = _revisioninfo(node, p1, p2, btext, textlen, cachedelta, flags) revinfo = _revisioninfo(node, p1, p2, btext, textlen, cachedelta, flags)
deltainfo = deltacomputer.finddeltainfo(revinfo, fh) deltainfo = deltacomputer.finddeltainfo(revinfo, fh)
# TODO support merging revlogs
unified_revlog_id = 0
marmouteUnsubmitted
Done

what does merging revlog means ?

marmoute: what does merging revlog means ?
# TODO compute rank (for revlog v2)
rank = 0
marmouteUnsubmitted
Done

This is premature and we should not include this in this series.

marmoute: This is premature and we should not include this in this series.
if sidedata:
serialized_sidedata = sidedatautil.serialize_sidedata(sidedata)
sidedata_offset = offset + deltainfo.deltalen
else:
serialized_sidedata = b""
# Don't store the offset if the sidedata is empty, that way
# we can easily detect empty sidedata and they will be no different
# than ones we manually add.
sidedata_offset = 0
e = ( e = (
offset_type(offset, flags), offset_type(offset, flags),
deltainfo.deltalen, deltainfo.deltalen,
textlen, textlen,
deltainfo.base, deltainfo.base,
link, link,
p1r, p1r,
p2r, p2r,
node, node,
0, unified_revlog_id,
0, rank,
0, sidedata_offset,
0, len(serialized_sidedata),
) )
marmouteUnsubmitted
Not Done

Ha, so this is what this is. I don't think we should include them in the format at this point. The only new feature we have are the sidedata and we should focus on that.

marmoute: Ha, so this is what this is. I don't think we should include them in the format at this point.
AlphareAuthorUnsubmitted
Done

Accounting for them in the format was easy enough, I don't think it's worth any more attention by reworking several changesets to remove them just to re-do that work in a few weeks.

Alphare: Accounting for them in the format was easy enough, I don't think it's worth any more attention…
marmouteUnsubmitted
Done

It is unlikley for that work to be done "in a few week" and even if so, we have no garantee the current format fit the need for these data when that work is actually done. So is is really premature.

To me, the important part of this revlogv2 it to setup a more versatile codebase where adding more field/data get simpler. So, not having them part of the initial batch should be fine, as adding these one, or others, later should be simpler.

marmoute: It is unlikley for that work to be done "in a few week" and even if so, we have no garantee the…
if self.version & 0xFFFF != REVLOGV2: if self.version & 0xFFFF != REVLOGV2:
e = e[:8] e = e[:8]
self.index.append(e) self.index.append(e)
entry = self._io.packentry(e, self.node, self.version, curr) entry = self._io.packentry(e, self.node, self.version, curr)
self._writeentry( self._writeentry(
transaction, ifh, dfh, entry, deltainfo.data, link, offset transaction,
ifh,
dfh,
entry,
deltainfo.data,
link,
offset,
serialized_sidedata,
) )
rawtext = btext[0] rawtext = btext[0]
if alwayscache and rawtext is None: if alwayscache and rawtext is None:
rawtext = deltacomputer.buildtext(revinfo, fh) rawtext = deltacomputer.buildtext(revinfo, fh)
if type(rawtext) == bytes: # only accept immutable objects if type(rawtext) == bytes: # only accept immutable objects
self._revisioncache = (node, curr, rawtext) self._revisioncache = (node, curr, rawtext)
self._chainbasecache[curr] = deltainfo.chainbase self._chainbasecache[curr] = deltainfo.chainbase
return curr return curr
def _writeentry(self, transaction, ifh, dfh, entry, data, link, offset): def _writeentry(
self, transaction, ifh, dfh, entry, data, link, offset, sidedata
):
# Files opened in a+ mode have inconsistent behavior on various # Files opened in a+ mode have inconsistent behavior on various
# platforms. Windows requires that a file positioning call be made # platforms. Windows requires that a file positioning call be made
# when the file handle transitions between reads and writes. See # when the file handle transitions between reads and writes. See
# 3686fa2b8eee and the mixedfilemodewrapper in windows.py. On other # 3686fa2b8eee and the mixedfilemodewrapper in windows.py. On other
# platforms, Python or the platform itself can be buggy. Some versions # platforms, Python or the platform itself can be buggy. Some versions
# of Solaris have been observed to not append at the end of the file # of Solaris have been observed to not append at the end of the file
# if the file was seeked to before the end. See issue4943 for more. # if the file was seeked to before the end. See issue4943 for more.
# #
# We work around this issue by inserting a seek() before writing. # We work around this issue by inserting a seek() before writing.
# Note: This is likely not necessary on Python 3. However, because # Note: This is likely not necessary on Python 3. However, because
# the file handle is reused for reads and may be seeked there, we need # the file handle is reused for reads and may be seeked there, we need
# to be careful before changing this. # to be careful before changing this.
ifh.seek(0, os.SEEK_END) ifh.seek(0, os.SEEK_END)
if dfh: if dfh:
dfh.seek(0, os.SEEK_END) dfh.seek(0, os.SEEK_END)
curr = len(self) - 1 curr = len(self) - 1
if not self._inline: if not self._inline:
transaction.add(self.datafile, offset) transaction.add(self.datafile, offset)
transaction.add(self.indexfile, curr * len(entry)) transaction.add(self.indexfile, curr * len(entry))
if data[0]: if data[0]:
dfh.write(data[0]) dfh.write(data[0])
dfh.write(data[1]) dfh.write(data[1])
if sidedata:
dfh.write(sidedata)
ifh.write(entry) ifh.write(entry)
else: else:
offset += curr * self._io.size offset += curr * self._io.size
transaction.add(self.indexfile, offset) transaction.add(self.indexfile, offset)
ifh.write(entry) ifh.write(entry)
ifh.write(data[0]) ifh.write(data[0])
ifh.write(data[1]) ifh.write(data[1])
if sidedata:
ifh.write(sidedata)
self._enforceinlinesize(transaction, ifh) self._enforceinlinesize(transaction, ifh)
nodemaputil.setup_persistent_nodemap(transaction, self) nodemaputil.setup_persistent_nodemap(transaction, self)
def addgroup( def addgroup(
self, self,
deltas, deltas,
linkmapper, linkmapper,
transaction, transaction,

mercurial/revlogutils/flagutil.py

msg = _(b"flag '%#x' undefined in REVIDX_FLAGS_ORDER.") % flag msg = _(b"flag '%#x' undefined in REVIDX_FLAGS_ORDER.") % flag
raise error.ProgrammingError(msg) raise error.ProgrammingError(msg)
if flag in flagprocessors: if flag in flagprocessors:
msg = _(b"cannot register multiple processors on flag '%#x'.") % flag msg = _(b"cannot register multiple processors on flag '%#x'.") % flag
raise error.Abort(msg) raise error.Abort(msg)
flagprocessors[flag] = processor flagprocessors[flag] = processor
def processflagswrite(revlog, text, flags, sidedata): def processflagswrite(revlog, text, flags):
"""Inspect revision data flags and applies write transformations defined """Inspect revision data flags and applies write transformations defined
by registered flag processors. by registered flag processors.
``text`` - the revision data to process ``text`` - the revision data to process
``flags`` - the revision flags ``flags`` - the revision flags
This method processes the flags in the order (or reverse order if This method processes the flags in the order (or reverse order if
``operation`` is 'write') defined by REVIDX_FLAGS_ORDER, applying the ``operation`` is 'write') defined by REVIDX_FLAGS_ORDER, applying the
flag processors registered for present flags. The order of flags defined flag processors registered for present flags. The order of flags defined
in REVIDX_FLAGS_ORDER needs to be stable to allow non-commutativity. in REVIDX_FLAGS_ORDER needs to be stable to allow non-commutativity.
Returns a 2-tuple of ``(text, validatehash)`` where ``text`` is the Returns a 2-tuple of ``(text, validatehash)`` where ``text`` is the
processed text and ``validatehash`` is a bool indicating whether the processed text and ``validatehash`` is a bool indicating whether the
returned text should be checked for hash integrity. returned text should be checked for hash integrity.
""" """
return _processflagsfunc(revlog, text, flags, b'write', sidedata=sidedata)[ return _processflagsfunc(
:2 revlog,
] text,
flags,
b'write',
)[:2]
def processflagsread(revlog, text, flags): def processflagsread(revlog, text, flags):
"""Inspect revision data flags and applies read transformations defined """Inspect revision data flags and applies read transformations defined
by registered flag processors. by registered flag processors.
``text`` - the revision data to process ``text`` - the revision data to process
``flags`` - the revision flags ``flags`` - the revision flags
Returns a 2-tuple of ``(text, validatehash)`` where ``text`` is the Returns a 2-tuple of ``(text, validatehash)`` where ``text`` is the
processed text and ``validatehash`` is a bool indicating whether the processed text and ``validatehash`` is a bool indicating whether the
returned text should be checked for hash integrity. returned text should be checked for hash integrity.
""" """
return _processflagsfunc(revlog, text, flags, b'raw')[1] return _processflagsfunc(revlog, text, flags, b'raw')[1]
def _processflagsfunc(revlog, text, flags, operation, sidedata=None): def _processflagsfunc(revlog, text, flags, operation):
"""internal function to process flag on a revlog """internal function to process flag on a revlog
This function is private to this module, code should never needs to call it This function is private to this module, code should never needs to call it
directly.""" directly."""
# fast path: no flag processors will run # fast path: no flag processors will run
if flags == 0: if flags == 0:
return text, True, {} return text, True
if operation not in (b'read', b'write', b'raw'): if operation not in (b'read', b'write', b'raw'):
raise error.ProgrammingError(_(b"invalid '%s' operation") % operation) raise error.ProgrammingError(_(b"invalid '%s' operation") % operation)
# Check all flags are known. # Check all flags are known.
if flags & ~REVIDX_KNOWN_FLAGS: if flags & ~REVIDX_KNOWN_FLAGS:
raise revlog._flagserrorclass( raise revlog._flagserrorclass(
_(b"incompatible revision flag '%#x'") _(b"incompatible revision flag '%#x'")
% (flags & ~REVIDX_KNOWN_FLAGS) % (flags & ~REVIDX_KNOWN_FLAGS)
) )
validatehash = True validatehash = True
# Depending on the operation (read or write), the order might be # Depending on the operation (read or write), the order might be
# reversed due to non-commutative transforms. # reversed due to non-commutative transforms.
orderedflags = REVIDX_FLAGS_ORDER orderedflags = REVIDX_FLAGS_ORDER
if operation == b'write': if operation == b'write':
orderedflags = reversed(orderedflags) orderedflags = reversed(orderedflags)
outsidedata = {}
for flag in orderedflags: for flag in orderedflags:
# If a flagprocessor has been registered for a known flag, apply the # If a flagprocessor has been registered for a known flag, apply the
# related operation transform and update result tuple. # related operation transform and update result tuple.
if flag & flags: if flag & flags:
vhash = True vhash = True
if flag not in revlog._flagprocessors: if flag not in revlog._flagprocessors:
message = _(b"missing processor for flag '%#x'") % flag message = _(b"missing processor for flag '%#x'") % flag
raise revlog._flagserrorclass(message) raise revlog._flagserrorclass(message)
processor = revlog._flagprocessors[flag] processor = revlog._flagprocessors[flag]
if processor is not None: if processor is not None:
readtransform, writetransform, rawtransform = processor readtransform, writetransform, rawtransform = processor
if operation == b'raw': if operation == b'raw':
vhash = rawtransform(revlog, text) vhash = rawtransform(revlog, text)
elif operation == b'read': elif operation == b'read':
text, vhash, s = readtransform(revlog, text) text, vhash = readtransform(revlog, text)
outsidedata.update(s)
else: # write operation else: # write operation
text, vhash = writetransform(revlog, text, sidedata) text, vhash = writetransform(revlog, text)
validatehash = validatehash and vhash validatehash = validatehash and vhash
return text, validatehash, outsidedata return text, validatehash

mercurial/revlogutils/sidedata.py

# sidedata.py - Logic around store extra data alongside revlog revisions # sidedata.py - Logic around store extra data alongside revlog revisions
# #
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net)
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
"""core code for "sidedata" support """core code for "sidedata" support
The "sidedata" are stored alongside the revision without actually being part of The "sidedata" are stored alongside the revision without actually being part of
its content and not affecting its hash. It's main use cases is to cache its content and not affecting its hash. It's main use cases is to cache
important information related to a changesets. important information related to a changesets.
The current implementation is experimental and subject to changes. Do not rely The current implementation is experimental and subject to changes. Do not rely
on it in production. on it in production.
Sidedata are stored in the revlog itself, within the revision rawtext. They Sidedata are stored in the revlog itself, thanks to a new version of the
are inserted and removed from it using the flagprocessors mechanism. The following revlog. The following format is currently used::
format is currently used::
initial header: initial header:
<number of sidedata; 2 bytes> <number of sidedata; 2 bytes>
sidedata (repeated N times): sidedata (repeated N times):
<sidedata-key; 2 bytes> <sidedata-key; 2 bytes>
<sidedata-entry-length: 4 bytes> <sidedata-entry-length: 4 bytes>
<sidedata-content-sha1-digest: 20 bytes> <sidedata-content-sha1-digest: 20 bytes>
<sidedata-content; X bytes> <sidedata-content; X bytes>
SD_FILESREMOVED = 11 SD_FILESREMOVED = 11
SD_FILES = 12 SD_FILES = 12
# internal format constant # internal format constant
SIDEDATA_HEADER = struct.Struct('>H') SIDEDATA_HEADER = struct.Struct('>H')
SIDEDATA_ENTRY = struct.Struct('>HL20s') SIDEDATA_ENTRY = struct.Struct('>HL20s')
def sidedatawriteprocessor(rl, text, sidedata): def serialize_sidedata(sidedata):
sidedata = list(sidedata.items()) sidedata = list(sidedata.items())
sidedata.sort() sidedata.sort()
rawtext = [SIDEDATA_HEADER.pack(len(sidedata))] buf = [SIDEDATA_HEADER.pack(len(sidedata))]
for key, value in sidedata: for key, value in sidedata:
digest = hashutil.sha1(value).digest() digest = hashutil.sha1(value).digest()
rawtext.append(SIDEDATA_ENTRY.pack(key, len(value), digest)) buf.append(SIDEDATA_ENTRY.pack(key, len(value), digest))
for key, value in sidedata: for key, value in sidedata:
rawtext.append(value) buf.append(value)
rawtext.append(bytes(text)) buf = b''.join(buf)
return b''.join(rawtext), False return buf
def sidedatareadprocessor(rl, text): def deserialize_sidedata(blob):
sidedata = {} sidedata = {}
offset = 0 offset = 0
(nbentry,) = SIDEDATA_HEADER.unpack(text[: SIDEDATA_HEADER.size]) (nbentry,) = SIDEDATA_HEADER.unpack(blob[: SIDEDATA_HEADER.size])
offset += SIDEDATA_HEADER.size offset += SIDEDATA_HEADER.size
dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry) dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry)
for i in range(nbentry): for i in range(nbentry):
nextoffset = offset + SIDEDATA_ENTRY.size nextoffset = offset + SIDEDATA_ENTRY.size
key, size, storeddigest = SIDEDATA_ENTRY.unpack(text[offset:nextoffset]) key, size, storeddigest = SIDEDATA_ENTRY.unpack(blob[offset:nextoffset])
offset = nextoffset offset = nextoffset
# read the data associated with that entry # read the data associated with that entry
nextdataoffset = dataoffset + size nextdataoffset = dataoffset + size
entrytext = text[dataoffset:nextdataoffset] entrytext = bytes(blob[dataoffset:nextdataoffset])
readdigest = hashutil.sha1(entrytext).digest() readdigest = hashutil.sha1(entrytext).digest()
if storeddigest != readdigest: if storeddigest != readdigest:
raise error.SidedataHashError(key, storeddigest, readdigest) raise error.SidedataHashError(key, storeddigest, readdigest)
sidedata[key] = entrytext sidedata[key] = entrytext
dataoffset = nextdataoffset dataoffset = nextdataoffset
text = text[dataoffset:] return sidedata
return text, True, sidedata
def sidedatarawprocessor(rl, text):
# side data modifies rawtext and prevent rawtext hash validation
return False
processors = (
sidedatareadprocessor,
sidedatawriteprocessor,
sidedatarawprocessor,
)

tests/flagprocessorext.py

def validatehash(self, text): def validatehash(self, text):
return True return True
def bypass(self, text): def bypass(self, text):
return False return False
def noopdonothing(self, text, sidedata): def noopdonothing(self, text):
return (text, True) return (text, True)
def noopdonothingread(self, text): def noopdonothingread(self, text):
return (text, True, {}) return (text, True)
def b64encode(self, text, sidedata): def b64encode(self, text):
return (base64.b64encode(text), False) return (base64.b64encode(text), False)
def b64decode(self, text): def b64decode(self, text):
return (base64.b64decode(text), True, {}) return (base64.b64decode(text), True)
def gzipcompress(self, text, sidedata): def gzipcompress(self, text):
return (zlib.compress(text), False) return (zlib.compress(text), False)
def gzipdecompress(self, text): def gzipdecompress(self, text):
return (zlib.decompress(text), True, {}) return (zlib.decompress(text), True)
def supportedoutgoingversions(orig, repo): def supportedoutgoingversions(orig, repo):
versions = orig(repo) versions = orig(repo)
versions.discard(b'01') versions.discard(b'01')
versions.discard(b'02') versions.discard(b'02')
versions.add(b'03') versions.add(b'03')
return versions return versions

tests/simplestorerepo.py

path = b'/'.join([self._storepath, hex(node)]) path = b'/'.join([self._storepath, hex(node)])
rawtext = self._svfs.read(path) rawtext = self._svfs.read(path)
if raw: if raw:
validatehash = flagutil.processflagsraw(self, rawtext, flags) validatehash = flagutil.processflagsraw(self, rawtext, flags)
text = rawtext text = rawtext
else: else:
r = flagutil.processflagsread(self, rawtext, flags) r = flagutil.processflagsread(self, rawtext, flags)
text, validatehash, sidedata = r text, validatehash = r
if validatehash: if validatehash:
self.checkhash(text, node, rev=rev) self.checkhash(text, node, rev=rev)
return text return text
def rawdata(self, nodeorrev): def rawdata(self, nodeorrev):
return self.revision(raw=True) return self.revision(raw=True)

tests/test-copies-in-changeset.t

saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob) saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob)
$ hg changesetcopies $ hg changesetcopies
files: j files: j
#else #else
$ hg ci --amend -m 'copy a to j, v2' $ hg ci --amend -m 'copy a to j, v2'
saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob) saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob)
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
1 sidedata entries 1 sidedata entries (missing-correct-output !)
entry-0014 size 24 entry-0014 size 24 (missing-correct-output !)
'\x00\x00\x00\x02\x00\x00\x00\x00\x01\x00\x00\x00\x00\x06\x00\x00\x00\x02\x00\x00\x00\x00aj' '\x00\x00\x00\x02\x00\x00\x00\x00\x01\x00\x00\x00\x00\x06\x00\x00\x00\x02\x00\x00\x00\x00aj' (missing-correct-output !)
#endif #endif
$ hg showcopies --config experimental.copies.read-from=filelog-only $ hg showcopies --config experimental.copies.read-from=filelog-only
a -> j a -> j (sidedata missing-correct-output !)
a -> j (no-sidedata !)
The entries should be written to extras even if they're empty (so the client The entries should be written to extras even if they're empty (so the client
won't have to fall back to reading from filelogs) won't have to fall back to reading from filelogs)
$ echo x >> j $ echo x >> j
#if extra #if extra
$ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility $ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility
$ hg changesetcopies $ hg changesetcopies
files: j files: j
filesadded: filesadded:
on-disk merge (no conflicts), so we force it to be in-memory on-disk merge (no conflicts), so we force it to be in-memory
with no fallback. with no fallback.
$ hg rebase -d 1 --config rebase.experimental.inmemory=yes --config devel.rebase.force-in-memory-merge=yes $ hg rebase -d 1 --config rebase.experimental.inmemory=yes --config devel.rebase.force-in-memory-merge=yes
rebasing 2:* tip "rename a to b" (glob) rebasing 2:* tip "rename a to b" (glob)
merging a and b to b merging a and b to b
saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/*-*-rebase.hg (glob) saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/*-*-rebase.hg (glob)
$ hg st --change . --copies $ hg st --change . --copies
A b A b
a a (sidedata missing-correct-output !)
a (no-sidedata !)
R a R a
$ cd .. $ cd ..
Test splitting a commit Test splitting a commit
$ hg init split $ hg init split
$ cd split $ cd split
$ echo a > a $ echo a > a

tests/test-revlog-raw.py

# insertion and replacement, and may be interesting to test revlog's line-based # insertion and replacement, and may be interesting to test revlog's line-based
# deltas. # deltas.
_extheader = b'E\n' _extheader = b'E\n'
def readprocessor(self, rawtext): def readprocessor(self, rawtext):
# True: the returned text could be used to verify hash # True: the returned text could be used to verify hash
text = rawtext[len(_extheader) :].replace(b'i', b'1') text = rawtext[len(_extheader) :].replace(b'i', b'1')
return text, True, {} return text, True
def writeprocessor(self, text, sidedata): def writeprocessor(self, text):
# False: the returned rawtext shouldn't be used to verify hash # False: the returned rawtext shouldn't be used to verify hash
rawtext = _extheader + text.replace(b'1', b'i') rawtext = _extheader + text.replace(b'1', b'i')
return rawtext, False return rawtext, False
def rawprocessor(self, rawtext): def rawprocessor(self, rawtext):
# False: do not verify hash. Only the content returned by "readprocessor" # False: do not verify hash. Only the content returned by "readprocessor"
# can be used to verify hash. # can be used to verify hash.
if isempty: if isempty:
text = b'' text = b''
else: else:
text = gentext(i) text = gentext(i)
rev = appendrev(rlog, text, tr, isext=isext, isdelta=isdelta) rev = appendrev(rlog, text, tr, isext=isext, isdelta=isdelta)
# Verify text, rawtext, and rawsize # Verify text, rawtext, and rawsize
if isext: if isext:
rawtext = writeprocessor(None, text, {})[0] rawtext = writeprocessor(None, text)[0]
else: else:
rawtext = text rawtext = text
if rlog.rawsize(rev) != len(rawtext): if rlog.rawsize(rev) != len(rawtext):
abort('rev %d: wrong rawsize' % rev) abort('rev %d: wrong rawsize' % rev)
if rlog.revision(rev, raw=False) != text: if rlog.revision(rev, raw=False) != text:
abort('rev %d: wrong text' % rev) abort('rev %d: wrong text' % rev)
if rlog.rawdata(rev) != rawtext: if rlog.rawdata(rev) != rawtext:
abort('rev %d: wrong rawtext' % rev) abort('rev %d: wrong rawtext' % rev)

tests/testlib/ext-sidedata.py

# text length # text length
sd[sidedata.SD_TEST1] = struct.pack('>I', len(text)) sd[sidedata.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes # and sha2 hashes
sha256 = hashlib.sha256(text).digest() sha256 = hashlib.sha256(text).digest()
sd[sidedata.SD_TEST2] = struct.pack('>32s', sha256) sd[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
return orig(self, text, transaction, link, p1, p2, *args, **kwargs) return orig(self, text, transaction, link, p1, p2, *args, **kwargs)
def wraprevision(orig, self, nodeorrev, *args, **kwargs): def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs):
text = orig(self, nodeorrev, *args, **kwargs) text, sd = orig(self, nodeorrev, *args, **kwargs)
if getattr(self, 'sidedatanocheck', False): if getattr(self, 'sidedatanocheck', False):
return text return text, sd
if self.version & 0xFFFF != 2:
return text, sd
if nodeorrev != nullrev and nodeorrev != nullid: if nodeorrev != nullrev and nodeorrev != nullid:
sd = self.sidedata(nodeorrev)
if len(text) != struct.unpack('>I', sd[sidedata.SD_TEST1])[0]: if len(text) != struct.unpack('>I', sd[sidedata.SD_TEST1])[0]:
raise RuntimeError('text size mismatch') raise RuntimeError('text size mismatch')
expected = sd[sidedata.SD_TEST2] expected = sd[sidedata.SD_TEST2]
got = hashlib.sha256(text).digest() got = hashlib.sha256(text).digest()
if got != expected: if got != expected:
raise RuntimeError('sha256 mismatch') raise RuntimeError('sha256 mismatch')
return text return text, sd
def wrapgetsidedatacompanion(orig, srcrepo, dstrepo): def wrapgetsidedatacompanion(orig, srcrepo, dstrepo):
sidedatacompanion = orig(srcrepo, dstrepo) sidedatacompanion = orig(srcrepo, dstrepo)
addedreqs = dstrepo.requirements - srcrepo.requirements addedreqs = dstrepo.requirements - srcrepo.requirements
if requirements.SIDEDATA_REQUIREMENT in addedreqs: if requirements.SIDEDATA_REQUIREMENT in addedreqs:
assert sidedatacompanion is None # deal with composition later assert sidedatacompanion is None # deal with composition later
update[sidedata.SD_TEST2] = struct.pack('>32s', sha256) update[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
return False, (), update, 0, 0 return False, (), update, 0, 0
return sidedatacompanion return sidedatacompanion
def extsetup(ui): def extsetup(ui):
extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision) extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision)
extensions.wrapfunction(revlog.revlog, 'revision', wraprevision) extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata)
extensions.wrapfunction( extensions.wrapfunction(
upgrade_engine, 'getsidedatacompanion', wrapgetsidedatacompanion upgrade_engine, 'getsidedatacompanion', wrapgetsidedatacompanion
) )