This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10787

revlog: store sidedata in their own file
ClosedPublic

Authored by marmoute on May 28 2021, 6:56 PM.

Details

Reviewers
indygreg
Alphare
Group Reviewers
hg-reviewers
Commits
rHGe6292eb33384: revlog: store sidedata in their own file
Summary

This makes sidedata manipulation simpler and results in more compact data when
traversing either data or sidedata.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.May 28 2021, 6:56 PM
marmoute updated this revision to Diff 28256.May 30 2021, 1:29 PM
marmoute updated this revision to Diff 28278.May 30 2021, 10:07 PM
baymax updated this revision to Diff 28300.May 31 2021, 6:38 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare requested changes to this revision.May 31 2021, 10:26 AM
Alphare added a subscriber: Alphare.
Alphare added inline comments.
mercurial/revlog.py
930

Agreed, but for now let's put a small comment explaining why the dance is needed, it wasn't immediately obvious to me when I was writing the code initially.

This revision now requires changes to proceed.May 31 2021, 10:26 AM
baymax updated this revision to Diff 28329.Jun 1 2021, 5:41 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 28397.Jun 1 2021, 11:10 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare accepted this revision as: Alphare.Jun 2 2021, 4:23 AM

There are some typos (even in the commit title), so if you can fix them, that's be nice, otherwise this looks good.

marmoute added a comment.Jun 2 2021, 7:02 AM
In D10787#165466, @Alphare wrote:

There are some typos (even in the commit title), so if you can fix them, that's be nice, otherwise this looks good.

Indeed, I am sending a new version that should be better.

baymax retitled this revision from revlog: store sidedata in there own file to revlog: store sidedata in their own file.Jun 4 2021, 9:54 AM
baymax edited the summary of this revision. (Show Details)
baymax updated this revision to Diff 28446.

✅ refresh by Heptapod after a successful CI run (🐙 💚)

marmoute added a commit: rHGe6292eb33384: revlog: store sidedata in their own file.Jun 7 2021, 5:23 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGe6292eb33384: revlog: store sidedata in their own file (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 28490

mercurial/configitems.py

# - for rollback operation # - for rollback operation
# * proper streaming (race free) of the docket file # * proper streaming (race free) of the docket file
# * track garbage data to evemtually allow rewriting -existing- sidedata. # * track garbage data to evemtually allow rewriting -existing- sidedata.
# * Exchange-wise, we will also need to do something more efficient than # * Exchange-wise, we will also need to do something more efficient than
# keeping references to the affected revlogs, especially memory-wise when # keeping references to the affected revlogs, especially memory-wise when
# rewriting sidedata. # rewriting sidedata.
# * introduce a proper solution to reduce the number of filelog related files. # * introduce a proper solution to reduce the number of filelog related files.
# * use caching for reading sidedata (similar to what we do for data). # * use caching for reading sidedata (similar to what we do for data).
# * no longer set offset=0 if sidedata_size=0 (simplify cutoff computation).
# * Improvement to consider # * Improvement to consider
# - avoid compression header in chunk using the default compression? # - avoid compression header in chunk using the default compression?
# - forbid "inline" compression mode entirely? # - forbid "inline" compression mode entirely?
# - split the data offset and flag field (the 2 bytes save are mostly trouble) # - split the data offset and flag field (the 2 bytes save are mostly trouble)
# - keep track of uncompressed -chunk- size (to preallocate memory better) # - keep track of uncompressed -chunk- size (to preallocate memory better)
# - keep track of chain base or size (probably not that useful anymore) # - keep track of chain base or size (probably not that useful anymore)
# - store data and sidedata in different files
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'revlogv2', b'revlogv2',
default=None, default=None,
) )
coreconfigitem( coreconfigitem(
b'experimental', b'experimental',
b'revisions.disambiguatewithin', b'revisions.disambiguatewithin',

mercurial/revlog.py

# revlog.py - storage back-end for mercurial # revlog.py - storage back-end for mercurial
# coding: utf8
# #
# Copyright 2005-2007 Olivia Mackall <olivia@selenic.com> # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
"""Storage back-end for Mercurial. """Storage back-end for Mercurial.
# corresponds to uncompressed length of indexformatng (2 gigs, 4-byte # corresponds to uncompressed length of indexformatng (2 gigs, 4-byte
# signed integer) # signed integer)
_maxentrysize = 0x7FFFFFFF _maxentrysize = 0x7FFFFFFF
PARTIAL_READ_MSG = _( PARTIAL_READ_MSG = _(
b'partial read of revlog %s; expected %d bytes from offset %d, got %d' b'partial read of revlog %s; expected %d bytes from offset %d, got %d'
) )
FILE_TOO_SHORT_MSG = _(
b'cannot read from revlog %s;'
b' expected %d bytes from offset %d, data size is %d'
)
class revlog(object): class revlog(object):
""" """
the underlying revision storage object the underlying revision storage object
A revlog consists of two parts, an index and the revision data. A revlog consists of two parts, an index and the revision data.
The index is a file with a fixed record size containing The index is a file with a fixed record size containing
""" """
self.upperboundcomp = upperboundcomp self.upperboundcomp = upperboundcomp
self.radix = radix self.radix = radix
self._docket_file = None self._docket_file = None
self._indexfile = None self._indexfile = None
self._datafile = None self._datafile = None
self._sidedatafile = None
self._nodemap_file = None self._nodemap_file = None
self.postfix = postfix self.postfix = postfix
self._trypending = trypending self._trypending = trypending
self.opener = opener self.opener = opener
if persistentnodemap: if persistentnodemap:
self._nodemap_file = nodemaputil.get_nodemap_file(self) self._nodemap_file = nodemaputil.get_nodemap_file(self)
assert target[0] in ALL_KINDS assert target[0] in ALL_KINDS
self.hassidedata = False self.hassidedata = False
self._srdensitythreshold = 0.50 self._srdensitythreshold = 0.50
self._srmingapsize = 262144 self._srmingapsize = 262144
# Make copy of flag processors so each revlog instance can support # Make copy of flag processors so each revlog instance can support
# custom flags. # custom flags.
self._flagprocessors = dict(flagutil.flagprocessors) self._flagprocessors = dict(flagutil.flagprocessors)
# 2-tuple of file handles being used for active writing. # 3-tuple of file handles being used for active writing.
self._writinghandles = None self._writinghandles = None
# prevent nesting of addgroup # prevent nesting of addgroup
self._adding_group = None self._adding_group = None
self._loadindex() self._loadindex()
self._concurrencychecker = concurrencychecker self._concurrencychecker = concurrencychecker
# generaldelta implied by version 2 revlogs. # generaldelta implied by version 2 revlogs.
self._generaldelta = True self._generaldelta = True
# the logic for persistent nodemap will be dealt with within the # the logic for persistent nodemap will be dealt with within the
# main docket, so disable it for now. # main docket, so disable it for now.
self._nodemap_file = None self._nodemap_file = None
if self._docket is not None: if self._docket is not None:
self._datafile = self._docket.data_filepath() self._datafile = self._docket.data_filepath()
self._sidedatafile = self._docket.sidedata_filepath()
elif self.postfix is None: elif self.postfix is None:
self._datafile = b'%s.d' % self.radix self._datafile = b'%s.d' % self.radix
else: else:
self._datafile = b'%s.d.%s' % (self.radix, self.postfix) self._datafile = b'%s.d.%s' % (self.radix, self.postfix)
self.nodeconstants = sha1nodeconstants self.nodeconstants = sha1nodeconstants
self.nullid = self.nodeconstants.nullid self.nullid = self.nodeconstants.nullid
else: else:
if self._inline: if self._inline:
func = self._indexfp func = self._indexfp
else: else:
func = self._datafp func = self._datafp
with func() as fp: with func() as fp:
yield fp yield fp
@contextlib.contextmanager
def _sidedatareadfp(self): def _sidedatareadfp(self):
"""file object suitable to read sidedata""" """file object suitable to read sidedata"""
return self._datareadfp() if self._writinghandles:
yield self._writinghandles[2]
else:
with self.opener(self._sidedatafile) as fp:
yield fp
def tiprev(self): def tiprev(self):
return len(self.index) - 1 return len(self.index) - 1
def tip(self): def tip(self):
return self.node(self.tiprev()) return self.node(self.tiprev())
def __contains__(self, rev): def __contains__(self, rev):
# Accessors for index entries. # Accessors for index entries.
# First tuple entry is 8 bytes. First 6 bytes are offset. Last 2 bytes # First tuple entry is 8 bytes. First 6 bytes are offset. Last 2 bytes
# are flags. # are flags.
def start(self, rev): def start(self, rev):
return int(self.index[rev][0] >> 16) return int(self.index[rev][0] >> 16)
def sidedata_cut_off(self, rev):
sd_cut_off = self.index[rev][8]
if sd_cut_off != 0:
return sd_cut_off
# This is some annoying dance, because entries without sidedata
# currently use 0 as their ofsset. (instead of previous-offset +
AlphareUnsubmitted
Not Done

Agreed, but for now let's put a small comment explaining why the dance is needed, it wasn't immediately obvious to me when I was writing the code initially.

Alphare: Agreed, but for now let's put a small comment explaining why the dance is needed, it wasn't…
# previous-size)
#
# We should reconsider this sidedata → 0 sidata_offset policy.
# In the meantime, we need this.
while 0 <= rev:
e = self.index[rev]
if e[9] != 0:
return e[8] + e[9]
rev -= 1
return 0
def flags(self, rev): def flags(self, rev):
return self.index[rev][0] & 0xFFFF return self.index[rev][0] & 0xFFFF
def length(self, rev): def length(self, rev):
return self.index[rev][1] return self.index[rev][1]
def sidedata_length(self, rev): def sidedata_length(self, rev):
if not self.hassidedata: if not self.hassidedata:
if self._inline: if self._inline:
sidedata_offset += self.index.entry_size * (1 + rev) sidedata_offset += self.index.entry_size * (1 + rev)
if sidedata_size == 0: if sidedata_size == 0:
return {} return {}
# XXX this need caching, as we do for data # XXX this need caching, as we do for data
with self._sidedatareadfp() as sdf: with self._sidedatareadfp() as sdf:
sdf.seek(sidedata_offset) if self._docket.sidedata_end < sidedata_offset + sidedata_size:
filename = self._sidedatafile
end = self._docket.sidedata_end
offset = sidedata_offset
length = sidedata_size
m = FILE_TOO_SHORT_MSG % (filename, length, offset, end)
raise error.RevlogError(m)
sdf.seek(sidedata_offset, os.SEEK_SET)
comp_segment = sdf.read(sidedata_size) comp_segment = sdf.read(sidedata_size)
if len(comp_segment) < sidedata_size: if len(comp_segment) < sidedata_size:
filename = self._datafile filename = self._sidedatafile
length = sidedata_size length = sidedata_size
offset = sidedata_offset offset = sidedata_offset
got = len(comp_segment) got = len(comp_segment)
m = PARTIAL_READ_MSG % (filename, length, offset, got) m = PARTIAL_READ_MSG % (filename, length, offset, got)
raise error.RevlogError(m) raise error.RevlogError(m)
comp = self.index[rev][11] comp = self.index[rev][11]
if comp == COMP_MODE_PLAIN: if comp == COMP_MODE_PLAIN:
tr.replace(self._indexfile, trindex * self.index.entry_size) tr.replace(self._indexfile, trindex * self.index.entry_size)
nodemaputil.setup_persistent_nodemap(tr, self) nodemaputil.setup_persistent_nodemap(tr, self)
self._chunkclear() self._chunkclear()
if existing_handles: if existing_handles:
# switched from inline to conventional reopen the index # switched from inline to conventional reopen the index
ifh = self.__index_write_fp() ifh = self.__index_write_fp()
self._writinghandles = (ifh, new_dfh) self._writinghandles = (ifh, new_dfh, None)
new_dfh = None new_dfh = None
finally: finally:
if new_dfh is not None: if new_dfh is not None:
new_dfh.close() new_dfh.close()
def _nodeduplicatecallback(self, transaction, node): def _nodeduplicatecallback(self, transaction, node):
"""called when trying to add a node already stored.""" """called when trying to add a node already stored."""
@contextlib.contextmanager @contextlib.contextmanager
def _writing(self, transaction): def _writing(self, transaction):
if self._trypending: if self._trypending:
msg = b'try to write in a `trypending` revlog: %s' msg = b'try to write in a `trypending` revlog: %s'
msg %= self.display_id msg %= self.display_id
raise error.ProgrammingError(msg) raise error.ProgrammingError(msg)
if self._writinghandles is not None: if self._writinghandles is not None:
yield yield
else: else:
ifh = dfh = None ifh = dfh = sdfh = None
try: try:
r = len(self) r = len(self)
# opening the data file. # opening the data file.
dsize = 0 dsize = 0
if r: if r:
dsize = self.end(r - 1) dsize = self.end(r - 1)
dfh = None dfh = None
if not self._inline: if not self._inline:
try: try:
dfh = self._datafp(b"r+") dfh = self._datafp(b"r+")
if self._docket is None: if self._docket is None:
dfh.seek(0, os.SEEK_END) dfh.seek(0, os.SEEK_END)
else: else:
dfh.seek(self._docket.data_end, os.SEEK_SET) dfh.seek(self._docket.data_end, os.SEEK_SET)
except IOError as inst: except IOError as inst:
if inst.errno != errno.ENOENT: if inst.errno != errno.ENOENT:
raise raise
dfh = self._datafp(b"w+") dfh = self._datafp(b"w+")
transaction.add(self._datafile, dsize) transaction.add(self._datafile, dsize)
if self._sidedatafile is not None:
try:
sdfh = self.opener(self._sidedatafile, mode=b"r+")
dfh.seek(self._docket.sidedata_end, os.SEEK_SET)
except IOError as inst:
if inst.errno != errno.ENOENT:
raise
sdfh = self.opener(self._sidedatafile, mode=b"w+")
transaction.add(
self._sidedatafile, self._docket.sidedata_end
)
# opening the index file. # opening the index file.
isize = r * self.index.entry_size isize = r * self.index.entry_size
ifh = self.__index_write_fp() ifh = self.__index_write_fp()
if self._inline: if self._inline:
transaction.add(self._indexfile, dsize + isize) transaction.add(self._indexfile, dsize + isize)
else: else:
transaction.add(self._indexfile, isize) transaction.add(self._indexfile, isize)
# exposing all file handle for writing. # exposing all file handle for writing.
self._writinghandles = (ifh, dfh) self._writinghandles = (ifh, dfh, sdfh)
yield yield
if self._docket is not None: if self._docket is not None:
self._write_docket(transaction) self._write_docket(transaction)
finally: finally:
self._writinghandles = None self._writinghandles = None
if dfh is not None: if dfh is not None:
dfh.close() dfh.close()
if sdfh is not None:
dfh.close()
# closing the index file last to avoid exposing referent to # closing the index file last to avoid exposing referent to
# potential unflushed data content. # potential unflushed data content.
if ifh is not None: if ifh is not None:
ifh.close() ifh.close()
def _write_docket(self, transaction): def _write_docket(self, transaction):
"""write the current docket on disk """write the current docket on disk
btext = [rawtext] btext = [rawtext]
curr = len(self) curr = len(self)
prev = curr - 1 prev = curr - 1
offset = self._get_data_offset(prev) offset = self._get_data_offset(prev)
if self._concurrencychecker: if self._concurrencychecker:
ifh, dfh = self._writinghandles ifh, dfh, sdfh = self._writinghandles
# XXX no checking for the sidedata file
if self._inline: if self._inline:
# offset is "as if" it were in the .d file, so we need to add on # offset is "as if" it were in the .d file, so we need to add on
# the size of the entry metadata. # the size of the entry metadata.
self._concurrencychecker( self._concurrencychecker(
ifh, self._indexfile, offset + curr * self.index.entry_size ifh, self._indexfile, offset + curr * self.index.entry_size
) )
else: else:
# Entries in the .i are a consistent size. # Entries in the .i are a consistent size.
h = b'' h = b''
compression_mode = COMP_MODE_PLAIN compression_mode = COMP_MODE_PLAIN
deltainfo = deltautil.drop_u_compression(deltainfo) deltainfo = deltautil.drop_u_compression(deltainfo)
sidedata_compression_mode = COMP_MODE_INLINE sidedata_compression_mode = COMP_MODE_INLINE
if sidedata and self.hassidedata: if sidedata and self.hassidedata:
sidedata_compression_mode = COMP_MODE_PLAIN sidedata_compression_mode = COMP_MODE_PLAIN
serialized_sidedata = sidedatautil.serialize_sidedata(sidedata) serialized_sidedata = sidedatautil.serialize_sidedata(sidedata)
sidedata_offset = offset + deltainfo.deltalen sidedata_offset = self._docket.sidedata_end
h, comp_sidedata = self.compress(serialized_sidedata) h, comp_sidedata = self.compress(serialized_sidedata)
if ( if (
h != b'u' h != b'u'
and comp_sidedata[0:1] != b'\0' and comp_sidedata[0:1] != b'\0'
and len(comp_sidedata) < len(serialized_sidedata) and len(comp_sidedata) < len(serialized_sidedata)
): ):
assert not h assert not h
if ( if (
entry = header + entry entry = header + entry
self._writeentry( self._writeentry(
transaction, transaction,
entry, entry,
deltainfo.data, deltainfo.data,
link, link,
offset, offset,
serialized_sidedata, serialized_sidedata,
sidedata_offset,
) )
rawtext = btext[0] rawtext = btext[0]
if alwayscache and rawtext is None: if alwayscache and rawtext is None:
rawtext = deltacomputer.buildtext(revinfo, fh) rawtext = deltacomputer.buildtext(revinfo, fh)
if type(rawtext) == bytes: # only accept immutable objects if type(rawtext) == bytes: # only accept immutable objects
to `n - 1`'s sidedata being written after `n`'s data. to `n - 1`'s sidedata being written after `n`'s data.
TODO cache this in a docket file before getting out of experimental.""" TODO cache this in a docket file before getting out of experimental."""
if self._docket is None: if self._docket is None:
return self.end(prev) return self.end(prev)
else: else:
return self._docket.data_end return self._docket.data_end
def _writeentry(self, transaction, entry, data, link, offset, sidedata): def _writeentry(
self, transaction, entry, data, link, offset, sidedata, sidedata_offset
):
# Files opened in a+ mode have inconsistent behavior on various # Files opened in a+ mode have inconsistent behavior on various
# platforms. Windows requires that a file positioning call be made # platforms. Windows requires that a file positioning call be made
# when the file handle transitions between reads and writes. See # when the file handle transitions between reads and writes. See
# 3686fa2b8eee and the mixedfilemodewrapper in windows.py. On other # 3686fa2b8eee and the mixedfilemodewrapper in windows.py. On other
# platforms, Python or the platform itself can be buggy. Some versions # platforms, Python or the platform itself can be buggy. Some versions
# of Solaris have been observed to not append at the end of the file # of Solaris have been observed to not append at the end of the file
# if the file was seeked to before the end. See issue4943 for more. # if the file was seeked to before the end. See issue4943 for more.
# #
# We work around this issue by inserting a seek() before writing. # We work around this issue by inserting a seek() before writing.
# Note: This is likely not necessary on Python 3. However, because # Note: This is likely not necessary on Python 3. However, because
# the file handle is reused for reads and may be seeked there, we need # the file handle is reused for reads and may be seeked there, we need
# to be careful before changing this. # to be careful before changing this.
if self._writinghandles is None: if self._writinghandles is None:
msg = b'adding revision outside `revlog._writing` context' msg = b'adding revision outside `revlog._writing` context'
raise error.ProgrammingError(msg) raise error.ProgrammingError(msg)
ifh, dfh = self._writinghandles ifh, dfh, sdfh = self._writinghandles
if self._docket is None: if self._docket is None:
ifh.seek(0, os.SEEK_END) ifh.seek(0, os.SEEK_END)
else: else:
ifh.seek(self._docket.index_end, os.SEEK_SET) ifh.seek(self._docket.index_end, os.SEEK_SET)
if dfh: if dfh:
if self._docket is None: if self._docket is None:
dfh.seek(0, os.SEEK_END) dfh.seek(0, os.SEEK_END)
else: else:
dfh.seek(self._docket.data_end, os.SEEK_SET) dfh.seek(self._docket.data_end, os.SEEK_SET)
if sdfh:
sdfh.seek(self._docket.sidedata_end, os.SEEK_SET)
curr = len(self) - 1 curr = len(self) - 1
if not self._inline: if not self._inline:
transaction.add(self._datafile, offset) transaction.add(self._datafile, offset)
if self._sidedatafile:
transaction.add(self._sidedatafile, sidedata_offset)
transaction.add(self._indexfile, curr * len(entry)) transaction.add(self._indexfile, curr * len(entry))
if data[0]: if data[0]:
dfh.write(data[0]) dfh.write(data[0])
dfh.write(data[1]) dfh.write(data[1])
if sidedata: if sidedata:
dfh.write(sidedata) sdfh.write(sidedata)
ifh.write(entry) ifh.write(entry)
else: else:
offset += curr * self.index.entry_size offset += curr * self.index.entry_size
transaction.add(self._indexfile, offset) transaction.add(self._indexfile, offset)
ifh.write(entry) ifh.write(entry)
ifh.write(data[0]) ifh.write(data[0])
ifh.write(data[1]) ifh.write(data[1])
if sidedata: assert not sidedata
ifh.write(sidedata)
self._enforceinlinesize(transaction) self._enforceinlinesize(transaction)
if self._docket is not None: if self._docket is not None:
self._docket.index_end = self._writinghandles[0].tell() self._docket.index_end = self._writinghandles[0].tell()
self._docket.data_end = self._writinghandles[1].tell() self._docket.data_end = self._writinghandles[1].tell()
self._docket.sidedata_end = self._writinghandles[2].tell()
nodemaputil.setup_persistent_nodemap(transaction, self) nodemaputil.setup_persistent_nodemap(transaction, self)
def addgroup( def addgroup(
self, self,
deltas, deltas,
linkmapper, linkmapper,
transaction, transaction,
# first truncate the files on disk # first truncate the files on disk
data_end = self.start(rev) data_end = self.start(rev)
if not self._inline: if not self._inline:
transaction.add(self._datafile, data_end) transaction.add(self._datafile, data_end)
end = rev * self.index.entry_size end = rev * self.index.entry_size
else: else:
end = data_end + (rev * self.index.entry_size) end = data_end + (rev * self.index.entry_size)
if self._sidedatafile:
sidedata_end = self.sidedata_cut_off(rev)
transaction.add(self._sidedatafile, sidedata_end)
transaction.add(self._indexfile, end) transaction.add(self._indexfile, end)
if self._docket is not None: if self._docket is not None:
# XXX we could, leverage the docket while stripping. However it is # XXX we could, leverage the docket while stripping. However it is
# not powerfull enough at the time of this comment # not powerfull enough at the time of this comment
self._docket.index_end = end self._docket.index_end = end
self._docket.data_end = data_end self._docket.data_end = data_end
self._docket.sidedata_end = sidedata_end
self._docket.write(transaction, stripping=True) self._docket.write(transaction, stripping=True)
# then reset internal state in memory to forget those revisions # then reset internal state in memory to forget those revisions
self._revisioncache = None self._revisioncache = None
self._chaininfocache = util.lrucachedict(500) self._chaininfocache = util.lrucachedict(500)
self._chunkclear() self._chunkclear()
del self.index[rev:-1] del self.index[rev:-1]
assert not self._inline assert not self._inline
if not helpers[1] and not helpers[2]: if not helpers[1] and not helpers[2]:
# Nothing to generate or remove # Nothing to generate or remove
return return
new_entries = [] new_entries = []
# append the new sidedata # append the new sidedata
with self._writing(transaction): with self._writing(transaction):
ifh, dfh = self._writinghandles ifh, dfh, sdfh = self._writinghandles
if self._docket is not None: dfh.seek(self._docket.sidedata_end, os.SEEK_SET)
dfh.seek(self._docket.data_end, os.SEEK_SET)
else:
dfh.seek(0, os.SEEK_END)
current_offset = dfh.tell() current_offset = sdfh.tell()
for rev in range(startrev, endrev + 1): for rev in range(startrev, endrev + 1):
entry = self.index[rev] entry = self.index[rev]
new_sidedata, flags = sidedatautil.run_sidedata_helpers( new_sidedata, flags = sidedatautil.run_sidedata_helpers(
store=self, store=self,
sidedata_helpers=helpers, sidedata_helpers=helpers,
sidedata={}, sidedata={},
rev=rev, rev=rev,
) )
entry_update = ( entry_update = (
current_offset, current_offset,
len(serialized_sidedata), len(serialized_sidedata),
new_offset_flags, new_offset_flags,
sidedata_compression_mode, sidedata_compression_mode,
) )
# the sidedata computation might have move the file cursors around # the sidedata computation might have move the file cursors around
dfh.seek(current_offset, os.SEEK_SET) sdfh.seek(current_offset, os.SEEK_SET)
dfh.write(serialized_sidedata) sdfh.write(serialized_sidedata)
new_entries.append(entry_update) new_entries.append(entry_update)
current_offset += len(serialized_sidedata) current_offset += len(serialized_sidedata)
if self._docket is not None: self._docket.sidedata_end = sdfh.tell()
self._docket.data_end = dfh.tell()
# rewrite the new index entries # rewrite the new index entries
ifh.seek(startrev * self.index.entry_size) ifh.seek(startrev * self.index.entry_size)
for i, e in enumerate(new_entries): for i, e in enumerate(new_entries):
rev = startrev + i rev = startrev + i
self.index.replace_sidedata_info(rev, *e) self.index.replace_sidedata_info(rev, *e)
packed = self.index.entry_binary(rev) packed = self.index.entry_binary(rev)
if rev == 0 and self._docket is None: if rev == 0 and self._docket is None:
header = self._format_flags | self._format_version header = self._format_flags | self._format_version
header = self.index.pack_header(header) header = self.index.pack_header(header)
packed = header + packed packed = header + packed
ifh.write(packed) ifh.write(packed)

mercurial/revlogutils/docket.py

# Docket format # Docket format
# #
# * 4 bytes: revlog version # * 4 bytes: revlog version
# | This is mandatory as docket must be compatible with the previous # | This is mandatory as docket must be compatible with the previous
# | revlog index header. # | revlog index header.
# * 1 bytes: size of index uuid # * 1 bytes: size of index uuid
# * 1 bytes: size of data uuid # * 1 bytes: size of data uuid
# * 1 bytes: size of sizedata uuid
# * 8 bytes: size of index-data # * 8 bytes: size of index-data
# * 8 bytes: pending size of index-data # * 8 bytes: pending size of index-data
# * 8 bytes: size of data # * 8 bytes: size of data
# * 8 bytes: size of sidedata
# * 8 bytes: pending size of data # * 8 bytes: pending size of data
# * 8 bytes: pending size of sidedata
# * 1 bytes: default compression header # * 1 bytes: default compression header
S_HEADER = struct.Struct(constants.INDEX_HEADER_FMT + b'BBLLLLc') S_HEADER = struct.Struct(constants.INDEX_HEADER_FMT + b'BBBLLLLLLc')
class RevlogDocket(object): class RevlogDocket(object):
"""metadata associated with revlog""" """metadata associated with revlog"""
def __init__( def __init__(
self, self,
revlog, revlog,
use_pending=False, use_pending=False,
version_header=None, version_header=None,
index_uuid=None, index_uuid=None,
data_uuid=None, data_uuid=None,
sidedata_uuid=None,
index_end=0, index_end=0,
pending_index_end=0, pending_index_end=0,
data_end=0, data_end=0,
pending_data_end=0, pending_data_end=0,
sidedata_end=0,
pending_sidedata_end=0,
default_compression_header=None, default_compression_header=None,
): ):
self._version_header = version_header self._version_header = version_header
self._read_only = bool(use_pending) self._read_only = bool(use_pending)
self._dirty = False self._dirty = False
self._radix = revlog.radix self._radix = revlog.radix
self._path = revlog._docket_file self._path = revlog._docket_file
self._opener = revlog.opener self._opener = revlog.opener
self._index_uuid = index_uuid self._index_uuid = index_uuid
self._data_uuid = data_uuid self._data_uuid = data_uuid
self._sidedata_uuid = sidedata_uuid
# thes asserts should be True as long as we have a single index filename # thes asserts should be True as long as we have a single index filename
assert index_end <= pending_index_end assert index_end <= pending_index_end
assert data_end <= pending_data_end assert data_end <= pending_data_end
assert sidedata_end <= pending_sidedata_end
self._initial_index_end = index_end self._initial_index_end = index_end
self._pending_index_end = pending_index_end self._pending_index_end = pending_index_end
self._initial_data_end = data_end self._initial_data_end = data_end
self._pending_data_end = pending_data_end self._pending_data_end = pending_data_end
self._initial_sidedata_end = sidedata_end
self._pending_sidedata_end = pending_sidedata_end
if use_pending: if use_pending:
self._index_end = self._pending_index_end self._index_end = self._pending_index_end
self._data_end = self._pending_data_end self._data_end = self._pending_data_end
self._sidedata_end = self._pending_sidedata_end
else: else:
self._index_end = self._initial_index_end self._index_end = self._initial_index_end
self._data_end = self._initial_data_end self._data_end = self._initial_data_end
self._sidedata_end = self._initial_sidedata_end
self.default_compression_header = default_compression_header self.default_compression_header = default_compression_header
def index_filepath(self): def index_filepath(self):
"""file path to the current index file associated to this docket""" """file path to the current index file associated to this docket"""
# very simplistic version at first # very simplistic version at first
if self._index_uuid is None: if self._index_uuid is None:
self._index_uuid = make_uid() self._index_uuid = make_uid()
return b"%s-%s.idx" % (self._radix, self._index_uuid) return b"%s-%s.idx" % (self._radix, self._index_uuid)
def data_filepath(self): def data_filepath(self):
"""file path to the current data file associated to this docket""" """file path to the current data file associated to this docket"""
# very simplistic version at first # very simplistic version at first
if self._data_uuid is None: if self._data_uuid is None:
self._data_uuid = make_uid() self._data_uuid = make_uid()
return b"%s-%s.dat" % (self._radix, self._data_uuid) return b"%s-%s.dat" % (self._radix, self._data_uuid)
def sidedata_filepath(self):
"""file path to the current sidedata file associated to this docket"""
# very simplistic version at first
if self._sidedata_uuid is None:
self._sidedata_uuid = make_uid()
return b"%s-%s.sda" % (self._radix, self._sidedata_uuid)
@property @property
def index_end(self): def index_end(self):
return self._index_end return self._index_end
@index_end.setter @index_end.setter
def index_end(self, new_size): def index_end(self, new_size):
if new_size != self._index_end: if new_size != self._index_end:
self._index_end = new_size self._index_end = new_size
self._dirty = True self._dirty = True
@property @property
def data_end(self): def data_end(self):
return self._data_end return self._data_end
@data_end.setter @data_end.setter
def data_end(self, new_size): def data_end(self, new_size):
if new_size != self._data_end: if new_size != self._data_end:
self._data_end = new_size self._data_end = new_size
self._dirty = True self._dirty = True
@property
def sidedata_end(self):
return self._sidedata_end
@sidedata_end.setter
def sidedata_end(self, new_size):
if new_size != self._sidedata_end:
self._sidedata_end = new_size
self._dirty = True
def write(self, transaction, pending=False, stripping=False): def write(self, transaction, pending=False, stripping=False):
"""write the modification of disk if any """write the modification of disk if any
This make the new content visible to all process""" This make the new content visible to all process"""
if not self._dirty: if not self._dirty:
return False return False
else: else:
if self._read_only: if self._read_only:
# if pending we still need to the write final data eventually # if pending we still need to the write final data eventually
self._dirty = pending self._dirty = pending
return True return True
def _serialize(self, pending=False): def _serialize(self, pending=False):
if pending: if pending:
official_index_end = self._initial_index_end official_index_end = self._initial_index_end
official_data_end = self._initial_data_end official_data_end = self._initial_data_end
official_sidedata_end = self._initial_sidedata_end
else: else:
official_index_end = self._index_end official_index_end = self._index_end
official_data_end = self._data_end official_data_end = self._data_end
official_sidedata_end = self._sidedata_end
# this assert should be True as long as we have a single index filename # this assert should be True as long as we have a single index filename
assert official_data_end <= self._data_end assert official_data_end <= self._data_end
assert official_sidedata_end <= self._sidedata_end
data = ( data = (
self._version_header, self._version_header,
len(self._index_uuid), len(self._index_uuid),
len(self._data_uuid), len(self._data_uuid),
len(self._sidedata_uuid),
official_index_end, official_index_end,
self._index_end, self._index_end,
official_data_end, official_data_end,
self._data_end, self._data_end,
official_sidedata_end,
self._sidedata_end,
self.default_compression_header, self.default_compression_header,
) )
s = [] s = []
s.append(S_HEADER.pack(*data)) s.append(S_HEADER.pack(*data))
s.append(self._index_uuid) s.append(self._index_uuid)
s.append(self._data_uuid) s.append(self._data_uuid)
s.append(self._sidedata_uuid)
return b''.join(s) return b''.join(s)
def default_docket(revlog, version_header): def default_docket(revlog, version_header):
"""given a revlog version a new docket object for the given revlog""" """given a revlog version a new docket object for the given revlog"""
rl_version = version_header & 0xFFFF rl_version = version_header & 0xFFFF
if rl_version not in (constants.REVLOGV2, constants.CHANGELOGV2): if rl_version not in (constants.REVLOGV2, constants.CHANGELOGV2):
return None return None
version_header = next(iheader) version_header = next(iheader)
index_uuid_size = next(iheader) index_uuid_size = next(iheader)
index_uuid = get_data(index_uuid_size) index_uuid = get_data(index_uuid_size)
data_uuid_size = next(iheader) data_uuid_size = next(iheader)
data_uuid = get_data(data_uuid_size) data_uuid = get_data(data_uuid_size)
sidedata_uuid_size = next(iheader)
sidedata_uuid = get_data(sidedata_uuid_size)
index_size = next(iheader) index_size = next(iheader)
pending_index_size = next(iheader) pending_index_size = next(iheader)
data_size = next(iheader) data_size = next(iheader)
pending_data_size = next(iheader) pending_data_size = next(iheader)
sidedata_size = next(iheader)
pending_sidedata_size = next(iheader)
default_compression_header = next(iheader) default_compression_header = next(iheader)
docket = RevlogDocket( docket = RevlogDocket(
revlog, revlog,
use_pending=use_pending, use_pending=use_pending,
version_header=version_header, version_header=version_header,
index_uuid=index_uuid, index_uuid=index_uuid,
data_uuid=data_uuid, data_uuid=data_uuid,
sidedata_uuid=sidedata_uuid,
index_end=index_size, index_end=index_size,
pending_index_end=pending_index_size, pending_index_end=pending_index_size,
data_end=data_size, data_end=data_size,
pending_data_end=pending_data_size, pending_data_end=pending_data_size,
sidedata_end=sidedata_size,
pending_sidedata_end=pending_sidedata_size,
default_compression_header=default_compression_header, default_compression_header=default_compression_header,
) )
return docket return docket

mercurial/store.py

REVLOG_FILES_MAIN_EXT = (b'.i', b'i.tmpcensored') REVLOG_FILES_MAIN_EXT = (b'.i', b'i.tmpcensored')
REVLOG_FILES_OTHER_EXT = ( REVLOG_FILES_OTHER_EXT = (
b'.idx', b'.idx',
b'.d', b'.d',
b'.dat', b'.dat',
b'.n', b'.n',
b'.nd', b'.nd',
b'.sda',
b'd.tmpcensored', b'd.tmpcensored',
) )
# files that are "volatile" and might change between listing and streaming # files that are "volatile" and might change between listing and streaming
# #
# note: the ".nd" file are nodemap data and won't "change" but they might be # note: the ".nd" file are nodemap data and won't "change" but they might be
# deleted. # deleted.
REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd') REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd')

tests/test-revlog-v2.t

-------------------------------- --------------------------------
We should have have: We should have have:
- a docket - a docket
- a index file with a unique name - a index file with a unique name
- a data file - a data file
$ ls .hg/store/00changelog* .hg/store/00manifest* $ ls .hg/store/00changelog* .hg/store/00manifest*
.hg/store/00changelog-6b8ab34b.dat .hg/store/00changelog-1335303a.sda
.hg/store/00changelog-88698448.idx .hg/store/00changelog-6b8ab34b.idx
.hg/store/00changelog-b875dfc5.dat
.hg/store/00changelog.i .hg/store/00changelog.i
.hg/store/00manifest-1335303a.dat .hg/store/00manifest-05a21d65.idx
.hg/store/00manifest-b875dfc5.idx .hg/store/00manifest-43c37dde.dat
.hg/store/00manifest-e2c9362a.sda
.hg/store/00manifest.i .hg/store/00manifest.i