mercurial/revlog.py
930	Agreed, but for now let's put a small comment explaining why the dance is needed, it wasn't immediately obvious to me when I was writing the code initially.

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	28247		May 28 2021, 6:56 PM	★	★
Diff 2	28256		May 30 2021, 1:29 PM	★	★
Diff 3	28278		May 30 2021, 10:07 PM	★	★
Diff 4	28300		May 31 2021, 6:38 AM	★	★
Diff 5	28329		Jun 1 2021, 5:40 AM	★	★
Diff 6	28397		Jun 1 2021, 11:10 PM	★	★
Diff 7	28446		Jun 4 2021, 9:54 AM	★	★
Diff 8	28490	rHGe6292eb3338438df982c7976bd2071e58037f6b8	May 28 2021, 5:41 PM	★	★

Commit	Parents	Author	Summary	Date
a8dd11622d26	97d3eab4d449	Pierre-Yves David		May 28 2021, 5:41 PM

Status	Author	Revision
Closed	marmoute	D10800 revlog: allow to pass an existing docket to `_loadindex()`
Closed	marmoute	D10799 revlog: add a ways to blacklist some revision when searching for a delta
Closed	marmoute	D10798 util: add `nb_bytes` argument to `copyfile` to partially copy a file
Closed	marmoute	D10797 revlog: move `revisioninfo` in `revlogutils`
Closed	marmoute	D10796 revlog: use the `entry` function in bundlerepo
Closed	marmoute	D10795 revlog: use entry in revlogv0.py
Closed	marmoute	D10794 revlog: use the new `entry` function in revlog.py
Closed	marmoute	D10793 revlog: add a function to build index entry tuple
Closed	marmoute	D10792 revlog: move `offset_type` to `revlogutils`
Closed	marmoute	D10791 revlog: move entry documentation alongside new related constants
Closed	marmoute	D10790 censor: drop size limitation on the tombstone
Closed	marmoute	D10789 revlog: move censoring code in a dedicated module
Closed	marmoute	D10788 censor: reduce risk of censor test blowing up output limit
Closed	marmoute	D10787 revlog: store sidedata in their own file
Closed	marmoute	D10786 revlog: fix docket.date_filepath docstring
Closed	marmoute	D10785 revlog: use dedicated code for reading sidedata
Closed	marmoute	D10784 revlog: simplify "partial read" error message
Closed	marmoute	D10783 revlog: no longer return sidedata from `_revisiondata`
Closed	marmoute	D10782 revlog: use `self.sidedata` directly in `revlog.clone`
Closed	marmoute	D10781 revlog: use `self.sidedata` directly to construct changelogrevision
Closed	marmoute	D10780 revlog: implement sidedata without using _revisiondata
Closed	marmoute	D10779 revlogv2: drop "sidedatacompression" from the list of todos
Closed	marmoute	D10831 index: use an explicit constant for INDEX_HEADER format and use it for docket

Diff 28397

mercurial/configitems.py

	# - for rollback operation			# - for rollback operation
	# * proper streaming (race free) of the docket file			# * proper streaming (race free) of the docket file
	# * track garbage data to evemtually allow rewriting -existing- sidedata.			# * track garbage data to evemtually allow rewriting -existing- sidedata.
	# * Exchange-wise, we will also need to do something more efficient than			# * Exchange-wise, we will also need to do something more efficient than
	# keeping references to the affected revlogs, especially memory-wise when			# keeping references to the affected revlogs, especially memory-wise when
	# rewriting sidedata.			# rewriting sidedata.
	# * introduce a proper solution to reduce the number of filelog related files.			# * introduce a proper solution to reduce the number of filelog related files.
	# * use caching for reading sidedata (similar to what we do for data).			# * use caching for reading sidedata (similar to what we do for data).
				# * no longer set offset=0 if sidedata_size=0 (simplify cutoff computation).
	# * Improvement to consider			# * Improvement to consider
	# - avoid compression header in chunk using the default compression?			# - avoid compression header in chunk using the default compression?
	# - forbid "inline" compression mode entirely?			# - forbid "inline" compression mode entirely?
	# - split the data offset and flag field (the 2 bytes save are mostly trouble)			# - split the data offset and flag field (the 2 bytes save are mostly trouble)
	# - keep track of uncompressed -chunk- size (to preallocate memory better)			# - keep track of uncompressed -chunk- size (to preallocate memory better)
	# - keep track of chain base or size (probably not that useful anymore)			# - keep track of chain base or size (probably not that useful anymore)
	# - store data and sidedata in different files
	coreconfigitem(			coreconfigitem(
	b'experimental',			b'experimental',
	b'revlogv2',			b'revlogv2',
	default=None,			default=None,
	)			)
	coreconfigitem(			coreconfigitem(
	b'experimental',			b'experimental',
	b'revisions.disambiguatewithin',			b'revisions.disambiguatewithin',

mercurial/revlog.py

	# revlog.py - storage back-end for mercurial			# revlog.py - storage back-end for mercurial
				# coding: utf8
	#			#
	# Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>			# Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
	#			#
	# This software may be used and distributed according to the terms of the			# This software may be used and distributed according to the terms of the
	# GNU General Public License version 2 or any later version.			# GNU General Public License version 2 or any later version.

	"""Storage back-end for Mercurial.			"""Storage back-end for Mercurial.

	# corresponds to uncompressed length of indexformatng (2 gigs, 4-byte			# corresponds to uncompressed length of indexformatng (2 gigs, 4-byte
	# signed integer)			# signed integer)
	_maxentrysize = 0x7FFFFFFF			_maxentrysize = 0x7FFFFFFF

	PARTIAL_READ_MSG = _(			PARTIAL_READ_MSG = _(
	b'partial read of revlog %s; expected %d bytes from offset %d, got %d'			b'partial read of revlog %s; expected %d bytes from offset %d, got %d'
	)			)

				FILE_TOO_SHORT_MSG = _(
				b'cannot read from revlog %s;'
				b' expected %d bytes from offset %d, data size is %d'
				)


	class revlog(object):			class revlog(object):
	"""			"""
	the underlying revision storage object			the underlying revision storage object

	A revlog consists of two parts, an index and the revision data.			A revlog consists of two parts, an index and the revision data.

	The index is a file with a fixed record size containing			The index is a file with a fixed record size containing
	"""			"""
	self.upperboundcomp = upperboundcomp			self.upperboundcomp = upperboundcomp

	self.radix = radix			self.radix = radix

	self._docket_file = None			self._docket_file = None
	self._indexfile = None			self._indexfile = None
	self._datafile = None			self._datafile = None
				self._sidedatafile = None
	self._nodemap_file = None			self._nodemap_file = None
	self.postfix = postfix			self.postfix = postfix
	self._trypending = trypending			self._trypending = trypending
	self.opener = opener			self.opener = opener
	if persistentnodemap:			if persistentnodemap:
	self._nodemap_file = nodemaputil.get_nodemap_file(self)			self._nodemap_file = nodemaputil.get_nodemap_file(self)

	assert target[0] in ALL_KINDS			assert target[0] in ALL_KINDS
	self.hassidedata = False			self.hassidedata = False
	self._srdensitythreshold = 0.50			self._srdensitythreshold = 0.50
	self._srmingapsize = 262144			self._srmingapsize = 262144

	# Make copy of flag processors so each revlog instance can support			# Make copy of flag processors so each revlog instance can support
	# custom flags.			# custom flags.
	self._flagprocessors = dict(flagutil.flagprocessors)			self._flagprocessors = dict(flagutil.flagprocessors)

	# 2-tuple of file handles being used for active writing.			# 3-tuple of file handles being used for active writing.
	self._writinghandles = None			self._writinghandles = None
	# prevent nesting of addgroup			# prevent nesting of addgroup
	self._adding_group = None			self._adding_group = None

	self._loadindex()			self._loadindex()

	self._concurrencychecker = concurrencychecker			self._concurrencychecker = concurrencychecker

	# generaldelta implied by version 2 revlogs.			# generaldelta implied by version 2 revlogs.
	self._generaldelta = True			self._generaldelta = True
	# the logic for persistent nodemap will be dealt with within the			# the logic for persistent nodemap will be dealt with within the
	# main docket, so disable it for now.			# main docket, so disable it for now.
	self._nodemap_file = None			self._nodemap_file = None

	if self._docket is not None:			if self._docket is not None:
	self._datafile = self._docket.data_filepath()			self._datafile = self._docket.data_filepath()
				self._sidedatafile = self._docket.sidedata_filepath()
	elif self.postfix is None:			elif self.postfix is None:
	self._datafile = b'%s.d' % self.radix			self._datafile = b'%s.d' % self.radix
	else:			else:
	self._datafile = b'%s.d.%s' % (self.radix, self.postfix)			self._datafile = b'%s.d.%s' % (self.radix, self.postfix)

	self.nodeconstants = sha1nodeconstants			self.nodeconstants = sha1nodeconstants
	self.nullid = self.nodeconstants.nullid			self.nullid = self.nodeconstants.nullid

	else:			else:
	if self._inline:			if self._inline:
	func = self._indexfp			func = self._indexfp
	else:			else:
	func = self._datafp			func = self._datafp
	with func() as fp:			with func() as fp:
	yield fp			yield fp

				@contextlib.contextmanager
	def _sidedatareadfp(self):			def _sidedatareadfp(self):
	"""file object suitable to read sidedata"""			"""file object suitable to read sidedata"""
	return self._datareadfp()			if self._writinghandles:
				yield self._writinghandles[2]
				else:
				with self.opener(self._sidedatafile) as fp:
				yield fp

	def tiprev(self):			def tiprev(self):
	return len(self.index) - 1			return len(self.index) - 1

	def tip(self):			def tip(self):
	return self.node(self.tiprev())			return self.node(self.tiprev())

	def __contains__(self, rev):			def __contains__(self, rev):

	# Accessors for index entries.			# Accessors for index entries.

	# First tuple entry is 8 bytes. First 6 bytes are offset. Last 2 bytes			# First tuple entry is 8 bytes. First 6 bytes are offset. Last 2 bytes
	# are flags.			# are flags.
	def start(self, rev):			def start(self, rev):
	return int(self.index[rev][0] >> 16)			return int(self.index[rev][0] >> 16)

				def sidedata_cut_off(self, rev):
				sd_cut_off = self.index[rev][8]
				if sd_cut_off != 0:
				return sd_cut_off
				# This is some annoying dance, because entry without sidedata currently
				# use 0 as their ofsset. (instead of previous-offset + previous-size)
				AlphareUnsubmitted Not Done Agreed, but for now let's put a small comment explaining why the dance is needed, it wasn't immediately obvious to me when I was writing the code initially. Alphare: Agreed, but for now let's put a small comment explaining why the dance is needed, it wasn't…
				#
				# We should reconsider actually reconsider this sidedata → 0
				# sidata_offset policy. In the meantime, we need this.
				while 0 <= rev:
				e = self.index[rev]
				if e[9] != 0:
				return e[8] + e[9]
				rev -= 1
				return 0

	def flags(self, rev):			def flags(self, rev):
	return self.index[rev][0] & 0xFFFF			return self.index[rev][0] & 0xFFFF

	def length(self, rev):			def length(self, rev):
	return self.index[rev][1]			return self.index[rev][1]

	def sidedata_length(self, rev):			def sidedata_length(self, rev):
	if not self.hassidedata:			if not self.hassidedata:

	if self._inline:			if self._inline:
	sidedata_offset += self.index.entry_size * (1 + rev)			sidedata_offset += self.index.entry_size * (1 + rev)
	if sidedata_size == 0:			if sidedata_size == 0:
	return {}			return {}

	# XXX this need caching, as we do for data			# XXX this need caching, as we do for data
	with self._sidedatareadfp() as sdf:			with self._sidedatareadfp() as sdf:
	sdf.seek(sidedata_offset)			if self._docket.sidedata_end < sidedata_offset + sidedata_size:
				filename = self._sidedatafile
				end = self._docket.sidedata_end
				offset = sidedata_offset
				length = sidedata_size
				m = FILE_TOO_SHORT_MSG % (filename, length, offset, end)
				raise error.RevlogError(m)

				sdf.seek(sidedata_offset, os.SEEK_SET)
	comp_segment = sdf.read(sidedata_size)			comp_segment = sdf.read(sidedata_size)

	if len(comp_segment) < sidedata_size:			if len(comp_segment) < sidedata_size:
	filename = self._datafile			filename = self._sidedatafile
	length = sidedata_size			length = sidedata_size
	offset = sidedata_offset			offset = sidedata_offset
	got = len(comp_segment)			got = len(comp_segment)
	m = PARTIAL_READ_MSG % (filename, length, offset, got)			m = PARTIAL_READ_MSG % (filename, length, offset, got)
	raise error.RevlogError(m)			raise error.RevlogError(m)

	comp = self.index[rev][11]			comp = self.index[rev][11]
	if comp == COMP_MODE_PLAIN:			if comp == COMP_MODE_PLAIN:

	tr.replace(self._indexfile, trindex * self.index.entry_size)			tr.replace(self._indexfile, trindex * self.index.entry_size)
	nodemaputil.setup_persistent_nodemap(tr, self)			nodemaputil.setup_persistent_nodemap(tr, self)
	self._chunkclear()			self._chunkclear()

	if existing_handles:			if existing_handles:
	# switched from inline to conventional reopen the index			# switched from inline to conventional reopen the index
	ifh = self.__index_write_fp()			ifh = self.__index_write_fp()
	self._writinghandles = (ifh, new_dfh)			self._writinghandles = (ifh, new_dfh, None)
	new_dfh = None			new_dfh = None
	finally:			finally:
	if new_dfh is not None:			if new_dfh is not None:
	new_dfh.close()			new_dfh.close()

	def _nodeduplicatecallback(self, transaction, node):			def _nodeduplicatecallback(self, transaction, node):
	"""called when trying to add a node already stored."""			"""called when trying to add a node already stored."""

	@contextlib.contextmanager			@contextlib.contextmanager
	def _writing(self, transaction):			def _writing(self, transaction):
	if self._trypending:			if self._trypending:
	msg = b'try to write in a `trypending` revlog: %s'			msg = b'try to write in a `trypending` revlog: %s'
	msg %= self.display_id			msg %= self.display_id
	raise error.ProgrammingError(msg)			raise error.ProgrammingError(msg)
	if self._writinghandles is not None:			if self._writinghandles is not None:
	yield			yield
	else:			else:
	ifh = dfh = None			ifh = dfh = sdfh = None
	try:			try:
	r = len(self)			r = len(self)
	# opening the data file.			# opening the data file.
	dsize = 0			dsize = 0
	if r:			if r:
	dsize = self.end(r - 1)			dsize = self.end(r - 1)
	dfh = None			dfh = None
	if not self._inline:			if not self._inline:
	try:			try:
	dfh = self._datafp(b"r+")			dfh = self._datafp(b"r+")
	if self._docket is None:			if self._docket is None:
	dfh.seek(0, os.SEEK_END)			dfh.seek(0, os.SEEK_END)
	else:			else:
	dfh.seek(self._docket.data_end, os.SEEK_SET)			dfh.seek(self._docket.data_end, os.SEEK_SET)
	except IOError as inst:			except IOError as inst:
	if inst.errno != errno.ENOENT:			if inst.errno != errno.ENOENT:
	raise			raise
	dfh = self._datafp(b"w+")			dfh = self._datafp(b"w+")
	transaction.add(self._datafile, dsize)			transaction.add(self._datafile, dsize)
				if self._sidedatafile is not None:
				try:
				sdfh = self.opener(self._sidedatafile, mode=b"r+")
				dfh.seek(self._docket.sidedata_end, os.SEEK_SET)
				except IOError as inst:
				if inst.errno != errno.ENOENT:
				raise
				sdfh = self.opener(self._sidedatafile, mode=b"w+")
				transaction.add(
				self._sidedatafile, self._docket.sidedata_end
				)

	# opening the index file.			# opening the index file.
	isize = r * self.index.entry_size			isize = r * self.index.entry_size
	ifh = self.__index_write_fp()			ifh = self.__index_write_fp()
	if self._inline:			if self._inline:
	transaction.add(self._indexfile, dsize + isize)			transaction.add(self._indexfile, dsize + isize)
	else:			else:
	transaction.add(self._indexfile, isize)			transaction.add(self._indexfile, isize)
	# exposing all file handle for writing.			# exposing all file handle for writing.
	self._writinghandles = (ifh, dfh)			self._writinghandles = (ifh, dfh, sdfh)
	yield			yield
	if self._docket is not None:			if self._docket is not None:
	self._write_docket(transaction)			self._write_docket(transaction)
	finally:			finally:
	self._writinghandles = None			self._writinghandles = None
	if dfh is not None:			if dfh is not None:
	dfh.close()			dfh.close()
				if sdfh is not None:
				dfh.close()
	# closing the index file last to avoid exposing referent to			# closing the index file last to avoid exposing referent to
	# potential unflushed data content.			# potential unflushed data content.
	if ifh is not None:			if ifh is not None:
	ifh.close()			ifh.close()

	def _write_docket(self, transaction):			def _write_docket(self, transaction):
	"""write the current docket on disk			"""write the current docket on disk

	btext = [rawtext]			btext = [rawtext]

	curr = len(self)			curr = len(self)
	prev = curr - 1			prev = curr - 1

	offset = self._get_data_offset(prev)			offset = self._get_data_offset(prev)

	if self._concurrencychecker:			if self._concurrencychecker:
	ifh, dfh = self._writinghandles			ifh, dfh, sdfh = self._writinghandles
				# XXX no checking for the sidedata file
	if self._inline:			if self._inline:
	# offset is "as if" it were in the .d file, so we need to add on			# offset is "as if" it were in the .d file, so we need to add on
	# the size of the entry metadata.			# the size of the entry metadata.
	self._concurrencychecker(			self._concurrencychecker(
	ifh, self._indexfile, offset + curr * self.index.entry_size			ifh, self._indexfile, offset + curr * self.index.entry_size
	)			)
	else:			else:
	# Entries in the .i are a consistent size.			# Entries in the .i are a consistent size.
	h = b''			h = b''
	compression_mode = COMP_MODE_PLAIN			compression_mode = COMP_MODE_PLAIN
	deltainfo = deltautil.drop_u_compression(deltainfo)			deltainfo = deltautil.drop_u_compression(deltainfo)

	sidedata_compression_mode = COMP_MODE_INLINE			sidedata_compression_mode = COMP_MODE_INLINE
	if sidedata and self.hassidedata:			if sidedata and self.hassidedata:
	sidedata_compression_mode = COMP_MODE_PLAIN			sidedata_compression_mode = COMP_MODE_PLAIN
	serialized_sidedata = sidedatautil.serialize_sidedata(sidedata)			serialized_sidedata = sidedatautil.serialize_sidedata(sidedata)
	sidedata_offset = offset + deltainfo.deltalen			sidedata_offset = self._docket.sidedata_end
	h, comp_sidedata = self.compress(serialized_sidedata)			h, comp_sidedata = self.compress(serialized_sidedata)
	if (			if (
	h != b'u'			h != b'u'
	and comp_sidedata[0:1] != b'\0'			and comp_sidedata[0:1] != b'\0'
	and len(comp_sidedata) < len(serialized_sidedata)			and len(comp_sidedata) < len(serialized_sidedata)
	):			):
	assert not h			assert not h
	if (			if (
	entry = header + entry			entry = header + entry
	self._writeentry(			self._writeentry(
	transaction,			transaction,
	entry,			entry,
	deltainfo.data,			deltainfo.data,
	link,			link,
	offset,			offset,
	serialized_sidedata,			serialized_sidedata,
				sidedata_offset,
	)			)

	rawtext = btext[0]			rawtext = btext[0]

	if alwayscache and rawtext is None:			if alwayscache and rawtext is None:
	rawtext = deltacomputer.buildtext(revinfo, fh)			rawtext = deltacomputer.buildtext(revinfo, fh)

	if type(rawtext) == bytes: # only accept immutable objects			if type(rawtext) == bytes: # only accept immutable objects
	to `n - 1`'s sidedata being written after `n`'s data.			to `n - 1`'s sidedata being written after `n`'s data.

	TODO cache this in a docket file before getting out of experimental."""			TODO cache this in a docket file before getting out of experimental."""
	if self._docket is None:			if self._docket is None:
	return self.end(prev)			return self.end(prev)
	else:			else:
	return self._docket.data_end			return self._docket.data_end

	def _writeentry(self, transaction, entry, data, link, offset, sidedata):			def _writeentry(
				self, transaction, entry, data, link, offset, sidedata, sidedata_offset
				):
	# Files opened in a+ mode have inconsistent behavior on various			# Files opened in a+ mode have inconsistent behavior on various
	# platforms. Windows requires that a file positioning call be made			# platforms. Windows requires that a file positioning call be made
	# when the file handle transitions between reads and writes. See			# when the file handle transitions between reads and writes. See
	# 3686fa2b8eee and the mixedfilemodewrapper in windows.py. On other			# 3686fa2b8eee and the mixedfilemodewrapper in windows.py. On other
	# platforms, Python or the platform itself can be buggy. Some versions			# platforms, Python or the platform itself can be buggy. Some versions
	# of Solaris have been observed to not append at the end of the file			# of Solaris have been observed to not append at the end of the file
	# if the file was seeked to before the end. See issue4943 for more.			# if the file was seeked to before the end. See issue4943 for more.
	#			#
	# We work around this issue by inserting a seek() before writing.			# We work around this issue by inserting a seek() before writing.
	# Note: This is likely not necessary on Python 3. However, because			# Note: This is likely not necessary on Python 3. However, because
	# the file handle is reused for reads and may be seeked there, we need			# the file handle is reused for reads and may be seeked there, we need
	# to be careful before changing this.			# to be careful before changing this.
	if self._writinghandles is None:			if self._writinghandles is None:
	msg = b'adding revision outside `revlog._writing` context'			msg = b'adding revision outside `revlog._writing` context'
	raise error.ProgrammingError(msg)			raise error.ProgrammingError(msg)
	ifh, dfh = self._writinghandles			ifh, dfh, sdfh = self._writinghandles
	if self._docket is None:			if self._docket is None:
	ifh.seek(0, os.SEEK_END)			ifh.seek(0, os.SEEK_END)
	else:			else:
	ifh.seek(self._docket.index_end, os.SEEK_SET)			ifh.seek(self._docket.index_end, os.SEEK_SET)
	if dfh:			if dfh:
	if self._docket is None:			if self._docket is None:
	dfh.seek(0, os.SEEK_END)			dfh.seek(0, os.SEEK_END)
	else:			else:
	dfh.seek(self._docket.data_end, os.SEEK_SET)			dfh.seek(self._docket.data_end, os.SEEK_SET)
				if sdfh:
				sdfh.seek(self._docket.sidedata_end, os.SEEK_SET)

	curr = len(self) - 1			curr = len(self) - 1
	if not self._inline:			if not self._inline:
	transaction.add(self._datafile, offset)			transaction.add(self._datafile, offset)
				if self._sidedatafile:
				transaction.add(self._sidedatafile, sidedata_offset)
	transaction.add(self._indexfile, curr * len(entry))			transaction.add(self._indexfile, curr * len(entry))
	if data[0]:			if data[0]:
	dfh.write(data[0])			dfh.write(data[0])
	dfh.write(data[1])			dfh.write(data[1])
	if sidedata:			if sidedata:
	dfh.write(sidedata)			sdfh.write(sidedata)
	ifh.write(entry)			ifh.write(entry)
	else:			else:
	offset += curr * self.index.entry_size			offset += curr * self.index.entry_size
	transaction.add(self._indexfile, offset)			transaction.add(self._indexfile, offset)
	ifh.write(entry)			ifh.write(entry)
	ifh.write(data[0])			ifh.write(data[0])
	ifh.write(data[1])			ifh.write(data[1])
	if sidedata:			assert not sidedata
	ifh.write(sidedata)
	self._enforceinlinesize(transaction)			self._enforceinlinesize(transaction)
	if self._docket is not None:			if self._docket is not None:
	self._docket.index_end = self._writinghandles[0].tell()			self._docket.index_end = self._writinghandles[0].tell()
	self._docket.data_end = self._writinghandles[1].tell()			self._docket.data_end = self._writinghandles[1].tell()
				self._docket.sidedata_end = self._writinghandles[2].tell()

	nodemaputil.setup_persistent_nodemap(transaction, self)			nodemaputil.setup_persistent_nodemap(transaction, self)

	def addgroup(			def addgroup(
	self,			self,
	deltas,			deltas,
	linkmapper,			linkmapper,
	transaction,			transaction,
	# first truncate the files on disk			# first truncate the files on disk
	data_end = self.start(rev)			data_end = self.start(rev)
	if not self._inline:			if not self._inline:
	transaction.add(self._datafile, data_end)			transaction.add(self._datafile, data_end)
	end = rev * self.index.entry_size			end = rev * self.index.entry_size
	else:			else:
	end = data_end + (rev * self.index.entry_size)			end = data_end + (rev * self.index.entry_size)

				if self._sidedatafile:
				sidedata_end = self.sidedata_cut_off(rev)
				transaction.add(self._sidedatafile, sidedata_end)

	transaction.add(self._indexfile, end)			transaction.add(self._indexfile, end)
	if self._docket is not None:			if self._docket is not None:
	# XXX we could, leverage the docket while stripping. However it is			# XXX we could, leverage the docket while stripping. However it is
	# not powerfull enough at the time of this comment			# not powerfull enough at the time of this comment
	self._docket.index_end = end			self._docket.index_end = end
	self._docket.data_end = data_end			self._docket.data_end = data_end
				self._docket.sidedata_end = sidedata_end
	self._docket.write(transaction, stripping=True)			self._docket.write(transaction, stripping=True)

	# then reset internal state in memory to forget those revisions			# then reset internal state in memory to forget those revisions
	self._revisioncache = None			self._revisioncache = None
	self._chaininfocache = util.lrucachedict(500)			self._chaininfocache = util.lrucachedict(500)
	self._chunkclear()			self._chunkclear()

	del self.index[rev:-1]			del self.index[rev:-1]
	assert not self._inline			assert not self._inline
	if not helpers[1] and not helpers[2]:			if not helpers[1] and not helpers[2]:
	# Nothing to generate or remove			# Nothing to generate or remove
	return			return

	new_entries = []			new_entries = []
	# append the new sidedata			# append the new sidedata
	with self._writing(transaction):			with self._writing(transaction):
	ifh, dfh = self._writinghandles			ifh, dfh, sdfh = self._writinghandles
	if self._docket is not None:			dfh.seek(self._docket.sidedata_end, os.SEEK_SET)
	dfh.seek(self._docket.data_end, os.SEEK_SET)
	else:
	dfh.seek(0, os.SEEK_END)

	current_offset = dfh.tell()			current_offset = sdfh.tell()
	for rev in range(startrev, endrev + 1):			for rev in range(startrev, endrev + 1):
	entry = self.index[rev]			entry = self.index[rev]
	new_sidedata, flags = sidedatautil.run_sidedata_helpers(			new_sidedata, flags = sidedatautil.run_sidedata_helpers(
	store=self,			store=self,
	sidedata_helpers=helpers,			sidedata_helpers=helpers,
	sidedata={},			sidedata={},
	rev=rev,			rev=rev,
	)			)
	entry_update = (			entry_update = (
	current_offset,			current_offset,
	len(serialized_sidedata),			len(serialized_sidedata),
	new_offset_flags,			new_offset_flags,
	sidedata_compression_mode,			sidedata_compression_mode,
	)			)

	# the sidedata computation might have move the file cursors around			# the sidedata computation might have move the file cursors around
	dfh.seek(current_offset, os.SEEK_SET)			sdfh.seek(current_offset, os.SEEK_SET)
	dfh.write(serialized_sidedata)			sdfh.write(serialized_sidedata)
	new_entries.append(entry_update)			new_entries.append(entry_update)
	current_offset += len(serialized_sidedata)			current_offset += len(serialized_sidedata)
	if self._docket is not None:			self._docket.sidedata_end = sdfh.tell()
	self._docket.data_end = dfh.tell()

	# rewrite the new index entries			# rewrite the new index entries
	ifh.seek(startrev * self.index.entry_size)			ifh.seek(startrev * self.index.entry_size)
	for i, e in enumerate(new_entries):			for i, e in enumerate(new_entries):
	rev = startrev + i			rev = startrev + i
	self.index.replace_sidedata_info(rev, *e)			self.index.replace_sidedata_info(rev, *e)
	packed = self.index.entry_binary(rev)			packed = self.index.entry_binary(rev)
	if rev == 0 and self._docket is None:			if rev == 0 and self._docket is None:
	header = self._format_flags \| self._format_version			header = self._format_flags \| self._format_version
	header = self.index.pack_header(header)			header = self.index.pack_header(header)
	packed = header + packed			packed = header + packed
	ifh.write(packed)			ifh.write(packed)

mercurial/revlogutils/docket.py


	# Docket format			# Docket format
	#			#
	# * 4 bytes: revlog version			# * 4 bytes: revlog version
	# \| This is mandatory as docket must be compatible with the previous			# \| This is mandatory as docket must be compatible with the previous
	# \| revlog index header.			# \| revlog index header.
	# * 1 bytes: size of index uuid			# * 1 bytes: size of index uuid
	# * 1 bytes: size of data uuid			# * 1 bytes: size of data uuid
				# * 1 bytes: size of sizedata uuid
	# * 8 bytes: size of index-data			# * 8 bytes: size of index-data
	# * 8 bytes: pending size of index-data			# * 8 bytes: pending size of index-data
	# * 8 bytes: size of data			# * 8 bytes: size of data
				# * 8 bytes: size of sidedata
	# * 8 bytes: pending size of data			# * 8 bytes: pending size of data
				# * 8 bytes: pending size of sidedata
	# * 1 bytes: default compression header			# * 1 bytes: default compression header
	S_HEADER = struct.Struct(constants.INDEX_HEADER_FMT + b'BBLLLLc')			S_HEADER = struct.Struct(constants.INDEX_HEADER_FMT + b'BBBLLLLLLc')


	class RevlogDocket(object):			class RevlogDocket(object):
	"""metadata associated with revlog"""			"""metadata associated with revlog"""

	def __init__(			def __init__(
	self,			self,
	revlog,			revlog,
	use_pending=False,			use_pending=False,
	version_header=None,			version_header=None,
	index_uuid=None,			index_uuid=None,
	data_uuid=None,			data_uuid=None,
				sidedata_uuid=None,
	index_end=0,			index_end=0,
	pending_index_end=0,			pending_index_end=0,
	data_end=0,			data_end=0,
	pending_data_end=0,			pending_data_end=0,
				sidedata_end=0,
				pending_sidedata_end=0,
	default_compression_header=None,			default_compression_header=None,
	):			):
	self._version_header = version_header			self._version_header = version_header
	self._read_only = bool(use_pending)			self._read_only = bool(use_pending)
	self._dirty = False			self._dirty = False
	self._radix = revlog.radix			self._radix = revlog.radix
	self._path = revlog._docket_file			self._path = revlog._docket_file
	self._opener = revlog.opener			self._opener = revlog.opener
	self._index_uuid = index_uuid			self._index_uuid = index_uuid
	self._data_uuid = data_uuid			self._data_uuid = data_uuid
				self._sidedata_uuid = sidedata_uuid
	# thes asserts should be True as long as we have a single index filename			# thes asserts should be True as long as we have a single index filename
	assert index_end <= pending_index_end			assert index_end <= pending_index_end
	assert data_end <= pending_data_end			assert data_end <= pending_data_end
				assert sidedata_end <= pending_sidedata_end
	self._initial_index_end = index_end			self._initial_index_end = index_end
	self._pending_index_end = pending_index_end			self._pending_index_end = pending_index_end
	self._initial_data_end = data_end			self._initial_data_end = data_end
	self._pending_data_end = pending_data_end			self._pending_data_end = pending_data_end
				self._initial_sidedata_end = sidedata_end
				self._pending_sidedata_end = pending_sidedata_end
	if use_pending:			if use_pending:
	self._index_end = self._pending_index_end			self._index_end = self._pending_index_end
	self._data_end = self._pending_data_end			self._data_end = self._pending_data_end
				self._sidedata_end = self._pending_sidedata_end
	else:			else:
	self._index_end = self._initial_index_end			self._index_end = self._initial_index_end
	self._data_end = self._initial_data_end			self._data_end = self._initial_data_end
				self._sidedata_end = self._initial_sidedata_end
	self.default_compression_header = default_compression_header			self.default_compression_header = default_compression_header

	def index_filepath(self):			def index_filepath(self):
	"""file path to the current index file associated to this docket"""			"""file path to the current index file associated to this docket"""
	# very simplistic version at first			# very simplistic version at first
	if self._index_uuid is None:			if self._index_uuid is None:
	self._index_uuid = make_uid()			self._index_uuid = make_uid()
	return b"%s-%s.idx" % (self._radix, self._index_uuid)			return b"%s-%s.idx" % (self._radix, self._index_uuid)

	def data_filepath(self):			def data_filepath(self):
	"""file path to the current data file associated to this docket"""			"""file path to the current data file associated to this docket"""
	# very simplistic version at first			# very simplistic version at first
	if self._data_uuid is None:			if self._data_uuid is None:
	self._data_uuid = make_uid()			self._data_uuid = make_uid()
	return b"%s-%s.dat" % (self._radix, self._data_uuid)			return b"%s-%s.dat" % (self._radix, self._data_uuid)

				def sidedata_filepath(self):
				"""file path to the current sidedata file associated to this docket"""
				# very simplistic version at first
				if self._sidedata_uuid is None:
				self._sidedata_uuid = make_uid()
				return b"%s-%s.sda" % (self._radix, self._sidedata_uuid)

	@property			@property
	def index_end(self):			def index_end(self):
	return self._index_end			return self._index_end

	@index_end.setter			@index_end.setter
	def index_end(self, new_size):			def index_end(self, new_size):
	if new_size != self._index_end:			if new_size != self._index_end:
	self._index_end = new_size			self._index_end = new_size
	self._dirty = True			self._dirty = True

	@property			@property
	def data_end(self):			def data_end(self):
	return self._data_end			return self._data_end

	@data_end.setter			@data_end.setter
	def data_end(self, new_size):			def data_end(self, new_size):
	if new_size != self._data_end:			if new_size != self._data_end:
	self._data_end = new_size			self._data_end = new_size
	self._dirty = True			self._dirty = True

				@property
				def sidedata_end(self):
				return self._sidedata_end

				@sidedata_end.setter
				def sidedata_end(self, new_size):
				if new_size != self._sidedata_end:
				self._sidedata_end = new_size
				self._dirty = True

	def write(self, transaction, pending=False, stripping=False):			def write(self, transaction, pending=False, stripping=False):
	"""write the modification of disk if any			"""write the modification of disk if any

	This make the new content visible to all process"""			This make the new content visible to all process"""
	if not self._dirty:			if not self._dirty:
	return False			return False
	else:			else:
	if self._read_only:			if self._read_only:
	# if pending we still need to the write final data eventually			# if pending we still need to the write final data eventually
	self._dirty = pending			self._dirty = pending
	return True			return True

	def _serialize(self, pending=False):			def _serialize(self, pending=False):
	if pending:			if pending:
	official_index_end = self._initial_index_end			official_index_end = self._initial_index_end
	official_data_end = self._initial_data_end			official_data_end = self._initial_data_end
				official_sidedata_end = self._initial_sidedata_end
	else:			else:
	official_index_end = self._index_end			official_index_end = self._index_end
	official_data_end = self._data_end			official_data_end = self._data_end
				official_sidedata_end = self._sidedata_end

	# this assert should be True as long as we have a single index filename			# this assert should be True as long as we have a single index filename
	assert official_data_end <= self._data_end			assert official_data_end <= self._data_end
				assert official_sidedata_end <= self._sidedata_end
	data = (			data = (
	self._version_header,			self._version_header,
	len(self._index_uuid),			len(self._index_uuid),
	len(self._data_uuid),			len(self._data_uuid),
				len(self._sidedata_uuid),
	official_index_end,			official_index_end,
	self._index_end,			self._index_end,
	official_data_end,			official_data_end,
	self._data_end,			self._data_end,
				official_sidedata_end,
				self._sidedata_end,
	self.default_compression_header,			self.default_compression_header,
	)			)
	s = []			s = []
	s.append(S_HEADER.pack(*data))			s.append(S_HEADER.pack(*data))
	s.append(self._index_uuid)			s.append(self._index_uuid)
	s.append(self._data_uuid)			s.append(self._data_uuid)
				s.append(self._sidedata_uuid)
	return b''.join(s)			return b''.join(s)


	def default_docket(revlog, version_header):			def default_docket(revlog, version_header):
	"""given a revlog version a new docket object for the given revlog"""			"""given a revlog version a new docket object for the given revlog"""
	rl_version = version_header & 0xFFFF			rl_version = version_header & 0xFFFF
	if rl_version not in (constants.REVLOGV2, constants.CHANGELOGV2):			if rl_version not in (constants.REVLOGV2, constants.CHANGELOGV2):
	return None			return None
	version_header = next(iheader)			version_header = next(iheader)

	index_uuid_size = next(iheader)			index_uuid_size = next(iheader)
	index_uuid = get_data(index_uuid_size)			index_uuid = get_data(index_uuid_size)

	data_uuid_size = next(iheader)			data_uuid_size = next(iheader)
	data_uuid = get_data(data_uuid_size)			data_uuid = get_data(data_uuid_size)

				sidedata_uuid_size = next(iheader)
				sidedata_uuid = get_data(sidedata_uuid_size)

	index_size = next(iheader)			index_size = next(iheader)

	pending_index_size = next(iheader)			pending_index_size = next(iheader)

	data_size = next(iheader)			data_size = next(iheader)

	pending_data_size = next(iheader)			pending_data_size = next(iheader)

				sidedata_size = next(iheader)

				pending_sidedata_size = next(iheader)

	default_compression_header = next(iheader)			default_compression_header = next(iheader)

	docket = RevlogDocket(			docket = RevlogDocket(
	revlog,			revlog,
	use_pending=use_pending,			use_pending=use_pending,
	version_header=version_header,			version_header=version_header,
	index_uuid=index_uuid,			index_uuid=index_uuid,
	data_uuid=data_uuid,			data_uuid=data_uuid,
				sidedata_uuid=sidedata_uuid,
	index_end=index_size,			index_end=index_size,
	pending_index_end=pending_index_size,			pending_index_end=pending_index_size,
	data_end=data_size,			data_end=data_size,
	pending_data_end=pending_data_size,			pending_data_end=pending_data_size,
				sidedata_end=sidedata_size,
				pending_sidedata_end=pending_sidedata_size,
	default_compression_header=default_compression_header,			default_compression_header=default_compression_header,
	)			)
	return docket			return docket

mercurial/store.py


	REVLOG_FILES_MAIN_EXT = (b'.i', b'i.tmpcensored')			REVLOG_FILES_MAIN_EXT = (b'.i', b'i.tmpcensored')
	REVLOG_FILES_OTHER_EXT = (			REVLOG_FILES_OTHER_EXT = (
	b'.idx',			b'.idx',
	b'.d',			b'.d',
	b'.dat',			b'.dat',
	b'.n',			b'.n',
	b'.nd',			b'.nd',
				b'.sda',
	b'd.tmpcensored',			b'd.tmpcensored',
	)			)
	# files that are "volatile" and might change between listing and streaming			# files that are "volatile" and might change between listing and streaming
	#			#
	# note: the ".nd" file are nodemap data and won't "change" but they might be			# note: the ".nd" file are nodemap data and won't "change" but they might be
	# deleted.			# deleted.
	REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd')			REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd')

tests/test-revlog-v2.t

	--------------------------------			--------------------------------

	We should have have:			We should have have:
	- a docket			- a docket
	- a index file with a unique name			- a index file with a unique name
	- a data file			- a data file

	$ ls .hg/store/00changelog* .hg/store/00manifest*			$ ls .hg/store/00changelog* .hg/store/00manifest*
	.hg/store/00changelog-6b8ab34b.dat			.hg/store/00changelog-1335303a.sda
	.hg/store/00changelog-88698448.idx			.hg/store/00changelog-6b8ab34b.idx
				.hg/store/00changelog-b875dfc5.dat
	.hg/store/00changelog.i			.hg/store/00changelog.i
	.hg/store/00manifest-1335303a.dat			.hg/store/00manifest-05a21d65.idx
	.hg/store/00manifest-b875dfc5.idx			.hg/store/00manifest-43c37dde.dat
				.hg/store/00manifest-e2c9362a.sda
	.hg/store/00manifest.i			.hg/store/00manifest.i

			Path	Packages
M			mercurial/configitems.py (2 lines)
M			mercurial/revlog.py (109 lines)
M			mercurial/revlogutils/docket.py (48 lines)
M			mercurial/store.py (1 line)
M			tests/test-revlog-v2.t (10 lines)

This is an archive of the discontinued Mercurial Phabricator instance.

revlog: store sidedata in their own file
ClosedPublic

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 28397

mercurial/configitems.py

mercurial/revlog.py

mercurial/revlogutils/docket.py

mercurial/store.py

tests/test-revlog-v2.t

This is an archive of the discontinued Mercurial Phabricator instance.

revlog: store sidedata in their own fileClosedPublic

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 28397

mercurial/configitems.py

mercurial/revlog.py

mercurial/revlogutils/docket.py

mercurial/store.py

tests/test-revlog-v2.t

revlog: store sidedata in their own file
ClosedPublic

Revision Contents
Changeset List