mercurial/revlogutils/nodemap.py
43	need b'' here and below for py3?
50	and here? this is a series that introduced a lot of new code, so try running all (previously passing) py3 test at least at the end of the series

mercurial/revlogutils/nodemap.py
51–52	Actually, does `revlog.opener.write(revlog.nodemap_file, data)` work?

mercurial/revlogutils/nodemap.py
43	not needed here, it seems, because `ProgrammingError` converts it for us

mercurial/revlogutils/nodemap.py
50	If you can taje the series starting at D8011, if would help me to make sure all angles are covered.
51–52	probably, but we will need the context manager approach later in the series so I don't think it is worth updating.

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	19155		Jan 11 2020, 12:03 PM	★	★
Diff 2	19172		Jan 13 2020, 10:06 AM	★	★
Diff 3	19185		Jan 13 2020, 1:04 PM	★	★
Diff 4	19425		Jan 17 2020, 1:05 PM	★	★
Diff 5	19751		Jan 31 2020, 10:27 AM	★	★
Diff 6	19780		Jan 31 2020, 2:43 PM	★	★
Diff 7	19825		Feb 2 2020, 3:17 AM	★	★
Diff 8	19883		Feb 4 2020, 7:23 PM	★	★
Diff 9	20016		Feb 7 2020, 5:28 PM	★	★
Diff 10	20103	rHG5962fd0d1045a6cbb1933da569b97a1b6df9a717	Jan 15 2020, 9:47 AM	★	★

Commit	Parents	Author	Summary	Date
898e2aec9853	8cb889bfb454	Pierre-Yves David		Jan 15 2020, 9:47 AM

Status	Author	Revision
Closed	marmoute	D7894 nodemap: introduce an option to use mmap to read the nodemap mapping
Closed	marmoute	D7893 nodemap: update the index with the newly written data (when appropriate)
Closed	marmoute	D7892 nodemap: never read more than the expected data amount
Closed	marmoute	D7891 nodemap: write new data from the expected current data length
Closed	marmoute	D7890 nodemap: double check the source docket when doing incremental update
Closed	marmoute	D7889 nodemap: track the total and unused amount of data in the rawdata file
Closed	marmoute	D7888 nodemap: track the maximum revision tracked in the nodemap
Closed	marmoute	D7887 nodemap: add a flag to dump the details of the docket
Closed	marmoute	D7886 nodemap: introduce append-only incremental update of the persisten data
Closed	marmoute	D7885 nodemap: keep track of the docket for loaded data
Closed	marmoute	D7884 nodemap: introduce an explicit class/object for the docket
Closed	marmoute	D7883 nodemap: keep track of the ondisk id of nodemap blocks
Closed	marmoute	D7847 nodemap: provide the on disk data to indexes who support it
Closed	marmoute	D7846 nodemap: all check that revision and nodes match in the nodemap
Closed	marmoute	D7845 nodemap: add basic checking of the on disk nodemap content
Closed	marmoute	D7844 nodemap: code to parse the persistent binary nodemap data
Closed	marmoute	D7843 nodemap: move the iteratio inside the Block object
Closed	marmoute	D7842 nodemap: use an explicit "Block" object in the reference implementation
Closed	marmoute	D7841 nodemap: add a optional `nodemap_add_full` method on indexes
Closed	marmoute	D7840 nodemap: add a (python) index class for persistent nodemap testing
Closed	marmoute	D7839 nodemap: delete older raw data file when creating a new ones
Closed	marmoute	D7838 nodemap: use an intermediate "docket" file to carry small metadata
Closed	marmoute	D7837 nodemap: only use persistent nodemap for non-inlined revlog
Closed	marmoute	D7836 nodemap: add a function to read the data from disk
Closed	marmoute	D7835 nodemap: write nodemap data on disk
Closed	marmoute	D7834 nodemap: have some python code writing a nodemap in persistent binary form

Diff 20016

mercurial/changelog.py

	datafile = b'00changelog.d'			datafile = b'00changelog.d'
	revlog.revlog.__init__(			revlog.revlog.__init__(
	self,			self,
	opener,			opener,
	indexfile,			indexfile,
	datafile=datafile,			datafile=datafile,
	checkambig=True,			checkambig=True,
	mmaplargeindex=True,			mmaplargeindex=True,
				persistentnodemap=opener.options.get(
				b'exp-persistent-nodemap', False
				),
	)			)

	if self._initempty and (self.version & 0xFFFF == revlog.REVLOGV1):			if self._initempty and (self.version & 0xFFFF == revlog.REVLOGV1):
	# changelogs don't benefit from generaldelta.			# changelogs don't benefit from generaldelta.

	self.version &= ~revlog.FLAG_GENERALDELTA			self.version &= ~revlog.FLAG_GENERALDELTA
	self._generaldelta = False			self._generaldelta = False

mercurial/configitems.py

	)			)
	coreconfigitem(			coreconfigitem(
	b'experimental', b'revisions.disambiguatewithin', default=None,			b'experimental', b'revisions.disambiguatewithin', default=None,
	)			)
	coreconfigitem(			coreconfigitem(
	b'experimental', b'rust.index', default=False,			b'experimental', b'rust.index', default=False,
	)			)
	coreconfigitem(			coreconfigitem(
				b'experimental', b'exp-persistent-nodemap', default=False,
				)
				coreconfigitem(
	b'experimental', b'server.filesdata.recommended-batch-size', default=50000,			b'experimental', b'server.filesdata.recommended-batch-size', default=50000,
	)			)
	coreconfigitem(			coreconfigitem(
	b'experimental',			b'experimental',
	b'server.manifestdata.recommended-batch-size',			b'server.manifestdata.recommended-batch-size',
	default=100000,			default=100000,
	)			)
	coreconfigitem(			coreconfigitem(

mercurial/localrepo.py

	msg = _(b'invalid value for `storage.revlog.zstd.level` config: %d')			msg = _(b'invalid value for `storage.revlog.zstd.level` config: %d')
	raise error.Abort(msg % options[b'zstd.level'])			raise error.Abort(msg % options[b'zstd.level'])

	if repository.NARROW_REQUIREMENT in requirements:			if repository.NARROW_REQUIREMENT in requirements:
	options[b'enableellipsis'] = True			options[b'enableellipsis'] = True

	if ui.configbool(b'experimental', b'rust.index'):			if ui.configbool(b'experimental', b'rust.index'):
	options[b'rust.index'] = True			options[b'rust.index'] = True
				if ui.configbool(b'experimental', b'exp-persistent-nodemap'):
				options[b'exp-persistent-nodemap'] = True

	return options			return options


	def makemain(**kwargs):			def makemain(**kwargs):
	"""Produce a type conforming to ``ilocalrepositorymain``."""			"""Produce a type conforming to ``ilocalrepositorymain``."""
	return localrepository			return localrepository

mercurial/revlog.py

	self,			self,
	opener,			opener,
	indexfile,			indexfile,
	datafile=None,			datafile=None,
	checkambig=False,			checkambig=False,
	mmaplargeindex=False,			mmaplargeindex=False,
	censorable=False,			censorable=False,
	upperboundcomp=None,			upperboundcomp=None,
				persistentnodemap=False,
	):			):
	"""			"""
	create a revlog object			create a revlog object

	opener is a function that abstracts the file opening operation			opener is a function that abstracts the file opening operation
	and can be used to implement COW semantics or the like.			and can be used to implement COW semantics or the like.

	"""			"""
	self.upperboundcomp = upperboundcomp			self.upperboundcomp = upperboundcomp
	self.indexfile = indexfile			self.indexfile = indexfile
	self.datafile = datafile or (indexfile[:-2] + b".d")			self.datafile = datafile or (indexfile[:-2] + b".d")
				self.nodemap_file = None
				if persistentnodemap:
				self.nodemap_file = indexfile[:-2] + b".n"

	self.opener = opener			self.opener = opener
	# When True, indexfile is opened with checkambig=True at writing, to			# When True, indexfile is opened with checkambig=True at writing, to
	# avoid file stat ambiguity.			# avoid file stat ambiguity.
	self._checkambig = checkambig			self._checkambig = checkambig
	self._mmaplargeindex = mmaplargeindex			self._mmaplargeindex = mmaplargeindex
	self._censorable = censorable			self._censorable = censorable
	# 3-tuple of (node, rev, text) for a raw revision.			# 3-tuple of (node, rev, text) for a raw revision.
	self._revisioncache = None			self._revisioncache = None
	ifh.write(entry)			ifh.write(entry)
	else:			else:
	offset += curr * self._io.size			offset += curr * self._io.size
	transaction.add(self.indexfile, offset, curr)			transaction.add(self.indexfile, offset, curr)
	ifh.write(entry)			ifh.write(entry)
	ifh.write(data[0])			ifh.write(data[0])
	ifh.write(data[1])			ifh.write(data[1])
	self._enforceinlinesize(transaction, ifh)			self._enforceinlinesize(transaction, ifh)
				nodemaputil.setup_persistent_nodemap(transaction, self)

	def addgroup(self, deltas, linkmapper, transaction, addrevisioncb=None):			def addgroup(self, deltas, linkmapper, transaction, addrevisioncb=None):
	"""			"""
	add a delta group			add a delta group

	given a set of deltas, add them to the revision log. the			given a set of deltas, add them to the revision log. the
	first delta is against its parent, which should be in our			first delta is against its parent, which should be in our
	log, the rest are against the previous delta.			log, the rest are against the previous delta.

mercurial/revlogutils/nodemap.py

	)			)


	class NodeMap(dict):			class NodeMap(dict):
	def __missing__(self, x):			def __missing__(self, x):
	raise error.RevlogError(b'unknown node: %s' % x)			raise error.RevlogError(b'unknown node: %s' % x)


				def setup_persistent_nodemap(tr, revlog):
				"""Install whatever is needed transaction side to persist a nodemap on disk

				(only actually persist the nodemap if this is relevant for this revlog)
				"""
				if revlog.nodemap_file is None:
				return # we do not use persistent_nodemap on this revlog
				callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file
				if tr.hasfinalize(callback_id):
				return # no need to register again
				tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))


				def _persist_nodemap(tr, revlog):
				"""Write nodemap data on disk for a given revlog
				"""
				if getattr(revlog, 'filteredrevs', ()):
				raise error.ProgrammingError(
				"cannot persist nodemap of a filtered changelog"
				martinvonzUnsubmitted Not Done need b'' here and below for py3? martinvonz: need b'' here and below for py3?
				martinvonzUnsubmitted Not Done not needed here, it seems, because `ProgrammingError` converts it for us martinvonz: not needed here, it seems, because `ProgrammingError` converts it for us
				)
				if revlog.nodemap_file is None:
				msg = "calling persist nodemap on a revlog without the feature enableb"
				raise error.ProgrammingError(msg)
				data = persistent_data(revlog.index)
				# EXP-TODO: if this is a cache, this should use a cache vfs, not a
				# store vfs
				martinvonzUnsubmitted Not Done and here? this is a series that introduced a lot of new code, so try running all (previously passing) py3 test at least at the end of the series martinvonz: and here? this is a series that introduced a lot of new code, so try running all (previously…
				marmouteAuthorUnsubmitted Done If you can taje the series starting at D8011, if would help me to make sure all angles are covered. marmoute: If you can taje the series starting at D8011, if would help me to make sure all angles are…
				with revlog.opener(revlog.nodemap_file, b'w') as f:
				f.write(data)
				martinvonzUnsubmitted Not Done Actually, does `revlog.opener.write(revlog.nodemap_file, data)` work? martinvonz: Actually, does `revlog.opener.write(revlog.nodemap_file, data)` work?
				marmouteAuthorUnsubmitted Done probably, but we will need the context manager approach later in the series so I don't think it is worth updating. marmoute: probably, but we will need the context manager approach later in the series so I don't think it…
				# EXP-TODO: if the transaction abort, we should remove the new data and
				# reinstall the old one. (This will be simpler when the file format get a
				# bit more advanced)


	### Nodemap Trie			### Nodemap Trie
	#			#
	# This is a simple reference implementation to compute and persist a nodemap			# This is a simple reference implementation to compute and persist a nodemap
	# trie. This reference implementation is write only. The python version of this			# trie. This reference implementation is write only. The python version of this
	# is not expected to be actually used, since it wont provide performance			# is not expected to be actually used, since it wont provide performance
	# improvement over existing non-persistent C implementation.			# improvement over existing non-persistent C implementation.
	#			#
	# The nodemap is persisted as Trie using 4bits-address/16-entries block. each			# The nodemap is persisted as Trie using 4bits-address/16-entries block. each

tests/test-persistent-nodemap.t

	===================================			===================================
	Test the persistent on-disk nodemap			Test the persistent on-disk nodemap
	===================================			===================================


	$ hg init test-repo			$ hg init test-repo
	$ cd test-repo			$ cd test-repo
				$ cat << EOF >> .hg/hgrc
				> [experimental]
				> exp-persistent-nodemap=yes
				> EOF
	$ hg debugbuilddag .+5000			$ hg debugbuilddag .+5000
	$ hg debugnodemap --dump \| f --sha256 --bytes=256 --hexdump --size			$ hg debugnodemap --dump \| f --sha256 --size
				size=122880, sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
				$ f --sha256 --bytes=256 --hexdump --size < .hg/store/00changelog.n
	size=122880, sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7			size=122880, sha256=b961925120e1c9bc345c199b2cc442abc477029fdece37ef9d99cbe59c0558b7
	0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|			0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|
	0010: ff ff ff ff ff ff ff ff ff ff fa c2 ff ff ff ff \|................\|			0010: ff ff ff ff ff ff ff ff ff ff fa c2 ff ff ff ff \|................\|
	0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|			0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|
	0030: ff ff ff ff ff ff ed b3 ff ff ff ff ff ff ff ff \|................\|			0030: ff ff ff ff ff ff ed b3 ff ff ff ff ff ff ff ff \|................\|
	0040: ff ff ff ff ff ff ee 34 00 00 00 00 ff ff ff ff \|.......4........\|			0040: ff ff ff ff ff ff ee 34 00 00 00 00 ff ff ff ff \|.......4........\|
	0050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|			0050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|
	0060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|			0060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|

			Path	Packages
M			mercurial/changelog.py (3 lines)
M			mercurial/configitems.py (3 lines)
M			mercurial/localrepo.py (2 lines)
M			mercurial/revlog.py (6 lines)
M			mercurial/revlogutils/nodemap.py (33 lines)
M			tests/test-persistent-nodemap.t (8 lines)

This is an archive of the discontinued Mercurial Phabricator instance.

nodemap: write nodemap data on disk
ClosedPublic

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 20016

mercurial/changelog.py

mercurial/configitems.py

mercurial/localrepo.py

mercurial/revlog.py

mercurial/revlogutils/nodemap.py

tests/test-persistent-nodemap.t

This is an archive of the discontinued Mercurial Phabricator instance.

nodemap: write nodemap data on diskClosedPublic

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 20016

mercurial/changelog.py

mercurial/configitems.py

mercurial/localrepo.py

mercurial/revlog.py

mercurial/revlogutils/nodemap.py

tests/test-persistent-nodemap.t

nodemap: write nodemap data on disk
ClosedPublic

Revision Contents
Changeset List