nodemap: add a (python) index class for persistent nodemap testing
ClosedPublic

Authored by marmoute on Jan 13 2020, 10:06 AM.

Download Raw Diff

Details

Reviewers

indygreg

Group Reviewers

hg-reviewers

Commits

rHG6f9e8e142cea: nodemap: add a (python) index class for persistent nodemap testing

Summary

Using the persistent nodemap require a compeling performance boost and an
existing implementation. The benefit of the persistent nodemap for pure python
code is unclear and we don't have a C implementation for it. Yet we would like
to actually start testing it in more details and define an API for using that
persistent nodemap.

We introduce a new devel config option to use an index class dedicated to
Nodemap Testing. This feature is "pure" only because having using a pure-python
index with the cext policy proved more difficult than I would like.

There is nothing going on in that class for now, but the coming changeset will
change that.

Diff Detail

Repository

rHG Mercurial

Lint

Lint Skipped

Unit

Unit Tests Skipped

Event Timeline

marmoute created this revision.Jan 13 2020, 10:06 AM

Herald added a reviewer: indygreg. · View Herald TranscriptJan 13 2020, 10:06 AM

Herald added a reviewer: hg-reviewers. · View Herald Transcript

Herald added a subscriber: mercurial-devel. · View Herald Transcript

marmoute added a child revision: D7841: nodemap: add a optional `nodemap_add_full` method on indexes.Jan 13 2020, 10:07 AM

marmoute updated this revision to Diff 19186.Jan 13 2020, 1:05 PM

marmoute updated this revision to Diff 19428.Jan 17 2020, 1:06 PM

marmoute updated this revision to Diff 19756.Jan 31 2020, 10:28 AM

rebase to latest default

marmoute updated this revision to Diff 20017.Feb 7 2020, 5:28 PM

marmoute added a commit: rHG6f9e8e142cea: nodemap: add a (python) index class for persistent nodemap testing.Feb 10 2020, 8:24 PM

This revision was not accepted when it landed; it landed in state Needs Review.

Closed by commit rHG6f9e8e142cea: nodemap: add a (python) index class for persistent nodemap testing (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/configitems.py (3 lines)
M			mercurial/localrepo.py (2 lines)
M			mercurial/pure/parsers.py (15 lines)
M			mercurial/revlog.py (23 lines)
M			tests/test-persistent-nodemap.t (2 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	19177		Jan 13 2020, 10:06 AM	★	★
Diff 2	19186		Jan 13 2020, 1:05 PM	★	★
Diff 3	19428		Jan 17 2020, 1:05 PM	★	★
Diff 4	19756		Jan 31 2020, 10:28 AM	★	★
Diff 5	19888		Feb 4 2020, 7:24 PM	★	★
Diff 6	20017		Feb 7 2020, 5:28 PM	★	★
Diff 7	20108	rHG6f9e8e142cea259aa3e74b69f763b67930ce38c6	Jan 15 2020, 9:48 AM	★	★

Commit	Parents	Author	Summary	Date
767327c0943f	299c0c0c98dc	Pierre-Yves David		Jan 15 2020, 9:48 AM

Status	Author	Revision
Closed	marmoute	D7894 nodemap: introduce an option to use mmap to read the nodemap mapping
Closed	marmoute	D7893 nodemap: update the index with the newly written data (when appropriate)
Closed	marmoute	D7892 nodemap: never read more than the expected data amount
Closed	marmoute	D7891 nodemap: write new data from the expected current data length
Closed	marmoute	D7890 nodemap: double check the source docket when doing incremental update
Closed	marmoute	D7889 nodemap: track the total and unused amount of data in the rawdata file
Closed	marmoute	D7888 nodemap: track the maximum revision tracked in the nodemap
Closed	marmoute	D7887 nodemap: add a flag to dump the details of the docket
Closed	marmoute	D7886 nodemap: introduce append-only incremental update of the persisten data
Closed	marmoute	D7885 nodemap: keep track of the docket for loaded data
Closed	marmoute	D7884 nodemap: introduce an explicit class/object for the docket
Closed	marmoute	D7883 nodemap: keep track of the ondisk id of nodemap blocks
Closed	marmoute	D7847 nodemap: provide the on disk data to indexes who support it
Closed	marmoute	D7846 nodemap: all check that revision and nodes match in the nodemap
Closed	marmoute	D7845 nodemap: add basic checking of the on disk nodemap content
Closed	marmoute	D7844 nodemap: code to parse the persistent binary nodemap data
Closed	marmoute	D7843 nodemap: move the iteratio inside the Block object
Closed	marmoute	D7842 nodemap: use an explicit "Block" object in the reference implementation
Closed	marmoute	D7841 nodemap: add a optional `nodemap_add_full` method on indexes
Closed	marmoute	D7840 nodemap: add a (python) index class for persistent nodemap testing
Closed	marmoute	D7839 nodemap: delete older raw data file when creating a new ones
Closed	marmoute	D7838 nodemap: use an intermediate "docket" file to carry small metadata
Closed	marmoute	D7837 nodemap: only use persistent nodemap for non-inlined revlog
Closed	marmoute	D7836 nodemap: add a function to read the data from disk
Closed	marmoute	D7835 nodemap: write nodemap data on disk
Closed	marmoute	D7834 nodemap: have some python code writing a nodemap in persistent binary form

Diff 19428

mercurial/configitems.py

	)			)
	coreconfigitem(			coreconfigitem(
	b'devel', b'warn-empty-changegroup', default=False,			b'devel', b'warn-empty-changegroup', default=False,
	)			)
	coreconfigitem(			coreconfigitem(
	b'devel', b'legacy.exchange', default=list,			b'devel', b'legacy.exchange', default=list,
	)			)
	coreconfigitem(			coreconfigitem(
				b'devel', b'persistent-nodemap', default=False,
				)
				coreconfigitem(
	b'devel', b'servercafile', default=b'',			b'devel', b'servercafile', default=b'',
	)			)
	coreconfigitem(			coreconfigitem(
	b'devel', b'serverexactprotocol', default=b'',			b'devel', b'serverexactprotocol', default=b'',
	)			)
	coreconfigitem(			coreconfigitem(
	b'devel', b'serverrequirecert', default=False,			b'devel', b'serverrequirecert', default=False,
	)			)

mercurial/localrepo.py


	if repository.NARROW_REQUIREMENT in requirements:			if repository.NARROW_REQUIREMENT in requirements:
	options[b'enableellipsis'] = True			options[b'enableellipsis'] = True

	if ui.configbool(b'experimental', b'rust.index'):			if ui.configbool(b'experimental', b'rust.index'):
	options[b'rust.index'] = True			options[b'rust.index'] = True
	if ui.configbool('experimental', 'exp-persistent-nodemap'):			if ui.configbool('experimental', 'exp-persistent-nodemap'):
	options[b'exp-persistent-nodemap'] = True			options[b'exp-persistent-nodemap'] = True
				if ui.configbool('devel', 'persistent-nodemap'):
				options[b'devel-force-nodemap'] = True

	return options			return options


	def makemain(**kwargs):			def makemain(**kwargs):
	"""Produce a type conforming to ``ilocalrepositorymain``."""			"""Produce a type conforming to ``ilocalrepositorymain``."""
	return localrepository			return localrepository

mercurial/pure/parsers.py

	if i < self._lgt:			if i < self._lgt:
	self._data = self._data[: i * indexsize]			self._data = self._data[: i * indexsize]
	self._lgt = i			self._lgt = i
	self._extra = []			self._extra = []
	else:			else:
	self._extra = self._extra[: i - self._lgt]			self._extra = self._extra[: i - self._lgt]


				class PersistentNodeMapIndexObject(IndexObject):
				"""a Debug oriented class to test persistent nodemap

				We need a simple python object to test API and higher level behavior. See
				the Rust implementation for more serious usage. This should be used only
				through the dedicated `devel.persistent-nodemap` config.
				"""


	class InlinedIndexObject(BaseIndexObject):			class InlinedIndexObject(BaseIndexObject):
	def __init__(self, data, inline=0):			def __init__(self, data, inline=0):
	self._data = data			self._data = data
	self._lgt = self._inline_scan(None)			self._lgt = self._inline_scan(None)
	self._inline_scan(self._lgt)			self._inline_scan(self._lgt)
	self._extra = []			self._extra = []

	def _inline_scan(self, lgt):			def _inline_scan(self, lgt):


	def parse_index2(data, inline):			def parse_index2(data, inline):
	if not inline:			if not inline:
	return IndexObject(data), None			return IndexObject(data), None
	return InlinedIndexObject(data, inline), (0, data)			return InlinedIndexObject(data, inline), (0, data)


				def parse_index_devel_nodemap(data, inline):
				"""like parse_index2, but alway return a PersistentNodeMapIndexObject
				"""
				return PersistentNodeMapIndexObject(data), None


	def parse_dirstate(dmap, copymap, st):			def parse_dirstate(dmap, copymap, st):
	parents = [st[:20], st[20:40]]			parents = [st[:20], st[20:40]]
	# dereference fields so they will be local in loop			# dereference fields so they will be local in loop
	format = b">cllll"			format = b">cllll"
	e_size = struct.calcsize(format)			e_size = struct.calcsize(format)
	pos1 = 40			pos1 = 40
	l = len(st)			l = len(st)

mercurial/revlog.py


	def packentry(self, entry, node, version, rev):			def packentry(self, entry, node, version, rev):
	p = indexformatng_pack(*entry)			p = indexformatng_pack(*entry)
	if rev == 0:			if rev == 0:
	p = versionformat_pack(version) + p[4:]			p = versionformat_pack(version) + p[4:]
	return p			return p


				NodemapRevlogIO = None

				if util.safehasattr(parsers, 'parse_index_devel_nodemap'):

				class NodemapRevlogIO(revlogio):
				"""A debug oriented IO class that return a PersistentNodeMapIndexObject

				The PersistentNodeMapIndexObject object is meant to test the persistent nodemap feature.
				"""

				def parseindex(self, data, inline):
				index, cache = parsers.parse_index_devel_nodemap(data, inline)
				return index, cache


	class rustrevlogio(revlogio):			class rustrevlogio(revlogio):
	def parseindex(self, data, inline):			def parseindex(self, data, inline):
	index, cache = super(rustrevlogio, self).parseindex(data, inline)			index, cache = super(rustrevlogio, self).parseindex(data, inline)
	return rustrevlog.MixedIndex(index), cache			return rustrevlog.MixedIndex(index), cache


	class revlog(object):			class revlog(object):
	"""			"""
	_(b'unknown version (%d) in revlog %s') % (fmt, self.indexfile)			_(b'unknown version (%d) in revlog %s') % (fmt, self.indexfile)
	)			)
	# sparse-revlog can't be on without general-delta (issue6056)			# sparse-revlog can't be on without general-delta (issue6056)
	if not self._generaldelta:			if not self._generaldelta:
	self._sparserevlog = False			self._sparserevlog = False

	self._storedeltachains = True			self._storedeltachains = True

				devel_nodemap = (
				self.nodemap_file
				and opts.get(b'devel-force-nodemap', False)
				and NodemapRevlogIO is not None
				)

	self._io = revlogio()			self._io = revlogio()
	if self.version == REVLOGV0:			if self.version == REVLOGV0:
	self._io = revlogoldio()			self._io = revlogoldio()
				elif devel_nodemap:
				self._io = NodemapRevlogIO()
	elif rustrevlog is not None and self.opener.options.get(b'rust.index'):			elif rustrevlog is not None and self.opener.options.get(b'rust.index'):
	self._io = rustrevlogio()			self._io = rustrevlogio()
	try:			try:
	d = self._io.parseindex(indexdata, self._inline)			d = self._io.parseindex(indexdata, self._inline)
	except (ValueError, IndexError):			except (ValueError, IndexError):
	raise error.RevlogError(			raise error.RevlogError(
	_(b"index %s is corrupted") % self.indexfile			_(b"index %s is corrupted") % self.indexfile
	)			)

tests/test-persistent-nodemap.t

	===================================			===================================
	Test the persistent on-disk nodemap			Test the persistent on-disk nodemap
	===================================			===================================


	$ hg init test-repo			$ hg init test-repo
	$ cd test-repo			$ cd test-repo
	$ cat << EOF >> .hg/hgrc			$ cat << EOF >> .hg/hgrc
	> [experimental]			> [experimental]
	> exp-persistent-nodemap=yes			> exp-persistent-nodemap=yes
				> [devel]
				> persistent-nodemap=yes
	> EOF			> EOF
	$ hg debugbuilddag .+5000			$ hg debugbuilddag .+5000
	$ f --size .hg/store/00changelog.n			$ f --size .hg/store/00changelog.n
	.hg/store/00changelog.n: size=18			.hg/store/00changelog.n: size=18
	$ f --sha256 .hg/store/00changelog-*.nd			$ f --sha256 .hg/store/00changelog-*.nd
	.hg/store/00changelog-????????????????.nd: sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c (glob)			.hg/store/00changelog-????????????????.nd: sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c (glob)
	$ hg debugnodemap --dump-new \| f --sha256 --size			$ hg debugnodemap --dump-new \| f --sha256 --size
	size=245760, sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c			size=245760, sha256=bc400bf49f11e83bbd25630439feee6628a80a8602d2e38972eac44cc3efe10c

This is an archive of the discontinued Mercurial Phabricator instance.

nodemap: add a (python) index class for persistent nodemap testingClosedPublic