This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5561

revlog: always enable generaldelta on version 2 revlogs
ClosedPublic

Authored by indygreg on Jan 10 2019, 7:32 PM.

Download Raw Diff

Details

Reviewers

None

Group Reviewers

hg-reviewers

Commits

rHGe7a2cc84dbc0: revlog: always enable generaldelta on version 2 revlogs

Summary

This commit starts the process of diverging version 2 revlogs
from version 1 revlogs.

generaldelta is a useful feature and has been enabled by
default for ages. I can't think of a good reason why the
feature should be disabled. Yes, it is true changelogs
today don't have generaldelta enabled. But that's because
they don't have delta chains enabled, so generaldelta makes
no sense there.

This commit makes generaldelta always enabled on version 2
revlogs.

As part of this, one-off code in changelog.py mucking with
revlog.version had to be made conditional on the revlog
version, as we don't want to change revlog feature
flags on version 2 revlogs. The fact this code exists
is horrible and stems from revlog options being shared by
the opener. We probably want a better API here. But that can
wait for another patch.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Jan 10 2019, 7:32 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptJan 10 2019, 7:32 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

indygreg added a child revision: D5562: revlog: store mmaplargeindex as an instance attribute.Jan 10 2019, 7:32 PM

if self._initempty:

# changelogs don't benefit from generaldelta

+ if self._initempty and (self.version & 0xFFFF == revlog.REVLOGV1):
+ # changelogs don't benefit from generaldelta.

Perhaps, debugdeltachain() and debugrevlog() has to be updated as well.
They peek .version attribute.

Closed by commit rHGe7a2cc84dbc0: revlog: always enable generaldelta on version 2 revlogs (authored by indygreg). · Explain WhyJan 12 2019, 10:27 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/changelog.py (5 lines)
M			mercurial/help/internals/revlogs.txt (8 lines)
M			mercurial/localrepo.py (2 lines)
M			mercurial/revlog.py (20 lines)
M			mercurial/revlogutils/constants.py (4 lines)
M			tests/test-revlog-v2.t (6 lines)

Status	Author	Revision
Needs Revision	indygreg	D5567 revlog: assign revlog._io in version lookup conditional
Needs Revision	indygreg	D5566 revlog: rename revlog I/O classes to reflect version (API)
Changes Planned	indygreg	D5565 revlog: add version 2 format flag to control sparse
Closed	indygreg	D5564 revlog: use separate variables to track version flags
Closed	indygreg	D5563 revlog: inline opener options logic into _loadindex()
Closed	indygreg	D5562 revlog: store mmaplargeindex as an instance attribute
Closed	indygreg	D5561 revlog: always enable generaldelta on version 2 revlogs
Closed	indygreg	D5560 revlog: rename v to versionflags
Closed	indygreg	D5559 revlog: always process opener options
Closed	indygreg	D5558 internals: minor rewriting of revlogs documentation

Diff 13193

mercurial/changelog.py

	indexfile = '00changelog.i.a'			indexfile = '00changelog.i.a'
	else:			else:
	indexfile = '00changelog.i'			indexfile = '00changelog.i'

	datafile = '00changelog.d'			datafile = '00changelog.d'
	revlog.revlog.__init__(self, opener, indexfile, datafile=datafile,			revlog.revlog.__init__(self, opener, indexfile, datafile=datafile,
	checkambig=True, mmaplargeindex=True)			checkambig=True, mmaplargeindex=True)

	if self._initempty:			if self._initempty and (self.version & 0xFFFF == revlog.REVLOGV1):
	# changelogs don't benefit from generaldelta			# changelogs don't benefit from generaldelta.

	self.version &= ~revlog.FLAG_GENERALDELTA			self.version &= ~revlog.FLAG_GENERALDELTA
	self._generaldelta = False			self._generaldelta = False

	# Delta chains for changelogs tend to be very small because entries			# Delta chains for changelogs tend to be very small because entries
	# tend to be small and don't delta well with each. So disable delta			# tend to be small and don't delta well with each. So disable delta
	# chains.			# chains.
	self._storedeltachains = False			self._storedeltachains = False

mercurial/help/internals/revlogs.txt

	Store revision data inline.			Store revision data inline.
	1			1
	Generaldelta encoding.			Generaldelta encoding.

	Version 2 revlogs have the following flags at the specified bit offsets:			Version 2 revlogs have the following flags at the specified bit offsets:

	0			0
	Store revision data inline.			Store revision data inline.
	1
	Generaldelta encoding.

	The following header values are common:			The following header values are common:

	00 00 00 01			00 00 00 01
	v1			v1
	00 01 00 01			00 01 00 01
	v1 + inline			v1 + inline
	00 02 00 01			00 02 00 01
	The first 4 bytes of the revlog are shared between the revlog header			The first 4 bytes of the revlog are shared between the revlog header
	and the 6 byte absolute offset field from the first revlog entry.			and the 6 byte absolute offset field from the first revlog entry.

	Version 2 Format			Version 2 Format
	================			================

	(In development. Format not finalized or stable.)			(In development. Format not finalized or stable.)

	Version 2 is currently identical to version 1. This will obviously			Version 2 is identical to version 2 with the following differences.
	change.
				There is no dedicated generaldelta revlog format flag. Instead,
				the feature is implied enabled by default.

	Delta Chains			Delta Chains
	============			============

	Revision data is encoded as a chain of chunks. Each chain begins with			Revision data is encoded as a chain of chunks. Each chain begins with
	the compressed original full text for that revision. Each subsequent			the compressed original full text for that revision. Each subsequent
	chunk is a delta against the previous revision. We therefore call			chunk is a delta against the previous revision. We therefore call
	these chains of chunks/deltas delta chains.			these chains of chunks/deltas delta chains.

mercurial/localrepo.py

	outgoing = discovery.outgoing(self._repo, missingroots=bases,			outgoing = discovery.outgoing(self._repo, missingroots=bases,
	missingheads=heads)			missingheads=heads)
	return changegroup.makechangegroup(self._repo, outgoing, '01', source)			return changegroup.makechangegroup(self._repo, outgoing, '01', source)

	# End of baselegacywirecommands interface.			# End of baselegacywirecommands interface.

	# Increment the sub-version when the revlog v2 format changes to lock out old			# Increment the sub-version when the revlog v2 format changes to lock out old
	# clients.			# clients.
	REVLOGV2_REQUIREMENT = 'exp-revlogv2.0'			REVLOGV2_REQUIREMENT = 'exp-revlogv2.1'

	# A repository with the sparserevlog feature will have delta chains that			# A repository with the sparserevlog feature will have delta chains that
	# can spread over a larger span. Sparse reading cuts these large spans into			# can spread over a larger span. Sparse reading cuts these large spans into
	# pieces, so that each piece isn't too big.			# pieces, so that each piece isn't too big.
	# Without the sparserevlog capability, reading from the repository could use			# Without the sparserevlog capability, reading from the repository could use
	# huge amounts of memory, because the whole span would be read at once,			# huge amounts of memory, because the whole span would be read at once,
	# including all the intermediate revisions that aren't pertinent for the chain.			# including all the intermediate revisions that aren't pertinent for the chain.
	# This is why once a repository has enabled sparse-read, it becomes required.			# This is why once a repository has enabled sparse-read, it becomes required.

mercurial/revlog.py


	# 2-tuple of file handles being used for active writing.			# 2-tuple of file handles being used for active writing.
	self._writinghandles = None			self._writinghandles = None

	mmapindexthreshold = None			mmapindexthreshold = None
	opts = getattr(opener, 'options', {}) or {}			opts = getattr(opener, 'options', {}) or {}

	if 'revlogv2' in opts:			if 'revlogv2' in opts:
	# version 2 revlogs always use generaldelta.			versionflags = REVLOGV2 \| FLAG_INLINE_DATA
	versionflags = REVLOGV2 \| FLAG_GENERALDELTA \| FLAG_INLINE_DATA
	elif 'revlogv1' in opts:			elif 'revlogv1' in opts:
	versionflags = REVLOGV1 \| FLAG_INLINE_DATA			versionflags = REVLOGV1 \| FLAG_INLINE_DATA
	if 'generaldelta' in opts:			if 'generaldelta' in opts:
	versionflags \|= FLAG_GENERALDELTA			versionflags \|= FLAG_GENERALDELTA
	else:			else:
	versionflags = REVLOG_DEFAULT_VERSION			versionflags = REVLOG_DEFAULT_VERSION

	if 'chunkcachesize' in opts:			if 'chunkcachesize' in opts:
	if len(indexdata) > 0:			if len(indexdata) > 0:
	versionflags = versionformat_unpack(indexdata[:4])[0]			versionflags = versionformat_unpack(indexdata[:4])[0]
	self._initempty = False			self._initempty = False
	except IOError as inst:			except IOError as inst:
	if inst.errno != errno.ENOENT:			if inst.errno != errno.ENOENT:
	raise			raise

	self.version = versionflags			self.version = versionflags
	self._inline = versionflags & FLAG_INLINE_DATA
	self._generaldelta = versionflags & FLAG_GENERALDELTA
	flags = versionflags & ~0xFFFF			flags = versionflags & ~0xFFFF
	fmt = versionflags & 0xFFFF			fmt = versionflags & 0xFFFF

	if fmt == REVLOGV0:			if fmt == REVLOGV0:
	if flags:			if flags:
	raise error.RevlogError(_('unknown flags (%#04x) in version %d '			raise error.RevlogError(_('unknown flags (%#04x) in version %d '
	'revlog %s') %			'revlog %s') %
	(flags >> 16, fmt, self.indexfile))			(flags >> 16, fmt, self.indexfile))

				self._inline = False
				self._generaldelta = False

	elif fmt == REVLOGV1:			elif fmt == REVLOGV1:
	if flags & ~REVLOGV1_FLAGS:			if flags & ~REVLOGV1_FLAGS:
	raise error.RevlogError(_('unknown flags (%#04x) in version %d '			raise error.RevlogError(_('unknown flags (%#04x) in version %d '
	'revlog %s') %			'revlog %s') %
	(flags >> 16, fmt, self.indexfile))			(flags >> 16, fmt, self.indexfile))

				self._inline = versionflags & FLAG_INLINE_DATA
				self._generaldelta = versionflags & FLAG_GENERALDELTA

	elif fmt == REVLOGV2:			elif fmt == REVLOGV2:
	if flags & ~REVLOGV2_FLAGS:			if flags & ~REVLOGV2_FLAGS:
	raise error.RevlogError(_('unknown flags (%#04x) in version %d '			raise error.RevlogError(_('unknown flags (%#04x) in version %d '
	'revlog %s') %			'revlog %s') %
	(flags >> 16, fmt, self.indexfile))			(flags >> 16, fmt, self.indexfile))

				self._inline = versionflags & FLAG_INLINE_DATA
				# generaldelta implied by version 2 revlogs.
				self._generaldelta = True

	else:			else:
	raise error.RevlogError(_('unknown version (%d) in revlog %s') %			raise error.RevlogError(_('unknown version (%d) in revlog %s') %
	(fmt, self.indexfile))			(fmt, self.indexfile))

	self._storedeltachains = True			self._storedeltachains = True

	self._io = revlogio()			self._io = revlogio()
	if self.version == REVLOGV0:			if self.version == REVLOGV0:

mercurial/revlogutils/constants.py

	)			)

	# revlog header flags			# revlog header flags
	REVLOGV0 = 0			REVLOGV0 = 0
	REVLOGV1 = 1			REVLOGV1 = 1
	# Dummy value until file format is finalized.			# Dummy value until file format is finalized.
	# Reminder: change the bounds check in revlog.__init__ when this is changed.			# Reminder: change the bounds check in revlog.__init__ when this is changed.
	REVLOGV2 = 0xDEAD			REVLOGV2 = 0xDEAD
				# Shared across v1 and v2.
	FLAG_INLINE_DATA = (1 << 16)			FLAG_INLINE_DATA = (1 << 16)
				# Only used by v1, implied by v2.
	FLAG_GENERALDELTA = (1 << 17)			FLAG_GENERALDELTA = (1 << 17)
	REVLOG_DEFAULT_FLAGS = FLAG_INLINE_DATA			REVLOG_DEFAULT_FLAGS = FLAG_INLINE_DATA
	REVLOG_DEFAULT_FORMAT = REVLOGV1			REVLOG_DEFAULT_FORMAT = REVLOGV1
	REVLOG_DEFAULT_VERSION = REVLOG_DEFAULT_FORMAT \| REVLOG_DEFAULT_FLAGS			REVLOG_DEFAULT_VERSION = REVLOG_DEFAULT_FORMAT \| REVLOG_DEFAULT_FLAGS
	REVLOGV1_FLAGS = FLAG_INLINE_DATA \| FLAG_GENERALDELTA			REVLOGV1_FLAGS = FLAG_INLINE_DATA \| FLAG_GENERALDELTA
	REVLOGV2_FLAGS = REVLOGV1_FLAGS			REVLOGV2_FLAGS = FLAG_INLINE_DATA

	# revlog index flags			# revlog index flags

	# For historical reasons, revlog's internal flags were exposed via the			# For historical reasons, revlog's internal flags were exposed via the
	# wire protocol and are even exposed in parts of the storage APIs.			# wire protocol and are even exposed in parts of the storage APIs.

	# revision has censor metadata, must be verified			# revision has censor metadata, must be verified
	REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED			REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED

tests/test-revlog-v2.t

	> [experimental]			> [experimental]
	> revlogv2 = enable-unstable-format-and-corrupt-my-data			> revlogv2 = enable-unstable-format-and-corrupt-my-data
	> EOF			> EOF

	$ hg init empty-repo			$ hg init empty-repo
	$ cd empty-repo			$ cd empty-repo
	$ cat .hg/requires			$ cat .hg/requires
	dotencode			dotencode
	exp-revlogv2.0			exp-revlogv2.1
	fncache			fncache
	sparserevlog			sparserevlog
	store			store

	$ hg log			$ hg log

	Unknown flags to revlog are rejected			Unknown flags to revlog are rejected


	$ hg log			$ hg log
	changeset: 0:96ee1d7354c4			changeset: 0:96ee1d7354c4
	tag: tip			tag: tip
	user: test			user: test
	date: Thu Jan 01 00:00:00 1970 +0000			date: Thu Jan 01 00:00:00 1970 +0000
	summary: initial			summary: initial

	Header written as expected (changelog always disables generaldelta)			Header written as expected

	$ f --hexdump --bytes 4 .hg/store/00changelog.i			$ f --hexdump --bytes 4 .hg/store/00changelog.i
	.hg/store/00changelog.i:			.hg/store/00changelog.i:
	0000: 00 01 de ad \|....\|			0000: 00 01 de ad \|....\|

	$ f --hexdump --bytes 4 .hg/store/data/foo.i			$ f --hexdump --bytes 4 .hg/store/data/foo.i
	.hg/store/data/foo.i:			.hg/store/data/foo.i:
	0000: 00 03 de ad \|....\|			0000: 00 01 de ad \|....\|

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	13157		Jan 10 2019, 7:32 PM	★	★
Diff 2	13193	rHGe7a2cc84dbc00deacaef6d4f13d6334c10a6903e	Jan 9 2019, 8:41 PM	★	★