This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5561

revlog: always enable generaldelta on version 2 revlogs
ClosedPublic

Authored by indygreg on Jan 10 2019, 7:32 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGe7a2cc84dbc0: revlog: always enable generaldelta on version 2 revlogs
Summary

This commit starts the process of diverging version 2 revlogs
from version 1 revlogs.

generaldelta is a useful feature and has been enabled by
default for ages. I can't think of a good reason why the
feature should be disabled. Yes, it is true changelogs
today don't have generaldelta enabled. But that's because
they don't have delta chains enabled, so generaldelta makes
no sense there.

This commit makes generaldelta always enabled on version 2
revlogs.

As part of this, one-off code in changelog.py mucking with
revlog.version had to be made conditional on the revlog
version, as we don't want to change revlog feature
flags on version 2 revlogs. The fact this code exists
is horrible and stems from revlog options being shared by
the opener. We probably want a better API here. But that can
wait for another patch.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Jan 10 2019, 7:32 PM
yuja added a subscriber: yuja.Jan 12 2019, 10:24 PM
  • if self._initempty:
  • # changelogs don't benefit from generaldelta

+ if self._initempty and (self.version & 0xFFFF == revlog.REVLOGV1):
+ # changelogs don't benefit from generaldelta.

Perhaps, debugdeltachain() and debugrevlog() has to be updated as well.
They peek .version attribute.

Closed by commit rHGe7a2cc84dbc0: revlog: always enable generaldelta on version 2 revlogs (authored by indygreg). · Explain WhyJan 12 2019, 10:27 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 13193

mercurial/changelog.py

indexfile = '00changelog.i.a' indexfile = '00changelog.i.a'
else: else:
indexfile = '00changelog.i' indexfile = '00changelog.i'
datafile = '00changelog.d' datafile = '00changelog.d'
revlog.revlog.__init__(self, opener, indexfile, datafile=datafile, revlog.revlog.__init__(self, opener, indexfile, datafile=datafile,
checkambig=True, mmaplargeindex=True) checkambig=True, mmaplargeindex=True)
if self._initempty: if self._initempty and (self.version & 0xFFFF == revlog.REVLOGV1):
# changelogs don't benefit from generaldelta # changelogs don't benefit from generaldelta.
self.version &= ~revlog.FLAG_GENERALDELTA self.version &= ~revlog.FLAG_GENERALDELTA
self._generaldelta = False self._generaldelta = False
# Delta chains for changelogs tend to be very small because entries # Delta chains for changelogs tend to be very small because entries
# tend to be small and don't delta well with each. So disable delta # tend to be small and don't delta well with each. So disable delta
# chains. # chains.
self._storedeltachains = False self._storedeltachains = False

mercurial/help/internals/revlogs.txt

Store revision data inline. Store revision data inline.
1 1
Generaldelta encoding. Generaldelta encoding.
Version 2 revlogs have the following flags at the specified bit offsets: Version 2 revlogs have the following flags at the specified bit offsets:
0 0
Store revision data inline. Store revision data inline.
1
Generaldelta encoding.
The following header values are common: The following header values are common:
00 00 00 01 00 00 00 01
v1 v1
00 01 00 01 00 01 00 01
v1 + inline v1 + inline
00 02 00 01 00 02 00 01
The first 4 bytes of the revlog are shared between the revlog header The first 4 bytes of the revlog are shared between the revlog header
and the 6 byte absolute offset field from the first revlog entry. and the 6 byte absolute offset field from the first revlog entry.
Version 2 Format Version 2 Format
================ ================
(In development. Format not finalized or stable.) (In development. Format not finalized or stable.)
Version 2 is currently identical to version 1. This will obviously Version 2 is identical to version 2 with the following differences.
change.
There is no dedicated *generaldelta* revlog format flag. Instead,
the feature is implied enabled by default.
Delta Chains Delta Chains
============ ============
Revision data is encoded as a chain of *chunks*. Each chain begins with Revision data is encoded as a chain of *chunks*. Each chain begins with
the compressed original full text for that revision. Each subsequent the compressed original full text for that revision. Each subsequent
*chunk* is a *delta* against the previous revision. We therefore call *chunk* is a *delta* against the previous revision. We therefore call
these chains of chunks/deltas *delta chains*. these chains of chunks/deltas *delta chains*.

mercurial/localrepo.py

outgoing = discovery.outgoing(self._repo, missingroots=bases, outgoing = discovery.outgoing(self._repo, missingroots=bases,
missingheads=heads) missingheads=heads)
return changegroup.makechangegroup(self._repo, outgoing, '01', source) return changegroup.makechangegroup(self._repo, outgoing, '01', source)
# End of baselegacywirecommands interface. # End of baselegacywirecommands interface.
# Increment the sub-version when the revlog v2 format changes to lock out old # Increment the sub-version when the revlog v2 format changes to lock out old
# clients. # clients.
REVLOGV2_REQUIREMENT = 'exp-revlogv2.0' REVLOGV2_REQUIREMENT = 'exp-revlogv2.1'
# A repository with the sparserevlog feature will have delta chains that # A repository with the sparserevlog feature will have delta chains that
# can spread over a larger span. Sparse reading cuts these large spans into # can spread over a larger span. Sparse reading cuts these large spans into
# pieces, so that each piece isn't too big. # pieces, so that each piece isn't too big.
# Without the sparserevlog capability, reading from the repository could use # Without the sparserevlog capability, reading from the repository could use
# huge amounts of memory, because the whole span would be read at once, # huge amounts of memory, because the whole span would be read at once,
# including all the intermediate revisions that aren't pertinent for the chain. # including all the intermediate revisions that aren't pertinent for the chain.
# This is why once a repository has enabled sparse-read, it becomes required. # This is why once a repository has enabled sparse-read, it becomes required.

mercurial/revlog.py

# 2-tuple of file handles being used for active writing. # 2-tuple of file handles being used for active writing.
self._writinghandles = None self._writinghandles = None
mmapindexthreshold = None mmapindexthreshold = None
opts = getattr(opener, 'options', {}) or {} opts = getattr(opener, 'options', {}) or {}
if 'revlogv2' in opts: if 'revlogv2' in opts:
# version 2 revlogs always use generaldelta. versionflags = REVLOGV2 | FLAG_INLINE_DATA
versionflags = REVLOGV2 | FLAG_GENERALDELTA | FLAG_INLINE_DATA
elif 'revlogv1' in opts: elif 'revlogv1' in opts:
versionflags = REVLOGV1 | FLAG_INLINE_DATA versionflags = REVLOGV1 | FLAG_INLINE_DATA
if 'generaldelta' in opts: if 'generaldelta' in opts:
versionflags |= FLAG_GENERALDELTA versionflags |= FLAG_GENERALDELTA
else: else:
versionflags = REVLOG_DEFAULT_VERSION versionflags = REVLOG_DEFAULT_VERSION
if 'chunkcachesize' in opts: if 'chunkcachesize' in opts:
if len(indexdata) > 0: if len(indexdata) > 0:
versionflags = versionformat_unpack(indexdata[:4])[0] versionflags = versionformat_unpack(indexdata[:4])[0]
self._initempty = False self._initempty = False
except IOError as inst: except IOError as inst:
if inst.errno != errno.ENOENT: if inst.errno != errno.ENOENT:
raise raise
self.version = versionflags self.version = versionflags
self._inline = versionflags & FLAG_INLINE_DATA
self._generaldelta = versionflags & FLAG_GENERALDELTA
flags = versionflags & ~0xFFFF flags = versionflags & ~0xFFFF
fmt = versionflags & 0xFFFF fmt = versionflags & 0xFFFF
if fmt == REVLOGV0: if fmt == REVLOGV0:
if flags: if flags:
raise error.RevlogError(_('unknown flags (%#04x) in version %d ' raise error.RevlogError(_('unknown flags (%#04x) in version %d '
'revlog %s') % 'revlog %s') %
(flags >> 16, fmt, self.indexfile)) (flags >> 16, fmt, self.indexfile))
self._inline = False
self._generaldelta = False
elif fmt == REVLOGV1: elif fmt == REVLOGV1:
if flags & ~REVLOGV1_FLAGS: if flags & ~REVLOGV1_FLAGS:
raise error.RevlogError(_('unknown flags (%#04x) in version %d ' raise error.RevlogError(_('unknown flags (%#04x) in version %d '
'revlog %s') % 'revlog %s') %
(flags >> 16, fmt, self.indexfile)) (flags >> 16, fmt, self.indexfile))
self._inline = versionflags & FLAG_INLINE_DATA
self._generaldelta = versionflags & FLAG_GENERALDELTA
elif fmt == REVLOGV2: elif fmt == REVLOGV2:
if flags & ~REVLOGV2_FLAGS: if flags & ~REVLOGV2_FLAGS:
raise error.RevlogError(_('unknown flags (%#04x) in version %d ' raise error.RevlogError(_('unknown flags (%#04x) in version %d '
'revlog %s') % 'revlog %s') %
(flags >> 16, fmt, self.indexfile)) (flags >> 16, fmt, self.indexfile))
self._inline = versionflags & FLAG_INLINE_DATA
# generaldelta implied by version 2 revlogs.
self._generaldelta = True
else: else:
raise error.RevlogError(_('unknown version (%d) in revlog %s') % raise error.RevlogError(_('unknown version (%d) in revlog %s') %
(fmt, self.indexfile)) (fmt, self.indexfile))
self._storedeltachains = True self._storedeltachains = True
self._io = revlogio() self._io = revlogio()
if self.version == REVLOGV0: if self.version == REVLOGV0:

mercurial/revlogutils/constants.py

) )
# revlog header flags # revlog header flags
REVLOGV0 = 0 REVLOGV0 = 0
REVLOGV1 = 1 REVLOGV1 = 1
# Dummy value until file format is finalized. # Dummy value until file format is finalized.
# Reminder: change the bounds check in revlog.__init__ when this is changed. # Reminder: change the bounds check in revlog.__init__ when this is changed.
REVLOGV2 = 0xDEAD REVLOGV2 = 0xDEAD
# Shared across v1 and v2.
FLAG_INLINE_DATA = (1 << 16) FLAG_INLINE_DATA = (1 << 16)
# Only used by v1, implied by v2.
FLAG_GENERALDELTA = (1 << 17) FLAG_GENERALDELTA = (1 << 17)
REVLOG_DEFAULT_FLAGS = FLAG_INLINE_DATA REVLOG_DEFAULT_FLAGS = FLAG_INLINE_DATA
REVLOG_DEFAULT_FORMAT = REVLOGV1 REVLOG_DEFAULT_FORMAT = REVLOGV1
REVLOG_DEFAULT_VERSION = REVLOG_DEFAULT_FORMAT | REVLOG_DEFAULT_FLAGS REVLOG_DEFAULT_VERSION = REVLOG_DEFAULT_FORMAT | REVLOG_DEFAULT_FLAGS
REVLOGV1_FLAGS = FLAG_INLINE_DATA | FLAG_GENERALDELTA REVLOGV1_FLAGS = FLAG_INLINE_DATA | FLAG_GENERALDELTA
REVLOGV2_FLAGS = REVLOGV1_FLAGS REVLOGV2_FLAGS = FLAG_INLINE_DATA
# revlog index flags # revlog index flags
# For historical reasons, revlog's internal flags were exposed via the # For historical reasons, revlog's internal flags were exposed via the
# wire protocol and are even exposed in parts of the storage APIs. # wire protocol and are even exposed in parts of the storage APIs.
# revision has censor metadata, must be verified # revision has censor metadata, must be verified
REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED

tests/test-revlog-v2.t

> [experimental] > [experimental]
> revlogv2 = enable-unstable-format-and-corrupt-my-data > revlogv2 = enable-unstable-format-and-corrupt-my-data
> EOF > EOF
$ hg init empty-repo $ hg init empty-repo
$ cd empty-repo $ cd empty-repo
$ cat .hg/requires $ cat .hg/requires
dotencode dotencode
exp-revlogv2.0 exp-revlogv2.1
fncache fncache
sparserevlog sparserevlog
store store
$ hg log $ hg log
Unknown flags to revlog are rejected Unknown flags to revlog are rejected
$ hg log $ hg log
changeset: 0:96ee1d7354c4 changeset: 0:96ee1d7354c4
tag: tip tag: tip
user: test user: test
date: Thu Jan 01 00:00:00 1970 +0000 date: Thu Jan 01 00:00:00 1970 +0000
summary: initial summary: initial
Header written as expected (changelog always disables generaldelta) Header written as expected
$ f --hexdump --bytes 4 .hg/store/00changelog.i $ f --hexdump --bytes 4 .hg/store/00changelog.i
.hg/store/00changelog.i: .hg/store/00changelog.i:
0000: 00 01 de ad |....| 0000: 00 01 de ad |....|
$ f --hexdump --bytes 4 .hg/store/data/foo.i $ f --hexdump --bytes 4 .hg/store/data/foo.i
.hg/store/data/foo.i: .hg/store/data/foo.i:
0000: 00 03 de ad |....| 0000: 00 01 de ad |....|