This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5565

revlog: add version 2 format flag to control sparse
Changes PlannedPublic

Authored by indygreg on Jan 10 2019, 7:32 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

Revlog behavior should live as close to the revlog as possible.
In version 1 revlogs, we engaged sparse semantics by using a
repository requirement. This constituted action at a distance
and wasn't optimal.

This commit introduces a feature flag on version 2 revlogs that
explicitly denotes whether sparse reading semantics are enabled.

The flag is enabled by default for version 2 revlogs.

In order to get the flag enabled by default, we had to tweak
localrepo's logic for populating opener options to only set
"sparse-revlog" if the requirement was present. This lets the
revlog initializer use sane defaults depending on the revlog
version. (The interaction between opener options and the revlog
class is kinda wonky and could use a re-think. But my little
brain isn't prepared to handle that at this time.)

I'm also a bit unsure why we have 2 attributes controlling
sparse behavior (revlog._sparserevlog and revlog._withsparseread).
I *think* they can be consolidated. But I'm not sure. The new
revlog feature flag implies both, which seems reasonable.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Jan 10 2019, 7:32 PM
lothiraldan added a subscriber: lothiraldan.Jan 11 2019, 5:06 AM

Revlog behavior should live as close to the revlog as possible.
In version 1 revlogs, we engaged sparse semantics by using a
repository requirement. This constituted action at a distance
and wasn't optimal.
This commit introduces a feature flag on version 2 revlogs that
explicitly denotes whether sparse reading semantics are enabled.

Why have a flag for revlog v2 at all? Since v2 have not been "released"
yet, why not use sparse-revlog logic in all case when using revlog v2?

The flag is enabled by default for version 2 revlogs.
In order to get the flag enabled by default, we had to tweak
localrepo's logic for populating opener options to only set
"sparse-revlog" if the requirement was present. This lets the
revlog initializer use sane defaults depending on the revlog
version. (The interaction between opener options and the revlog
class is kinda wonky and could use a re-think. But my little
brain isn't prepared to handle that at this time.)
I'm also a bit unsure why we have 2 attributes controlling
sparse behavior (revlog._sparserevlog and revlog._withsparseread).
I *think* they can be consolidated. But I'm not sure. The new
revlog feature flag implies both, which seems reasonable.

The revlog._sparserevlog flag enables sparse writing (and reading),
writing sparse revlog requires readers to use sparse-reading and is
gated behind a new requirement (introduced in hg 4.7).

Sparse-reading can be used for non-sparse repositories, it is just a
reader thing.

indygreg planned changes to this revision.Jan 11 2019, 6:11 PM
In D5565#82224, @lothiraldan wrote:

Why have a flag for revlog v2 at all? Since v2 have not been "released"
yet, why not use sparse-revlog logic in all case when using revlog v2?

That's a good point and I'm glad you weighed in on this. I meant to ask that question in the commit message!

If we don't have a good reason to disable sparse revlogs, I'd be OK removing the feature flag and implying the feature by default. I'll update this commit accordingly unless others object to making is always on.

The revlog._sparserevlog flag enables sparse writing (and reading),
writing sparse revlog requires readers to use sparse-reading and is
gated behind a new requirement (introduced in hg 4.7).
Sparse-reading can be used for non-sparse repositories, it is just a
reader thing.

Thank you for clarifying this! It might be useful to have this better documented in the code...

Revision Contents
Changeset List

Diff 13161

mercurial/help/internals/revlogs.txt

Store revision data inline. Store revision data inline.
1 1
Generaldelta encoding. Generaldelta encoding.
Version 2 revlogs have the following flags at the specified bit offsets: Version 2 revlogs have the following flags at the specified bit offsets:
0 0
Store revision data inline. Store revision data inline.
1
Sparse delta chains. When set, delta chains (see below) may cover
long spans in the revlog and readers should perform sparse reads to
minimize the number of bytes that must be read to resolve a revision.
Historically, revlogs limited the total linear span between a revision's
first and last delta and readers performed a read of all the bytes
in between.
The following header values are common: The following header values are common:
00 00 00 01 00 00 00 01
v1 v1
00 01 00 01 00 01 00 01
v1 + inline v1 + inline
00 02 00 01 00 02 00 01
(In development. Format not finalized or stable.) (In development. Format not finalized or stable.)
Version 2 is identical to version 2 with the following differences. Version 2 is identical to version 2 with the following differences.
There is no dedicated *generaldelta* revlog format flag. Instead, There is no dedicated *generaldelta* revlog format flag. Instead,
the feature is implied enabled by default. the feature is implied enabled by default.
There is a revlog format flag that indicates whether sparse delta chains
and reading should be used.
Delta Chains Delta Chains
============ ============
Revision data is encoded as a chain of *chunks*. Each chain begins with Revision data is encoded as a chain of *chunks*. Each chain begins with
the compressed original full text for that revision. Each subsequent the compressed original full text for that revision. Each subsequent
*chunk* is a *delta* against the previous revision. We therefore call *chunk* is a *delta* against the previous revision. We therefore call
these chains of chunks/deltas *delta chains*. these chains of chunks/deltas *delta chains*.

mercurial/localrepo.py

outgoing = discovery.outgoing(self._repo, missingroots=bases, outgoing = discovery.outgoing(self._repo, missingroots=bases,
missingheads=heads) missingheads=heads)
return changegroup.makechangegroup(self._repo, outgoing, '01', source) return changegroup.makechangegroup(self._repo, outgoing, '01', source)
# End of baselegacywirecommands interface. # End of baselegacywirecommands interface.
# Increment the sub-version when the revlog v2 format changes to lock out old # Increment the sub-version when the revlog v2 format changes to lock out old
# clients. # clients.
REVLOGV2_REQUIREMENT = 'exp-revlogv2.1' REVLOGV2_REQUIREMENT = 'exp-revlogv2.2'
# A repository with the sparserevlog feature will have delta chains that # A repository with the sparserevlog feature will have delta chains that
# can spread over a larger span. Sparse reading cuts these large spans into # can spread over a larger span. Sparse reading cuts these large spans into
# pieces, so that each piece isn't too big. # pieces, so that each piece isn't too big.
# Without the sparserevlog capability, reading from the repository could use # Without the sparserevlog capability, reading from the repository could use
# huge amounts of memory, because the whole span would be read at once, # huge amounts of memory, because the whole span would be read at once,
# including all the intermediate revisions that aren't pertinent for the chain. # including all the intermediate revisions that aren't pertinent for the chain.
# This is why once a repository has enabled sparse-read, it becomes required. # This is why once a repository has enabled sparse-read, it becomes required.
b'sparse-read.density-threshold')) b'sparse-read.density-threshold'))
srmingapsize = ui.configbytes(b'experimental', srmingapsize = ui.configbytes(b'experimental',
b'sparse-read.min-gap-size') b'sparse-read.min-gap-size')
options[b'with-sparse-read'] = withsparseread options[b'with-sparse-read'] = withsparseread
options[b'sparse-read-density-threshold'] = srdensitythres options[b'sparse-read-density-threshold'] = srdensitythres
options[b'sparse-read-min-gap-size'] = srmingapsize options[b'sparse-read-min-gap-size'] = srmingapsize
sparserevlog = SPARSEREVLOG_REQUIREMENT in requirements sparserevlog = SPARSEREVLOG_REQUIREMENT in requirements
options[b'sparse-revlog'] = sparserevlog
if sparserevlog: if sparserevlog:
options[b'sparse-revlog'] = True
options[b'generaldelta'] = True options[b'generaldelta'] = True
maxchainlen = None maxchainlen = None
if sparserevlog: if sparserevlog:
maxchainlen = revlogconst.SPARSE_REVLOG_MAX_CHAIN_LENGTH maxchainlen = revlogconst.SPARSE_REVLOG_MAX_CHAIN_LENGTH
# experimental config: format.maxchainlen # experimental config: format.maxchainlen
maxchainlen = ui.configint(b'format', b'maxchainlen', maxchainlen) maxchainlen = ui.configint(b'format', b'maxchainlen', maxchainlen)
if maxchainlen is not None: if maxchainlen is not None:

mercurial/revlog.py

wdirhex, wdirhex,
wdirid, wdirid,
wdirrev, wdirrev,
) )
from .i18n import _ from .i18n import _
from .revlogutils.constants import ( from .revlogutils.constants import (
FLAG_GENERALDELTA, FLAG_GENERALDELTA,
FLAG_INLINE_DATA, FLAG_INLINE_DATA,
FLAG_SPARSE_DELTA_CHAINS,
REVIDX_DEFAULT_FLAGS, REVIDX_DEFAULT_FLAGS,
REVIDX_ELLIPSIS, REVIDX_ELLIPSIS,
REVIDX_EXTSTORED, REVIDX_EXTSTORED,
REVIDX_FLAGS_ORDER, REVIDX_FLAGS_ORDER,
REVIDX_ISCENSORED, REVIDX_ISCENSORED,
REVIDX_KNOWN_FLAGS, REVIDX_KNOWN_FLAGS,
REVIDX_RAWTEXT_CHANGING_FLAGS, REVIDX_RAWTEXT_CHANGING_FLAGS,
REVLOGV0, REVLOGV0,
# blanked usage of all the name to prevent pyflakes constraints # blanked usage of all the name to prevent pyflakes constraints
# We need these name available in the module for extensions. # We need these name available in the module for extensions.
REVLOGV0 REVLOGV0
REVLOGV1 REVLOGV1
REVLOGV2 REVLOGV2
FLAG_INLINE_DATA FLAG_INLINE_DATA
FLAG_GENERALDELTA FLAG_GENERALDELTA
FLAG_SPARSE_DELTA_CHAINS
REVLOG_DEFAULT_FLAGS REVLOG_DEFAULT_FLAGS
REVLOG_DEFAULT_FORMAT REVLOG_DEFAULT_FORMAT
REVLOG_DEFAULT_VERSION REVLOG_DEFAULT_VERSION
REVLOGV1_FLAGS REVLOGV1_FLAGS
REVLOGV2_FLAGS REVLOGV2_FLAGS
REVIDX_ISCENSORED REVIDX_ISCENSORED
REVIDX_ELLIPSIS REVIDX_ELLIPSIS
REVIDX_EXTSTORED REVIDX_EXTSTORED
self._loadindex() self._loadindex()
def _loadindex(self): def _loadindex(self):
mmapindexthreshold = None mmapindexthreshold = None
opts = getattr(self.opener, 'options', {}) or {} opts = getattr(self.opener, 'options', {}) or {}
if 'revlogv2' in opts: if 'revlogv2' in opts:
newversionflags = REVLOGV2 | FLAG_INLINE_DATA newversionflags = REVLOGV2 | FLAG_INLINE_DATA
if 'sparse-revlog' not in opts or opts.get('sparse-revlog'):
newversionflags |= FLAG_SPARSE_DELTA_CHAINS
elif 'revlogv1' in opts: elif 'revlogv1' in opts:
newversionflags = REVLOGV1 | FLAG_INLINE_DATA newversionflags = REVLOGV1 | FLAG_INLINE_DATA
if 'generaldelta' in opts: if 'generaldelta' in opts:
newversionflags |= FLAG_GENERALDELTA newversionflags |= FLAG_GENERALDELTA
else: else:
newversionflags = REVLOG_DEFAULT_VERSION newversionflags = REVLOG_DEFAULT_VERSION
if 'chunkcachesize' in opts: if 'chunkcachesize' in opts:
self._chunkcachesize = opts['chunkcachesize'] self._chunkcachesize = opts['chunkcachesize']
if 'maxchainlen' in opts: if 'maxchainlen' in opts:
self._maxchainlen = opts['maxchainlen'] self._maxchainlen = opts['maxchainlen']
if 'deltabothparents' in opts: if 'deltabothparents' in opts:
self._deltabothparents = opts['deltabothparents'] self._deltabothparents = opts['deltabothparents']
self._lazydeltabase = bool(opts.get('lazydeltabase', False)) self._lazydeltabase = bool(opts.get('lazydeltabase', False))
if 'compengine' in opts: if 'compengine' in opts:
self._compengine = opts['compengine'] self._compengine = opts['compengine']
if 'maxdeltachainspan' in opts: if 'maxdeltachainspan' in opts:
self._maxdeltachainspan = opts['maxdeltachainspan'] self._maxdeltachainspan = opts['maxdeltachainspan']
if self._mmaplargeindex and 'mmapindexthreshold' in opts: if self._mmaplargeindex and 'mmapindexthreshold' in opts:
mmapindexthreshold = opts['mmapindexthreshold'] mmapindexthreshold = opts['mmapindexthreshold']
self._sparserevlog = bool(opts.get('sparse-revlog', False))
withsparseread = bool(opts.get('with-sparse-read', False))
# sparse-revlog forces sparse-read
self._withsparseread = self._sparserevlog or withsparseread
if 'sparse-read-density-threshold' in opts: if 'sparse-read-density-threshold' in opts:
self._srdensitythreshold = opts['sparse-read-density-threshold'] self._srdensitythreshold = opts['sparse-read-density-threshold']
if 'sparse-read-min-gap-size' in opts: if 'sparse-read-min-gap-size' in opts:
self._srmingapsize = opts['sparse-read-min-gap-size'] self._srmingapsize = opts['sparse-read-min-gap-size']
if opts.get('enableellipsis'): if opts.get('enableellipsis'):
self._flagprocessors[REVIDX_ELLIPSIS] = ellipsisprocessor self._flagprocessors[REVIDX_ELLIPSIS] = ellipsisprocessor
# revlog v0 doesn't have flag processors # revlog v0 doesn't have flag processors
if fmt == REVLOGV0: if fmt == REVLOGV0:
if flags: if flags:
raise error.RevlogError(_('unknown flags (%#04x) in version %d ' raise error.RevlogError(_('unknown flags (%#04x) in version %d '
'revlog %s') % 'revlog %s') %
(flags >> 16, fmt, self.indexfile)) (flags >> 16, fmt, self.indexfile))
self._inline = False self._inline = False
self._generaldelta = False self._generaldelta = False
self._sparserevlog = False
self._withsparseread = False
elif fmt == REVLOGV1: elif fmt == REVLOGV1:
if flags & ~REVLOGV1_FLAGS: if flags & ~REVLOGV1_FLAGS:
raise error.RevlogError(_('unknown flags (%#04x) in version %d ' raise error.RevlogError(_('unknown flags (%#04x) in version %d '
'revlog %s') % 'revlog %s') %
(flags >> 16, fmt, self.indexfile)) (flags >> 16, fmt, self.indexfile))
self._inline = versionflags & FLAG_INLINE_DATA self._inline = versionflags & FLAG_INLINE_DATA
self._generaldelta = versionflags & FLAG_GENERALDELTA self._generaldelta = versionflags & FLAG_GENERALDELTA
self._sparserevlog = bool(opts.get('sparse-revlog', False))
withsparseread = bool(opts.get('with-sparse-read', False))
# sparse-revlog forces sparse-read
self._withsparseread = self._sparserevlog or withsparseread
elif fmt == REVLOGV2: elif fmt == REVLOGV2:
if flags & ~REVLOGV2_FLAGS: if flags & ~REVLOGV2_FLAGS:
raise error.RevlogError(_('unknown flags (%#04x) in version %d ' raise error.RevlogError(_('unknown flags (%#04x) in version %d '
'revlog %s') % 'revlog %s') %
(flags >> 16, fmt, self.indexfile)) (flags >> 16, fmt, self.indexfile))
self._inline = versionflags & FLAG_INLINE_DATA self._inline = versionflags & FLAG_INLINE_DATA
# generaldelta implied by version 2 revlogs. # generaldelta implied by version 2 revlogs.
self._generaldelta = True self._generaldelta = True
self._sparserevlog = versionflags & FLAG_SPARSE_DELTA_CHAINS
self._withsparseread = versionflags & FLAG_SPARSE_DELTA_CHAINS
else: else:
raise error.RevlogError(_('unknown version (%d) in revlog %s') % raise error.RevlogError(_('unknown version (%d) in revlog %s') %
(fmt, self.indexfile)) (fmt, self.indexfile))
self._storedeltachains = True self._storedeltachains = True
self._io = revlogio() self._io = revlogio()
if self.version == REVLOGV0: if self.version == REVLOGV0:

mercurial/revlogutils/constants.py

REVLOGV1 = 1 REVLOGV1 = 1
# Dummy value until file format is finalized. # Dummy value until file format is finalized.
# Reminder: change the bounds check in revlog.__init__ when this is changed. # Reminder: change the bounds check in revlog.__init__ when this is changed.
REVLOGV2 = 0xDEAD REVLOGV2 = 0xDEAD
# Shared across v1 and v2. # Shared across v1 and v2.
FLAG_INLINE_DATA = (1 << 16) FLAG_INLINE_DATA = (1 << 16)
# Only used by v1, implied by v2. # Only used by v1, implied by v2.
FLAG_GENERALDELTA = (1 << 17) FLAG_GENERALDELTA = (1 << 17)
# Only used by v2.
FLAG_SPARSE_DELTA_CHAINS = (1 << 17)
REVLOG_DEFAULT_FLAGS = FLAG_INLINE_DATA REVLOG_DEFAULT_FLAGS = FLAG_INLINE_DATA
REVLOG_DEFAULT_FORMAT = REVLOGV1 REVLOG_DEFAULT_FORMAT = REVLOGV1
REVLOG_DEFAULT_VERSION = REVLOG_DEFAULT_FORMAT | REVLOG_DEFAULT_FLAGS REVLOG_DEFAULT_VERSION = REVLOG_DEFAULT_FORMAT | REVLOG_DEFAULT_FLAGS
REVLOGV1_FLAGS = FLAG_INLINE_DATA | FLAG_GENERALDELTA REVLOGV1_FLAGS = FLAG_INLINE_DATA | FLAG_GENERALDELTA
REVLOGV2_FLAGS = FLAG_INLINE_DATA REVLOGV2_FLAGS = FLAG_INLINE_DATA | FLAG_SPARSE_DELTA_CHAINS
# revlog index flags # revlog index flags
# For historical reasons, revlog's internal flags were exposed via the # For historical reasons, revlog's internal flags were exposed via the
# wire protocol and are even exposed in parts of the storage APIs. # wire protocol and are even exposed in parts of the storage APIs.
# revision has censor metadata, must be verified # revision has censor metadata, must be verified
REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED

tests/test-revlog-v2.t

> [experimental] > [experimental]
> revlogv2 = enable-unstable-format-and-corrupt-my-data > revlogv2 = enable-unstable-format-and-corrupt-my-data
> EOF > EOF
$ hg init empty-repo $ hg init empty-repo
$ cd empty-repo $ cd empty-repo
$ cat .hg/requires $ cat .hg/requires
dotencode dotencode
exp-revlogv2.1 exp-revlogv2.2
fncache fncache
sparserevlog sparserevlog
store store
$ hg log $ hg log
Unknown flags to revlog are rejected Unknown flags to revlog are rejected
user: test user: test
date: Thu Jan 01 00:00:00 1970 +0000 date: Thu Jan 01 00:00:00 1970 +0000
summary: initial summary: initial
Header written as expected Header written as expected
$ f --hexdump --bytes 4 .hg/store/00changelog.i $ f --hexdump --bytes 4 .hg/store/00changelog.i
.hg/store/00changelog.i: .hg/store/00changelog.i:
0000: 00 01 de ad |....| 0000: 00 03 de ad |....|
$ f --hexdump --bytes 4 .hg/store/data/foo.i $ f --hexdump --bytes 4 .hg/store/data/foo.i
.hg/store/data/foo.i: .hg/store/data/foo.i:
0000: 00 01 de ad |....| 0000: 00 03 de ad |....|