This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4576

localrepo: extract resolving of opener options to standalone functions
ClosedPublic

Authored by indygreg on Sep 13 2018, 12:29 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGb10d145837bc: localrepo: extract resolving of opener options to standalone functions
Summary

Requirements and config options are converted into a dict which is
available to the store vfs to consult. This is how storage options
are communicated from the repo layer to the storage layer.

Currently, we do that option resolution in a private method on the
repo instance. And there is a single method doing that resolution.

Opener options are logically specific to the storage backend they
apply to. And, opener options may wish to influence how the repo
object/type is constructed. So it makes sense to have more granular
storage option resolution that occurs before the repo object is
instantiated.

This commit extracts the code for resolving opener options into new
module-level functions. These functions are run before the repo
instance is constructed.

As part of the code move, we split the option resolution into
generic and revlog-specific options. After this commit, we no longer
add revlog-specific options to repos that don't have a revlog
requirement.

Some of these opener options and associated config options might make
sense on alternate storage backends. We can always reuse config
options and opener option names for other backends. But we shouldn't
be passing opener options to storage backends that won't recognize
them. I haven't done it here, but after this commit it should be
possible for store backends to validate the set of opener options
it receives.

Because localrepository.openerreqs is no longer used after this commit,
it has been removed.

I'm not super thrilled about the code outside of localrepo that is
adding requirements and updating opener options. We'll probably want
to create a more formal API for that use case that constructs a new
repo instance and poisons the old repo object. But this was a
pre-existing issue and can be dealt with later. I have little doubt
it will cause me troubles as I continue to refactor how repository
objects are instantiated.

.. api::

``localrepository.openerreqs`` has been removed. Override
``localrepo.resolvestorevfsoptions()`` to add custom opener options.

.. api::

``localrepository._applyopenerreqs()`` has been removed. Use
``localrepo.resolvestorevfsoptions()`` to add custom opener options.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Sep 13 2018, 12:29 PM
durin42 added a subscriber: durin42.Sep 18 2018, 5:58 PM

I kind of hope that eventually you're going to break all the requirements-handling nonsense out into its own module somehow?

Closed by commit rHGb10d145837bc: localrepo: extract resolving of opener options to standalone functions (authored by indygreg). · Explain WhySep 18 2018, 6:00 PM
This revision was automatically updated to reflect the committed changes.
indygreg added a comment.Sep 18 2018, 6:12 PM
In D4576#70668, @durin42 wrote:

I kind of hope that eventually you're going to break all the requirements-handling nonsense out into its own module somehow?

I suspect I'm trending in the direction of creating a storage module/package to hold all things storage. I would like to e.g. isolate code for revlog-based repositories from other backends from generic, storage-agnostic code. Right now, it's definitely easier to keep things in localrepo.py until we're closer to that future world. But a few more series like this and we may be close enough...

Revision Contents
Changeset List

Diff 11174

mercurial/bundle2.py

@parthandler('changegroup', ('version', 'nbchanges', 'treemanifest', @parthandler('changegroup', ('version', 'nbchanges', 'treemanifest',
'targetphase')) 'targetphase'))
def handlechangegroup(op, inpart): def handlechangegroup(op, inpart):
"""apply a changegroup part on the repo """apply a changegroup part on the repo
This is a very early implementation that will massive rework before being This is a very early implementation that will massive rework before being
inflicted to any end-user. inflicted to any end-user.
""" """
from . import localrepo
tr = op.gettransaction() tr = op.gettransaction()
unpackerversion = inpart.params.get('version', '01') unpackerversion = inpart.params.get('version', '01')
# We should raise an appropriate exception here # We should raise an appropriate exception here
cg = changegroup.getunbundler(unpackerversion, inpart, None) cg = changegroup.getunbundler(unpackerversion, inpart, None)
# the source and url passed here are overwritten by the one contained in # the source and url passed here are overwritten by the one contained in
# the transaction.hookargs argument. So 'bundle2' is a placeholder # the transaction.hookargs argument. So 'bundle2' is a placeholder
nbchangesets = None nbchangesets = None
if 'nbchanges' in inpart.params: if 'nbchanges' in inpart.params:
nbchangesets = int(inpart.params.get('nbchanges')) nbchangesets = int(inpart.params.get('nbchanges'))
if ('treemanifest' in inpart.params and if ('treemanifest' in inpart.params and
'treemanifest' not in op.repo.requirements): 'treemanifest' not in op.repo.requirements):
if len(op.repo.changelog) != 0: if len(op.repo.changelog) != 0:
raise error.Abort(_( raise error.Abort(_(
"bundle contains tree manifests, but local repo is " "bundle contains tree manifests, but local repo is "
"non-empty and does not use tree manifests")) "non-empty and does not use tree manifests"))
op.repo.requirements.add('treemanifest') op.repo.requirements.add('treemanifest')
op.repo._applyopenerreqs() op.repo.svfs.options = localrepo.resolvestorevfsoptions(
op.repo.ui, op.repo.requirements)
op.repo._writerequirements() op.repo._writerequirements()
extrakwargs = {} extrakwargs = {}
targetphase = inpart.params.get('targetphase') targetphase = inpart.params.get('targetphase')
if targetphase is not None: if targetphase is not None:
extrakwargs[r'targetphase'] = int(targetphase) extrakwargs[r'targetphase'] = int(targetphase)
ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2', ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2',
expectedtotal=nbchangesets, **extrakwargs) expectedtotal=nbchangesets, **extrakwargs)
if op.reply is not None: if op.reply is not None:

mercurial/localrepo.py

storebasepath = hgvfs.base storebasepath = hgvfs.base
cachepath = hgvfs.join(b'cache') cachepath = hgvfs.join(b'cache')
# The store has changed over time and the exact layout is dictated by # The store has changed over time and the exact layout is dictated by
# requirements. The store interface abstracts differences across all # requirements. The store interface abstracts differences across all
# of them. # of them.
store = makestore(requirements, storebasepath, store = makestore(requirements, storebasepath,
lambda base: vfsmod.vfs(base, cacheaudited=True)) lambda base: vfsmod.vfs(base, cacheaudited=True))
hgvfs.createmode = store.createmode hgvfs.createmode = store.createmode
storevfs = store.vfs
storevfs.options = resolvestorevfsoptions(ui, requirements)
# The cache vfs is used to manage cache files. # The cache vfs is used to manage cache files.
cachevfs = vfsmod.vfs(cachepath, cacheaudited=True) cachevfs = vfsmod.vfs(cachepath, cacheaudited=True)
cachevfs.createmode = store.createmode cachevfs.createmode = store.createmode
return localrepository( return localrepository(
baseui=baseui, baseui=baseui,
ui=ui, ui=ui,
origroot=path, origroot=path,
if b'fncache' in requirements: if b'fncache' in requirements:
return storemod.fncachestore(path, vfstype, return storemod.fncachestore(path, vfstype,
b'dotencode' in requirements) b'dotencode' in requirements)
return storemod.encodedstore(path, vfstype) return storemod.encodedstore(path, vfstype)
return storemod.basicstore(path, vfstype) return storemod.basicstore(path, vfstype)
def resolvestorevfsoptions(ui, requirements):
"""Resolve the options to pass to the store vfs opener.
The returned dict is used to influence behavior of the storage layer.
"""
options = {}
if b'treemanifest' in requirements:
options[b'treemanifest'] = True
# experimental config: format.manifestcachesize
manifestcachesize = ui.configint(b'format', b'manifestcachesize')
if manifestcachesize is not None:
options[b'manifestcachesize'] = manifestcachesize
# In the absence of another requirement superseding a revlog-related
# requirement, we have to assume the repo is using revlog version 0.
# This revlog format is super old and we don't bother trying to parse
# opener options for it because those options wouldn't do anything
# meaningful on such old repos.
if b'revlogv1' in requirements or REVLOGV2_REQUIREMENT in requirements:
options.update(resolverevlogstorevfsoptions(ui, requirements))
return options
def resolverevlogstorevfsoptions(ui, requirements):
"""Resolve opener options specific to revlogs."""
options = {}
if b'revlogv1' in requirements:
options[b'revlogv1'] = True
if REVLOGV2_REQUIREMENT in requirements:
options[b'revlogv2'] = True
if b'generaldelta' in requirements:
options[b'generaldelta'] = True
# experimental config: format.chunkcachesize
chunkcachesize = ui.configint(b'format', b'chunkcachesize')
if chunkcachesize is not None:
options[b'chunkcachesize'] = chunkcachesize
deltabothparents = ui.configbool(b'storage',
b'revlog.optimize-delta-parent-choice')
options[b'deltabothparents'] = deltabothparents
options[b'lazydeltabase'] = not scmutil.gddeltaconfig(ui)
chainspan = ui.configbytes(b'experimental', b'maxdeltachainspan')
if 0 <= chainspan:
options[b'maxdeltachainspan'] = chainspan
mmapindexthreshold = ui.configbytes(b'experimental',
b'mmapindexthreshold')
if mmapindexthreshold is not None:
options[b'mmapindexthreshold'] = mmapindexthreshold
withsparseread = ui.configbool(b'experimental', b'sparse-read')
srdensitythres = float(ui.config(b'experimental',
b'sparse-read.density-threshold'))
srmingapsize = ui.configbytes(b'experimental',
b'sparse-read.min-gap-size')
options[b'with-sparse-read'] = withsparseread
options[b'sparse-read-density-threshold'] = srdensitythres
options[b'sparse-read-min-gap-size'] = srmingapsize
sparserevlog = SPARSEREVLOG_REQUIREMENT in requirements
options[b'sparse-revlog'] = sparserevlog
if sparserevlog:
options[b'generaldelta'] = True
maxchainlen = None
if sparserevlog:
maxchainlen = revlogconst.SPARSE_REVLOG_MAX_CHAIN_LENGTH
# experimental config: format.maxchainlen
maxchainlen = ui.configint(b'format', b'maxchainlen', maxchainlen)
if maxchainlen is not None:
options[b'maxchainlen'] = maxchainlen
for r in requirements:
if r.startswith(b'exp-compression-'):
options[b'compengine'] = r[len(b'exp-compression-'):]
return options
@interfaceutil.implementer(repository.completelocalrepository) @interfaceutil.implementer(repository.completelocalrepository)
class localrepository(object): class localrepository(object):
# obsolete experimental requirements: # obsolete experimental requirements:
# - manifestv2: An experimental new manifest format that allowed # - manifestv2: An experimental new manifest format that allowed
# for stem compression of long paths. Experiment ended up not # for stem compression of long paths. Experiment ended up not
# being successful (repository sizes went up due to worse delta # being successful (repository sizes went up due to worse delta
# chains), and the code was deleted in 4.6. # chains), and the code was deleted in 4.6.
supportedformats = { supportedformats = {
'revlogv1', 'revlogv1',
'generaldelta', 'generaldelta',
'treemanifest', 'treemanifest',
REVLOGV2_REQUIREMENT, REVLOGV2_REQUIREMENT,
SPARSEREVLOG_REQUIREMENT, SPARSEREVLOG_REQUIREMENT,
} }
_basesupported = supportedformats | { _basesupported = supportedformats | {
'store', 'store',
'fncache', 'fncache',
'shared', 'shared',
'relshared', 'relshared',
'dotencode', 'dotencode',
'exp-sparse', 'exp-sparse',
'internal-phase' 'internal-phase'
} }
openerreqs = {
'revlogv1',
'generaldelta',
'treemanifest',
}
# list of prefix for file which can be written without 'wlock' # list of prefix for file which can be written without 'wlock'
# Extensions should extend this list when needed # Extensions should extend this list when needed
_wlockfreeprefix = { _wlockfreeprefix = {
# We migh consider requiring 'wlock' for the next # We migh consider requiring 'wlock' for the next
# two, but pretty much all the existing code assume # two, but pretty much all the existing code assume
# wlock is not needed so we keep them excluded for # wlock is not needed so we keep them excluded for
# now. # now.
self.svfs = self.store.vfs self.svfs = self.store.vfs
self.sjoin = self.store.join self.sjoin = self.store.join
if (self.ui.configbool('devel', 'all-warnings') or if (self.ui.configbool('devel', 'all-warnings') or
self.ui.configbool('devel', 'check-locks')): self.ui.configbool('devel', 'check-locks')):
if util.safehasattr(self.svfs, 'vfs'): # this is filtervfs if util.safehasattr(self.svfs, 'vfs'): # this is filtervfs
self.svfs.vfs.audit = self._getsvfsward(self.svfs.vfs.audit) self.svfs.vfs.audit = self._getsvfsward(self.svfs.vfs.audit)
else: # standard vfs else: # standard vfs
self.svfs.audit = self._getsvfsward(self.svfs.audit) self.svfs.audit = self._getsvfsward(self.svfs.audit)
self._applyopenerreqs()
self._dirstatevalidatewarned = False self._dirstatevalidatewarned = False
self._branchcaches = {} self._branchcaches = {}
self._revbranchcache = None self._revbranchcache = None
self._filterpats = {} self._filterpats = {}
self._datafilters = {} self._datafilters = {}
self._transref = self._lockref = self._wlockref = None self._transref = self._lockref = self._wlockref = None
def _restrictcapabilities(self, caps): def _restrictcapabilities(self, caps):
if self.ui.configbool('experimental', 'bundle2-advertise'): if self.ui.configbool('experimental', 'bundle2-advertise'):
caps = set(caps) caps = set(caps)
capsblob = bundle2.encodecaps(bundle2.getrepocaps(self, capsblob = bundle2.encodecaps(bundle2.getrepocaps(self,
role='client')) role='client'))
caps.add('bundle2=' + urlreq.quote(capsblob)) caps.add('bundle2=' + urlreq.quote(capsblob))
return caps return caps
def _applyopenerreqs(self):
self.svfs.options = {r: True for r in self.requirements
if r in self.openerreqs}
# experimental config: format.chunkcachesize
chunkcachesize = self.ui.configint('format', 'chunkcachesize')
if chunkcachesize is not None:
self.svfs.options['chunkcachesize'] = chunkcachesize
# experimental config: format.manifestcachesize
manifestcachesize = self.ui.configint('format', 'manifestcachesize')
if manifestcachesize is not None:
self.svfs.options['manifestcachesize'] = manifestcachesize
deltabothparents = self.ui.configbool('storage',
'revlog.optimize-delta-parent-choice')
self.svfs.options['deltabothparents'] = deltabothparents
self.svfs.options['lazydeltabase'] = not scmutil.gddeltaconfig(self.ui)
chainspan = self.ui.configbytes('experimental', 'maxdeltachainspan')
if 0 <= chainspan:
self.svfs.options['maxdeltachainspan'] = chainspan
mmapindexthreshold = self.ui.configbytes('experimental',
'mmapindexthreshold')
if mmapindexthreshold is not None:
self.svfs.options['mmapindexthreshold'] = mmapindexthreshold
withsparseread = self.ui.configbool('experimental', 'sparse-read')
srdensitythres = float(self.ui.config('experimental',
'sparse-read.density-threshold'))
srmingapsize = self.ui.configbytes('experimental',
'sparse-read.min-gap-size')
self.svfs.options['with-sparse-read'] = withsparseread
self.svfs.options['sparse-read-density-threshold'] = srdensitythres
self.svfs.options['sparse-read-min-gap-size'] = srmingapsize
sparserevlog = SPARSEREVLOG_REQUIREMENT in self.requirements
self.svfs.options['sparse-revlog'] = sparserevlog
if sparserevlog:
self.svfs.options['generaldelta'] = True
maxchainlen = None
if sparserevlog:
maxchainlen = revlogconst.SPARSE_REVLOG_MAX_CHAIN_LENGTH
# experimental config: format.maxchainlen
maxchainlen = self.ui.configint('format', 'maxchainlen', maxchainlen)
if maxchainlen is not None:
self.svfs.options['maxchainlen'] = maxchainlen
for r in self.requirements:
if r.startswith('exp-compression-'):
self.svfs.options['compengine'] = r[len('exp-compression-'):]
# TODO move "revlogv2" to openerreqs once finalized.
if REVLOGV2_REQUIREMENT in self.requirements:
self.svfs.options['revlogv2'] = True
def _writerequirements(self): def _writerequirements(self):
scmutil.writerequires(self.vfs, self.requirements) scmutil.writerequires(self.vfs, self.requirements)
# Don't cache auditor/nofsauditor, or you'll end up with reference cycle: # Don't cache auditor/nofsauditor, or you'll end up with reference cycle:
# self -> auditor -> self._checknested -> self # self -> auditor -> self._checknested -> self
@property @property
def auditor(self): def auditor(self):

mercurial/repository.py

""" """
supportedformats = interfaceutil.Attribute( supportedformats = interfaceutil.Attribute(
"""Set of requirements that apply to stream clone. """Set of requirements that apply to stream clone.
This is actually a class attribute and is shared among all instances. This is actually a class attribute and is shared among all instances.
""") """)
openerreqs = interfaceutil.Attribute(
"""Set of requirements that are passed to the opener.
This is actually a class attribute and is shared among all instances.
""")
supported = interfaceutil.Attribute( supported = interfaceutil.Attribute(
"""Set of requirements that this repo is capable of opening.""") """Set of requirements that this repo is capable of opening.""")
requirements = interfaceutil.Attribute( requirements = interfaceutil.Attribute(
"""Set of requirements this repo uses.""") """Set of requirements this repo uses.""")
filtername = interfaceutil.Attribute( filtername = interfaceutil.Attribute(
"""Name of the repoview that is active on this repo.""") """Name of the repoview that is active on this repo.""")

mercurial/streamclone.py

"""Possibly perform a legacy stream clone operation. """Possibly perform a legacy stream clone operation.
Legacy stream clones are performed as part of pull but before all other Legacy stream clones are performed as part of pull but before all other
operations. operations.
A legacy stream clone will not be performed if a bundle2 stream clone is A legacy stream clone will not be performed if a bundle2 stream clone is
supported. supported.
""" """
from . import localrepo
supported, requirements = canperformstreamclone(pullop) supported, requirements = canperformstreamclone(pullop)
if not supported: if not supported:
return return
repo = pullop.repo repo = pullop.repo
remote = pullop.remote remote = pullop.remote
with repo.lock(): with repo.lock():
consumev1(repo, fp, filecount, bytecount) consumev1(repo, fp, filecount, bytecount)
# new requirements = old non-format requirements + # new requirements = old non-format requirements +
# new format-related remote requirements # new format-related remote requirements
# requirements from the streamed-in repository # requirements from the streamed-in repository
repo.requirements = requirements | ( repo.requirements = requirements | (
repo.requirements - repo.supportedformats) repo.requirements - repo.supportedformats)
repo._applyopenerreqs() repo.svfs.options = localrepo.resolvestorevfsoptions(
repo.ui, repo.requirements)
repo._writerequirements() repo._writerequirements()
if rbranchmap: if rbranchmap:
branchmap.replacecache(repo, rbranchmap) branchmap.replacecache(repo, rbranchmap)
repo.invalidate() repo.invalidate()
def allowservergeneration(repo): def allowservergeneration(repo):
if elapsed <= 0: if elapsed <= 0:
elapsed = 0.001 elapsed = 0.001
repo.ui.status(_('transferred %s in %.1f seconds (%s/sec)\n') % repo.ui.status(_('transferred %s in %.1f seconds (%s/sec)\n') %
(util.bytecount(progress.pos), elapsed, (util.bytecount(progress.pos), elapsed,
util.bytecount(progress.pos / elapsed))) util.bytecount(progress.pos / elapsed)))
progress.complete() progress.complete()
def applybundlev2(repo, fp, filecount, filesize, requirements): def applybundlev2(repo, fp, filecount, filesize, requirements):
from . import localrepo
missingreqs = [r for r in requirements if r not in repo.supported] missingreqs = [r for r in requirements if r not in repo.supported]
if missingreqs: if missingreqs:
raise error.Abort(_('unable to apply stream clone: ' raise error.Abort(_('unable to apply stream clone: '
'unsupported format: %s') % 'unsupported format: %s') %
', '.join(sorted(missingreqs))) ', '.join(sorted(missingreqs)))
consumev2(repo, fp, filecount, filesize) consumev2(repo, fp, filecount, filesize)
# new requirements = old non-format requirements + # new requirements = old non-format requirements +
# new format-related remote requirements # new format-related remote requirements
# requirements from the streamed-in repository # requirements from the streamed-in repository
repo.requirements = set(requirements) | ( repo.requirements = set(requirements) | (
repo.requirements - repo.supportedformats) repo.requirements - repo.supportedformats)
repo._applyopenerreqs() repo.svfs.options = localrepo.resolvestorevfsoptions(
repo.ui, repo.requirements)
repo._writerequirements() repo._writerequirements()