This is an archive of the discontinued Mercurial Phabricator instance.

Differential D110

sparse: add a requirement when a repository uses sparse (BC)
ClosedPublic

Authored by indygreg on Jul 17 2017, 3:15 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG22371eabb3b1: sparse: add a requirement when a repository uses sparse (BC)
rHG9b2647a7a70a: sparse: add a requirement when a repository uses sparse (BC)
Summary

The presence of a sparse checkout can confuse legacy clients or
clients without sparse enabled for reasons that should be obvious.

This commit introduces a new repository requirement that tracks
whether sparse is enabled. The requirement is added when a sparse
config is activated and removed when the sparse config is reset.

The localrepository constructor has been taught to not open repos
with this requirement unless the sparse feature is enabled. It yields
a more actionable error message than what you would get if the
lockout were handled strictly at the requirements verification phase.
Old clients that aren't sparse aware will see the generic
"repository requires features unknown to this Mercurial" error,
however.

The new requirement has "exp" in its name to reflect the
experimental nature of sparse. There's a chance that the eventual
non-experimental feature won't change significantly and we could
have squatted on the "sparse" requirement without ill effect. If
that happens, we can teach new clients to still recognize the old
name. But I suspect we'll sneak in some BC and we'll want a new
requirement to convey new meaning.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Jul 17 2017, 3:15 PM
martinvonz added a subscriber: martinvonz.Jul 17 2017, 5:37 PM
martinvonz added inline comments.
mercurial/sparse.py
526

It feels like this would be clearer as two method: the old one and a new one that just turns sparse off. That would ideally unlink .hg/sparse, no? The new method would not accept includes, excludes, or profiles, and force also doesn't seem relevant as far as I can tell.

durin42 accepted this revision.Jul 17 2017, 5:49 PM
This revision is now accepted and ready to land.Jul 17 2017, 5:49 PM
indygreg added inline comments.Jul 17 2017, 6:26 PM
mercurial/sparse.py
526

Yes, it would be cleaner to have a dedicated function to turn off sparse. Yes, it would unlink .hg/sparse. I was being a bit lazy when I wrote this patch because I wanted to get something in before the freeze :)

martinvonz added inline comments.Jul 17 2017, 7:03 PM
mercurial/sparse.py
526

I've accepted this version, but please send a follow-up (probably after the freeze), because this got pretty hard to read (relative to what it could be).

Closed by commit rHG9b2647a7a70a: sparse: add a requirement when a repository uses sparse (BC) (authored by indygreg). · Explain WhyJul 18 2017, 2:41 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 277

mercurial/help/internals/requirements.txt

============ ============
Denotes that tree manifests are being used. Tree manifests are Denotes that tree manifests are being used. Tree manifests are
one manifest per directory (as opposed to a single flat manifest). one manifest per directory (as opposed to a single flat manifest).
Support for this requirement was added in Mercurial 3.4 (released Support for this requirement was added in Mercurial 3.4 (released
August 2015). The requirement is currently experimental and is August 2015). The requirement is currently experimental and is
disabled by default. disabled by default.
exp-sparse
==========
The working directory is sparse (only contains a subset of files).
Support for this requirement was added in Mercurial 4.3 (released
August 2017). This requirement and feature are experimental and may
disappear in a future Mercurial release. The requirement will only
be present on repositories that have opted in to a sparse working
directory.

mercurial/localrepo.py

REVLOGV2_REQUIREMENT, REVLOGV2_REQUIREMENT,
} }
_basesupported = supportedformats | { _basesupported = supportedformats | {
'store', 'store',
'fncache', 'fncache',
'shared', 'shared',
'relshared', 'relshared',
'dotencode', 'dotencode',
'exp-sparse',
} }
openerreqs = { openerreqs = {
'revlogv1', 'revlogv1',
'generaldelta', 'generaldelta',
'treemanifest', 'treemanifest',
'manifestv2', 'manifestv2',
} }
if not vfs.exists(): if not vfs.exists():
raise error.RepoError( raise error.RepoError(
_('.hg/sharedpath points to nonexistent directory %s') % s) _('.hg/sharedpath points to nonexistent directory %s') % s)
self.sharedpath = s self.sharedpath = s
except IOError as inst: except IOError as inst:
if inst.errno != errno.ENOENT: if inst.errno != errno.ENOENT:
raise raise
if 'exp-sparse' in self.requirements and not sparse.enabled:
raise error.RepoError(_('repository is using sparse feature but '
'sparse is not enabled; enable the '
'"sparse" extensions to access'))
self.store = store.store( self.store = store.store(
self.requirements, self.sharedpath, vfsmod.vfs) self.requirements, self.sharedpath, vfsmod.vfs)
self.spath = self.store.path self.spath = self.store.path
self.svfs = self.store.vfs self.svfs = self.store.vfs
self.sjoin = self.store.join self.sjoin = self.store.join
self.vfs.createmode = self.store.createmode self.vfs.createmode = self.store.createmode
self.cachevfs = vfsmod.vfs(cachepath) self.cachevfs = vfsmod.vfs(cachepath)
self.cachevfs.createmode = self.store.createmode self.cachevfs.createmode = self.store.createmode

mercurial/sparse.py

from .i18n import _ from .i18n import _
from .node import nullid from .node import nullid
from . import ( from . import (
error, error,
match as matchmod, match as matchmod,
merge as mergemod, merge as mergemod,
pycompat, pycompat,
scmutil,
util, util,
) )
# Whether sparse features are enabled. This variable is intended to be # Whether sparse features are enabled. This variable is intended to be
# temporary to facilitate porting sparse to core. It should eventually be # temporary to facilitate porting sparse to core. It should eventually be
# a per-repo option, possibly a repo requirement. # a per-repo option, possibly a repo requirement.
enabled = False enabled = False
if profiles & set(ctx.files()): if profiles & set(ctx.files()):
origstatus = repo.status() origstatus = repo.status()
origsparsematch = matcher(repo) origsparsematch = matcher(repo)
refreshwdir(repo, origstatus, origsparsematch, force=True) refreshwdir(repo, origstatus, origsparsematch, force=True)
prunetemporaryincludes(repo) prunetemporaryincludes(repo)
def _updateconfigandrefreshwdir(repo, includes, excludes, profiles, def _updateconfigandrefreshwdir(repo, includes, excludes, profiles,
force=False): force=False, removing=False):
martinvonzUnsubmitted
Not Done

It feels like this would be clearer as two method: the old one and a new one that just turns sparse off. That would ideally unlink .hg/sparse, no? The new method would not accept includes, excludes, or profiles, and force also doesn't seem relevant as far as I can tell.

martinvonz: It feels like this would be clearer as two method: the old one and a new one that just turns…
indygregAuthorUnsubmitted
Not Done

Yes, it would be cleaner to have a dedicated function to turn off sparse. Yes, it would unlink .hg/sparse. I was being a bit lazy when I wrote this patch because I wanted to get something in before the freeze :)

indygreg: Yes, it would be cleaner to have a dedicated function to turn off sparse. Yes, it would unlink .
martinvonzUnsubmitted
Not Done

I've accepted this version, but please send a follow-up (probably after the freeze), because this got pretty hard to read (relative to what it could be).

martinvonz: I've accepted this version, but please send a follow-up (probably after the freeze), because…
"""Update the sparse config and working directory state.""" """Update the sparse config and working directory state."""
raw = repo.vfs.tryread('sparse') raw = repo.vfs.tryread('sparse')
oldincludes, oldexcludes, oldprofiles = parseconfig(repo.ui, raw) oldincludes, oldexcludes, oldprofiles = parseconfig(repo.ui, raw)
oldstatus = repo.status() oldstatus = repo.status()
oldmatch = matcher(repo) oldmatch = matcher(repo)
oldrequires = set(repo.requirements)
# TODO remove this try..except once the matcher integrates better # TODO remove this try..except once the matcher integrates better
# with dirstate. We currently have to write the updated config # with dirstate. We currently have to write the updated config
# because that will invalidate the matcher cache and force a # because that will invalidate the matcher cache and force a
# re-read. We ideally want to update the cached matcher on the # re-read. We ideally want to update the cached matcher on the
# repo instance then flush the new config to disk once wdir is # repo instance then flush the new config to disk once wdir is
# updated. But this requires massive rework to matcher() and its # updated. But this requires massive rework to matcher() and its
# consumers. # consumers.
writeconfig(repo, includes, excludes, profiles) if 'exp-sparse' in oldrequires and removing:
repo.requirements.discard('exp-sparse')
scmutil.writerequires(repo.vfs, repo.requirements)
elif 'exp-sparse' not in oldrequires:
repo.requirements.add('exp-sparse')
scmutil.writerequires(repo.vfs, repo.requirements)
try: try:
writeconfig(repo, includes, excludes, profiles)
return refreshwdir(repo, oldstatus, oldmatch, force=force) return refreshwdir(repo, oldstatus, oldmatch, force=force)
except Exception: except Exception:
if repo.requirements != oldrequires:
repo.requirements.clear()
repo.requirements |= oldrequires
scmutil.writerequires(repo.vfs, repo.requirements)
writeconfig(repo, oldincludes, oldexcludes, oldprofiles) writeconfig(repo, oldincludes, oldexcludes, oldprofiles)
raise raise
def clearrules(repo, force=False): def clearrules(repo, force=False):
"""Clears include/exclude rules from the sparse config. """Clears include/exclude rules from the sparse config.
The remaining sparse config only has profiles, if defined. The working The remaining sparse config only has profiles, if defined. The working
directory is refreshed, as needed. directory is refreshed, as needed.
profilecount = (len(newprofiles - oldprofiles) - profilecount = (len(newprofiles - oldprofiles) -
len(oldprofiles - newprofiles)) len(oldprofiles - newprofiles))
includecount = (len(newinclude - oldinclude) - includecount = (len(newinclude - oldinclude) -
len(oldinclude - newinclude)) len(oldinclude - newinclude))
excludecount = (len(newexclude - oldexclude) - excludecount = (len(newexclude - oldexclude) -
len(oldexclude - newexclude)) len(oldexclude - newexclude))
fcounts = map(len, _updateconfigandrefreshwdir( fcounts = map(len, _updateconfigandrefreshwdir(
repo, newinclude, newexclude, newprofiles, force=force)) repo, newinclude, newexclude, newprofiles, force=force,
removing=reset))
printchanges(repo.ui, opts, profilecount, includecount, printchanges(repo.ui, opts, profilecount, includecount,
excludecount, *fcounts) excludecount, *fcounts)
def printchanges(ui, opts, profilecount=0, includecount=0, excludecount=0, def printchanges(ui, opts, profilecount=0, includecount=0, excludecount=0,
added=0, dropped=0, conflicting=0): added=0, dropped=0, conflicting=0):
"""Print output summarizing sparse config changes.""" """Print output summarizing sparse config changes."""
with ui.formatter('sparse', opts) as fm: with ui.formatter('sparse', opts) as fm:

tests/test-sparse-requirement.t

  • This file was added.
$ hg init repo
$ cd repo
$ touch a.html b.html c.py d.py
$ cat > frontend.sparse << EOF
> [include]
> *.html
> EOF
$ hg -q commit -A -m initial
$ echo 1 > a.html
$ echo 1 > c.py
$ hg commit -m 'commit 1'
Enable sparse profile
$ cat .hg/requires
dotencode
fncache
generaldelta
revlogv1
store
$ hg debugsparse --config extensions.sparse= --enable-profile frontend.sparse
$ ls
a.html
b.html
Requirement for sparse added when sparse is enabled
$ cat .hg/requires
dotencode
exp-sparse
fncache
generaldelta
revlogv1
store
Client without sparse enabled reacts properly
$ hg files
abort: repository is using sparse feature but sparse is not enabled; enable the "sparse" extensions to access!
[255]
Requirement for sparse is removed when sparse is disabled
$ hg debugsparse --reset --config extensions.sparse=
$ cat .hg/requires
dotencode
fncache
generaldelta
revlogv1
store
And client without sparse can access
$ hg files
a.html
b.html
c.py
d.py
frontend.sparse