This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4901

narrow: move remaining narrow-limited dirstate walks to core
ClosedPublic

Authored by martinvonz on Oct 5 2018, 6:35 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG1d09ba0d2ed3: narrow: move remaining narrow-limited dirstate walks to core
Summary

In most places we now filter at a higher level (the context object),
but there are few places that relied on the dirstate walk to be
filtered by the narrowspec. The important cases are those used by `hg
add` and hg addremove. This patch updates them to pass in a matcher
instead of relying on the dirstate to do the filtering. The dirstate
filtering is also dropped in narrowdirstate.py.

Not always filtering in the dirstate should be useful for a future `hg
status --include-outside-narrow` option.

These places now end up doing an unrestricted dirstate walk after this
patch:

  • debugfileset
  • perfwalk
  • sparse (but restricted to sparse config)
  • largefiles

I'll let anyone who cares about these cases adapt them to work with
narrow if necessary.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

martinvonz created this revision.Oct 5 2018, 6:35 PM
pulkit added a subscriber: pulkit.Oct 5 2018, 6:51 PM

These places now end up doing an unrestricted dirstate walk after this
patch:
debugfileset
perfwalk
sparse (but restricted to sparse config)
largefiles
I'll let anyone who cares about these cases adapt them to work with
narrow if necessary.

There is a TODO.rst in hgext/narrow I think it's worth adding this thing there. That will help people to realize that it's a know behavior and a TODO.

martinvonz updated this revision to Diff 11726.Oct 5 2018, 6:56 PM
martinvonz added a comment.Oct 5 2018, 6:57 PM
In D4901#73949, @pulkit wrote:

These places now end up doing an unrestricted dirstate walk after this
patch:
debugfileset
perfwalk
sparse (but restricted to sparse config)
largefiles
I'll let anyone who cares about these cases adapt them to work with
narrow if necessary.

There is a TODO.rst in hgext/narrow I think it's worth adding this thing there. That will help people to realize that it's a know behavior and a TODO.

Good point. Done. I'll send some other patches to update that TODO with things I've cleaned up earlier.

Closed by commit rHG1d09ba0d2ed3: narrow: move remaining narrow-limited dirstate walks to core (authored by martinvonz). · Explain WhyOct 6 2018, 1:08 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 11739

hgext/narrow/TODO.rst

Integration with the share extension needs improvement. Right now Integration with the share extension needs improvement. Right now
we've seen some odd bugs. we've seen some odd bugs.
Address commentary in manifest.excludedmanifestrevlog.add - Address commentary in manifest.excludedmanifestrevlog.add -
specifically we should improve the collaboration with core so that specifically we should improve the collaboration with core so that
add() never gets called on an excluded directory and we can improve add() never gets called on an excluded directory and we can improve
the stand-in to raise a ProgrammingError. the stand-in to raise a ProgrammingError.
Reason more completely about rename-filtering logic in Reason more completely about rename-filtering logic in
narrowfilelog. There could be some surprises lurking there. narrowfilelog. There could be some surprises lurking there.
Formally document the narrowspec format. For bonus points, unify with the Formally document the narrowspec format. For bonus points, unify with the
server-specified narrowspec format. server-specified narrowspec format.
narrowrepo.setnarrowpats() or narrowspec.save() need to make sure narrowrepo.setnarrowpats() or narrowspec.save() need to make sure
they're holding the wlock. they're holding the wlock.
The follinwg places do an unrestricted dirstate walk (including files outside the
narrowspec). Some of them should perhaps not do that.
* debugfileset
* perfwalk
* sparse (but restricted to sparse config)
* largefiles

hgext/narrow/narrowdirstate.py

for f in args: for f in args:
if f is not None and not narrowmatch(f) and f not in self: if f is not None and not narrowmatch(f) and f not in self:
raise error.Abort(_("cannot track '%s' - it is outside " + raise error.Abort(_("cannot track '%s' - it is outside " +
"the narrow clone") % f) "the narrow clone") % f)
return fn(self, *args) return fn(self, *args)
return _wrapper return _wrapper
class narrowdirstate(dirstate.__class__): class narrowdirstate(dirstate.__class__):
def walk(self, match, subrepos, unknown, ignored, full=True,
narrowonly=True):
if narrowonly:
match = repo.narrowmatch(match, includeexact=True)
return super(narrowdirstate, self).walk(match, subrepos, unknown,
ignored, full)
# Prevent adding/editing/copying/deleting files that are outside the # Prevent adding/editing/copying/deleting files that are outside the
# sparse checkout # sparse checkout
@_editfunc @_editfunc
def normal(self, *args): def normal(self, *args):
return super(narrowdirstate, self).normal(*args) return super(narrowdirstate, self).normal(*args)
@_editfunc @_editfunc
def add(self, *args): def add(self, *args):

mercurial/cmdutil.py

badfn = lambda x, y: bad.append(x) or match.bad(x, y) badfn = lambda x, y: bad.append(x) or match.bad(x, y)
names = [] names = []
wctx = repo[None] wctx = repo[None]
cca = None cca = None
abort, warn = scmutil.checkportabilityalert(ui) abort, warn = scmutil.checkportabilityalert(ui)
if abort or warn: if abort or warn:
cca = scmutil.casecollisionauditor(ui, abort, repo.dirstate) cca = scmutil.casecollisionauditor(ui, abort, repo.dirstate)
match = repo.narrowmatch(match, includeexact=True)
badmatch = matchmod.badmatch(match, badfn) badmatch = matchmod.badmatch(match, badfn)
dirstate = repo.dirstate dirstate = repo.dirstate
# We don't want to just call wctx.walk here, since it would return a lot of # We don't want to just call wctx.walk here, since it would return a lot of
# clean files, which we aren't interested in and takes time. # clean files, which we aren't interested in and takes time.
for f in sorted(dirstate.walk(badmatch, subrepos=sorted(wctx.substate), for f in sorted(dirstate.walk(badmatch, subrepos=sorted(wctx.substate),
unknown=True, ignored=False, full=False)): unknown=True, ignored=False, full=False)):
exact = match.exact(f) exact = match.exact(f)
if exact or not explicitonly and f not in wctx and repo.wvfs.lexists(f): if exact or not explicitonly and f not in wctx and repo.wvfs.lexists(f):

mercurial/scmutil.py

This is different from dirstate.status because it doesn't care about This is different from dirstate.status because it doesn't care about
whether files are modified or clean.''' whether files are modified or clean.'''
added, unknown, deleted, removed, forgotten = [], [], [], [], [] added, unknown, deleted, removed, forgotten = [], [], [], [], []
audit_path = pathutil.pathauditor(repo.root, cached=True) audit_path = pathutil.pathauditor(repo.root, cached=True)
ctx = repo[None] ctx = repo[None]
dirstate = repo.dirstate dirstate = repo.dirstate
matcher = repo.narrowmatch(matcher, includeexact=True)
walkresults = dirstate.walk(matcher, subrepos=sorted(ctx.substate), walkresults = dirstate.walk(matcher, subrepos=sorted(ctx.substate),
unknown=True, ignored=False, full=False) unknown=True, ignored=False, full=False)
for abs, st in walkresults.iteritems(): for abs, st in walkresults.iteritems():
dstate = dirstate[abs] dstate = dirstate[abs]
if dstate == '?' and audit_path.check(abs): if dstate == '?' and audit_path.check(abs):
unknown.append(abs) unknown.append(abs)
elif dstate != 'r' and not st: elif dstate != 'r' and not st:
deleted.append(abs) deleted.append(abs)