This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5183

narrow: rework logic to check whether we need to widen and narrow
ClosedPublic

Authored by pulkit on Oct 23 2018, 9:52 AM.

Details

Reviewers
durin42
martinvonz
Group Reviewers
hg-reviewers
Commits
rHG30a7d3b6b281: narrow: rework logic to check whether we need to widen and narrow
Summary

This patch reworks logic which calculates whether we need to extend or narrow
our working copy or not.

We filter the addincludes, removeincludes, addexcludes and removeexcludes passed
from user to the actual added and removed includes and excludes. What that means
is a user can pass an already included path as addincludes, a path which is not
included as removeincludes etc. In such situations the old logic use to think we
need to do some work, whereas we don't need to do that work.

In old logic, even if we don't have anything new to include but it believes we
need to call widen, this adds some good amount of work on large repository. A
widen calls involves computing incomming csets, calling the narrow_widen() which
in non-ellipses cases goes through all the set of csets which are available
which can take ~2-3 mins on large repos. Those 2-3 minutes are spend on doing
nothing which a client can prevent by checking is there really anything which
needs to be included.

The tests changes shows that we don't go to the server anymore in such cases
which is nice.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

pulkit created this revision.Oct 23 2018, 9:52 AM
martinvonz accepted this revision.Oct 23 2018, 11:24 AM
martinvonz added inline comments.
hgext/narrow/narrowcommands.py
429

nit: s/extend/widen/? that seems like the natural opposite to (the verb) "narrow"

This revision is now accepted and ready to land.Oct 23 2018, 11:24 AM
pulkit added inline comments.Oct 23 2018, 11:34 AM
hgext/narrow/narrowcommands.py
429

I am fine with anyone, you can fix this in flight or I can send an updated version too.

pulkit updated this revision to Diff 12325.Oct 23 2018, 12:07 PM
Closed by commit rHG30a7d3b6b281: narrow: rework logic to check whether we need to widen and narrow (authored by pulkit). · Explain WhyOct 23 2018, 12:23 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 12327

hgext/narrow/narrowcommands.py

"is not supported in narrowspec")) "is not supported in narrowspec"))
opts['addinclude'].extend(includepats) opts['addinclude'].extend(includepats)
opts['addexclude'].extend(excludepats) opts['addexclude'].extend(excludepats)
addedincludes = narrowspec.parsepatterns(opts['addinclude']) addedincludes = narrowspec.parsepatterns(opts['addinclude'])
removedincludes = narrowspec.parsepatterns(opts['removeinclude']) removedincludes = narrowspec.parsepatterns(opts['removeinclude'])
addedexcludes = narrowspec.parsepatterns(opts['addexclude']) addedexcludes = narrowspec.parsepatterns(opts['addexclude'])
removedexcludes = narrowspec.parsepatterns(opts['removeexclude']) removedexcludes = narrowspec.parsepatterns(opts['removeexclude'])
only_show = not (addedincludes or removedincludes or addedexcludes or
removedexcludes or newrules)
oldincludes, oldexcludes = repo.narrowpats
# filter the user passed additions and deletions into actual additions and
# deletions of excludes and includes
addedincludes = set([i for i in addedincludes if i not in oldincludes])
removedincludes = set([i for i in removedincludes if i in oldincludes])
addedexcludes = set([i for i in addedexcludes if i not in oldexcludes])
removedexcludes = set([i for i in removedexcludes if i in oldexcludes])
widening = addedincludes or removedexcludes widening = addedincludes or removedexcludes
narrowing = removedincludes or addedexcludes narrowing = removedincludes or addedexcludes
only_show = not widening and not narrowing
# Only print the current narrowspec. # Only print the current narrowspec.
if only_show: if only_show:
include, exclude = repo.narrowpats include, exclude = repo.narrowpats
ui.pager('tracked') ui.pager('tracked')
fm = ui.formatter('narrow', opts) fm = ui.formatter('narrow', opts)
for i in sorted(include): for i in sorted(include):
fm.startitem() fm.startitem()
fm.write('status', '%s ', 'I', label='narrow.included') fm.write('status', '%s ', 'I', label='narrow.included')
fm.write('pat', '%s\n', i, label='narrow.included') fm.write('pat', '%s\n', i, label='narrow.included')
for i in sorted(exclude): for i in sorted(exclude):
fm.startitem() fm.startitem()
fm.write('status', '%s ', 'X', label='narrow.excluded') fm.write('status', '%s ', 'X', label='narrow.excluded')
fm.write('pat', '%s\n', i, label='narrow.excluded') fm.write('pat', '%s\n', i, label='narrow.excluded')
fm.end() fm.end()
return 0 return 0
if not widening and not narrowing:
ui.status(_("nothing to widen or narrow\n"))
martinvonzUnsubmitted
Not Done

nit: s/extend/widen/? that seems like the natural opposite to (the verb) "narrow"

martinvonz: nit: s/extend/widen/? that seems like the natural opposite to (the verb) "narrow"
pulkitAuthorUnsubmitted
Not Done

I am fine with anyone, you can fix this in flight or I can send an updated version too.

pulkit: I am fine with anyone, you can fix this in flight or I can send an updated version too.
return 0
with repo.wlock(), repo.lock(): with repo.wlock(), repo.lock():
cmdutil.bailifchanged(repo) cmdutil.bailifchanged(repo)
# Find the revisions we have in common with the remote. These will # Find the revisions we have in common with the remote. These will
# be used for finding local-only changes for narrowing. They will # be used for finding local-only changes for narrowing. They will
# also define the set of revisions to update for widening. # also define the set of revisions to update for widening.
remotepath = ui.expandpath(remotepath or 'default') remotepath = ui.expandpath(remotepath or 'default')
url, branches = hg.parseurl(remotepath) url, branches = hg.parseurl(remotepath)
ui.status(_('comparing with %s\n') % util.hidepassword(url)) ui.status(_('comparing with %s\n') % util.hidepassword(url))
remote = hg.peer(repo, opts, url) remote = hg.peer(repo, opts, url)
# check narrow support before doing anything if widening needs to be # check narrow support before doing anything if widening needs to be
# performed. In future we should also abort if client is ellipses and # performed. In future we should also abort if client is ellipses and
# server does not support ellipses # server does not support ellipses
if widening and wireprototypes.NARROWCAP not in remote.capabilities(): if widening and wireprototypes.NARROWCAP not in remote.capabilities():
raise error.Abort(_("server does not support narrow clones")) raise error.Abort(_("server does not support narrow clones"))
commoninc = discovery.findcommonincoming(repo, remote) commoninc = discovery.findcommonincoming(repo, remote)
oldincludes, oldexcludes = repo.narrowpats
if narrowing: if narrowing:
newincludes = oldincludes - removedincludes newincludes = oldincludes - removedincludes
newexcludes = oldexcludes | addedexcludes newexcludes = oldexcludes | addedexcludes
_narrow(ui, repo, remote, commoninc, oldincludes, oldexcludes, _narrow(ui, repo, remote, commoninc, oldincludes, oldexcludes,
newincludes, newexcludes, newincludes, newexcludes,
opts['force_delete_local_changes']) opts['force_delete_local_changes'])
# _narrow() updated the narrowspec and _widen() below needs to # _narrow() updated the narrowspec and _widen() below needs to
# use the updated values as its base (otherwise removed includes # use the updated values as its base (otherwise removed includes

tests/test-narrow-clone-non-narrow-server.t

new changesets * (glob) new changesets * (glob)
updating to branch default updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd narrowclone $ cd narrowclone
$ hg tracked --addexclude f2 http://localhost:$HGPORT1/ $ hg tracked --addexclude f2 http://localhost:$HGPORT1/
comparing with http://localhost:$HGPORT1/ comparing with http://localhost:$HGPORT1/
searching for changes searching for changes
looking for local changes to affected paths looking for local changes to affected paths
$ hg tracked --addinclude f1 http://localhost:$HGPORT1/ $ hg tracked --addinclude f1 http://localhost:$HGPORT1/
nothing to widen or narrow
$ hg tracked --addinclude f9 http://localhost:$HGPORT1/
comparing with http://localhost:$HGPORT1/ comparing with http://localhost:$HGPORT1/
abort: server does not support narrow clones abort: server does not support narrow clones
[255] [255]

tests/test-narrow-widen-no-ellipsis.t

$ hg id -n $ hg id -n
2 2
Test that extending already included files should not call narrow_widen Test that extending already included files should not call narrow_widen
wireprotocol command wireprotocol command
$ hg tracked --addinclude widest/f $ hg tracked --addinclude widest/f
comparing with ssh://user@dummy/master nothing to widen or narrow
searching for changes
adding changesets
adding manifests
adding file changes
added 0 changesets with 0 changes to 0 files
Pull down the newly added upstream revision. Pull down the newly added upstream revision.
$ hg pull $ hg pull
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests