This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4614

narrow: use diffmatcher to send only new filelogs in non-ellipses widening
ClosedPublic

Authored by pulkit on Sep 17 2018, 9:32 AM.

Details

Reviewers
durin42
martinvonz
Group Reviewers
hg-reviewers
Commits
rHGc73c7653dfb9: narrow: use diffmatcher to send only new filelogs in non-ellipses widening
Summary

Before this patch, when we widen a non-ellipses narrow clone, we downloads all
the filelogs matching the resulting new matcher. This is same as the ellipses
case but can be improved because, we don't pull new csets in non-ellipses cases,
we can only download the new added files instead of downloading all the files
which matches the new matcher.

So, we only download files which matches the new matcher but does not matches
the old matcher. There exists a match.differencematcher() which is used here.

This will lead to significant amount of speedup in extending a non-ellipses
narrow copy on large repos because we will download and process only the new
required filelogs.

The tests changes demonstrate that we are downloading now less files.

Thanks to Augie for pointing that functionality of differencematcher exists in
core.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

pulkit created this revision.Sep 17 2018, 9:32 AM
martinvonz accepted this revision.Sep 17 2018, 1:07 PM
martinvonz added a subscriber: martinvonz.

Nice! That was a lot simpler than I expected. I expected some changes would be needed on the client to prevent it from stripping the existing file nodes, but I suppose is still does that and it still works because it re-applies that stripped bundle afterwards.

This revision is now accepted and ready to land.Sep 17 2018, 1:07 PM
Closed by commit rHGc73c7653dfb9: narrow: use diffmatcher to send only new filelogs in non-ellipses widening (authored by pulkit). · Explain WhySep 17 2018, 1:14 PM
This revision was automatically updated to reflect the committed changes.
pulkit added a comment.Sep 17 2018, 1:19 PM
In D4614#70349, @martinvonz wrote:

I expected some changes would be needed on the client to prevent it from stripping the existing file nodes, but I suppose is still does that and it still works because it re-applies that stripped bundle afterwards.

I don't know the details but it works magically ;)

Revision Contents
Changeset List

Diff 11111

hgext/narrow/narrowbundle2.py

nullid, nullid,
) )
from mercurial import ( from mercurial import (
bundle2, bundle2,
changegroup, changegroup,
error, error,
exchange, exchange,
extensions, extensions,
match as matchmod,
narrowspec, narrowspec,
repair, repair,
repository, repository,
util, util,
wireprototypes, wireprototypes,
) )
from mercurial.utils import ( from mercurial.utils import (
stringutil, stringutil,
raise ValueError(_("server does not advertise changegroup version," raise ValueError(_("server does not advertise changegroup version,"
" can't negotiate support for ellipsis nodes")) " can't negotiate support for ellipsis nodes"))
include = sorted(filter(bool, kwargs.get(r'includepats', []))) include = sorted(filter(bool, kwargs.get(r'includepats', [])))
exclude = sorted(filter(bool, kwargs.get(r'excludepats', []))) exclude = sorted(filter(bool, kwargs.get(r'excludepats', [])))
newmatch = narrowspec.match(repo.root, include=include, exclude=exclude) newmatch = narrowspec.match(repo.root, include=include, exclude=exclude)
oldinclude = sorted(filter(bool, kwargs.get(r'oldincludepats', []))) oldinclude = sorted(filter(bool, kwargs.get(r'oldincludepats', [])))
oldexclude = sorted(filter(bool, kwargs.get(r'oldexcludepats', []))) oldexclude = sorted(filter(bool, kwargs.get(r'oldexcludepats', [])))
oldmatch = narrowspec.match(repo.root, include=oldinclude,
exclude=oldexclude)
diffmatch = matchmod.differencematcher(newmatch, oldmatch)
common = set(common or [nullid]) common = set(common or [nullid])
if (oldinclude != include or oldexclude != exclude): if (oldinclude != include or oldexclude != exclude):
common = repo.revs("::%ln", common) common = repo.revs("::%ln", common)
commonnodes = set() commonnodes = set()
cl = repo.changelog cl = repo.changelog
for c in common: for c in common:
commonnodes.add(cl.node(c)) commonnodes.add(cl.node(c))
if commonnodes: if commonnodes:
# XXX: we should only send the filelogs (and treemanifest). user # XXX: we should only send the filelogs (and treemanifest). user
# already has the changelog and manifest # already has the changelog and manifest
packer = changegroup.getbundler(version, repo, packer = changegroup.getbundler(version, repo,
filematcher=newmatch, filematcher=diffmatch,
fullnodes=commonnodes) fullnodes=commonnodes)
cgdata = packer.generate(set([nullid]), list(commonnodes), False, cgdata = packer.generate(set([nullid]), list(commonnodes), False,
source) source)
part = bundler.newpart('changegroup', data=cgdata) part = bundler.newpart('changegroup', data=cgdata)
part.addparam('version', version) part.addparam('version', version)
if 'treemanifest' in repo.requirements: if 'treemanifest' in repo.requirements:
part.addparam('treemanifest', '1') part.addparam('treemanifest', '1')

tests/test-narrow-widen-no-ellipsis.t

$ cd narrow $ cd narrow
$ hg tracked --addinclude widest/f $ hg tracked --addinclude widest/f
comparing with ssh://user@dummy/master comparing with ssh://user@dummy/master
searching for changes searching for changes
no changes found no changes found
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 0 changesets with 1 changes to 2 files added 0 changesets with 1 changes to 1 files
3 local changesets published 3 local changesets published
$ hg tracked $ hg tracked
I path:inside I path:inside
I path:widest/f I path:widest/f
$ cat widest/f $ cat widest/f
widest widest
$ hg tracked --addinclude wider $ hg tracked --addinclude wider
comparing with ssh://user@dummy/master comparing with ssh://user@dummy/master
searching for changes searching for changes
no changes found no changes found
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 0 changesets with 1 changes to 3 files added 0 changesets with 1 changes to 1 files
5 local changesets published 5 local changesets published
$ hg tracked $ hg tracked
I path:inside I path:inside
I path:wider I path:wider
I path:widest/f I path:widest/f
$ hg update 'desc("add widest")' $ hg update 'desc("add widest")'
2 files updated, 0 files merged, 1 files removed, 0 files unresolved 2 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ cat widest/f $ cat widest/f
0: add d0/f 0: add d0/f
$ hg tracked --addinclude d1 $ hg tracked --addinclude d1
comparing with ssh://user@dummy/upstream comparing with ssh://user@dummy/upstream
searching for changes searching for changes
no changes found no changes found
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 0 changesets with 1 changes to 5 files added 0 changesets with 1 changes to 1 files
11 local changesets published 11 local changesets published
$ hg tracked $ hg tracked
I path:d0 I path:d0
I path:d1 I path:d1
I path:d3 I path:d3
I path:d6 I path:d6
I path:d9 I path:d9
$ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n" $ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n"
$ hg bookmarks bookmark $ hg bookmarks bookmark
$ hg --config hooks.pretxnchangegroup.bad=false tracked --addinclude d1 $ hg --config hooks.pretxnchangegroup.bad=false tracked --addinclude d1
comparing with ssh://user@dummy/upstream comparing with ssh://user@dummy/upstream
searching for changes searching for changes
no changes found no changes found
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 0 changesets with 1 changes to 2 files added 0 changesets with 1 changes to 1 files
11 local changesets published 11 local changesets published
$ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n" $ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n"
11: local 11: local
10: add d10/f 10: add d10/f
9: add d9/f 9: add d9/f
8: add d8/f 8: add d8/f
7: add d7/f 7: add d7/f
6: add d6/f 6: add d6/f