This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4931

narrow: only send the narrowspecs back if ACL in play
ClosedPublic

Authored by pulkit on Oct 10 2018, 10:49 AM.

Details

Reviewers
durin42
martinvonz
Group Reviewers
hg-reviewers
Commits
rHGcb516a854bc7: narrow: only send the narrowspecs back if ACL in play
Summary

I am unable to think why we need to send narrowspecs back from the server. The
current state adds a 'narrow:spec' part to each changegroup which is generated
when narrow extension is enabled. So we are sending narrowspecs on pull also.

There is a problem with sending the narrowspecs the way we are doing it right
now. We add include and exclude as parameter of the 'narrow:spec' bundle2 part.
The the len of include or exclude string increase 255 which is obvious while
working on large repos, bundle2 generation code breaks. For more on that refer
issue5952 on bugzilla.

I was thinking why we need to send the narrowspecs back, and deleted the
'narrow:spec' bundle2 part generation code and found that only narrow-acl test
has some failure.

With this patch, we will only send the 'narrow:spec' bundle2 part if ACL is
enabled because the original narrowspecs in those cases can be a subset of
narrowspecs user requested.

There are phase related output change in couple of tests. The output change
shows that we are now dealing in public phases completely. So maybe sending the
narrow:spec bundle2 part was preventing phases being exchanged or phase bundle2
data being applied.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

pulkit created this revision.Oct 10 2018, 10:49 AM
pulkit added a comment.Oct 16 2018, 9:57 AM

@martinvonz Looks like this got missed for discussing in sprint, can you have a look at this please? This is important because once we have narrowspecs greater than 255 length and and don't use ellipses, we hit https://bz.mercurial-scm.org/show_bug.cgi?id=5952

martinvonz added a comment.Oct 16 2018, 3:00 PM
In D4931#76476, @pulkit wrote:

@martinvonz Looks like this got missed for discussing in sprint, can you have a look at this please? This is important because once we have narrowspecs greater than 255 length and and don't use ellipses, we hit https://bz.mercurial-scm.org/show_bug.cgi?id=5952

Maybe we can instead require experimental.httppostargs to be on when using narrow?

indygreg added a subscriber: indygreg.Oct 16 2018, 3:05 PM
In D4931#76568, @martinvonz wrote:
In D4931#76476, @pulkit wrote:

@martinvonz Looks like this got missed for discussing in sprint, can you have a look at this please? This is important because once we have narrowspecs greater than 255 length and and don't use ellipses, we hit https://bz.mercurial-scm.org/show_bug.cgi?id=5952

Maybe we can instead require experimental.httppostargs to be on when using narrow?

The problem is the narrow specs are being transported as a bundle2 part parameter and those have a limit of 255 bytes. The narrow spec needs to be transported as a bundle2 part payload, which can be infinite length. (Or not transferred at all.)

martinvonz added a comment.Oct 16 2018, 3:33 PM
In D4931#76571, @indygreg wrote:
In D4931#76568, @martinvonz wrote:
In D4931#76476, @pulkit wrote:

@martinvonz Looks like this got missed for discussing in sprint, can you have a look at this please? This is important because once we have narrowspecs greater than 255 length and and don't use ellipses, we hit https://bz.mercurial-scm.org/show_bug.cgi?id=5952

Maybe we can instead require experimental.httppostargs to be on when using narrow?

The problem is the narrow specs are being transported as a bundle2 part parameter and those have a limit of 255 bytes. The narrow spec needs to be transported as a bundle2 part payload, which can be infinite length. (Or not transferred at all.)

Oh, I had completely forgotten that we sometimes send that back. This patch looks fine then, I just have one little suggestion.

hgext/narrow/narrowbundle2.py
273

maybe this line belongs on exchange.py:1903?

pulkit updated this revision to Diff 12192.Oct 17 2018, 7:32 AM
martinvonz accepted this revision.Oct 17 2018, 6:28 PM
This revision is now accepted and ready to land.Oct 17 2018, 6:28 PM
Closed by commit rHGcb516a854bc7: narrow: only send the narrowspecs back if ACL in play (authored by pulkit). · Explain WhyOct 17 2018, 6:33 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 11774

hgext/narrow/narrowbundle2.py

getbundleargs['known'] = 'csv' getbundleargs['known'] = 'csv'
# Extend changegroup serving to handle requests from narrow clients. # Extend changegroup serving to handle requests from narrow clients.
origcgfn = exchange.getbundle2partsmapping['changegroup'] origcgfn = exchange.getbundle2partsmapping['changegroup']
def wrappedcgfn(*args, **kwargs): def wrappedcgfn(*args, **kwargs):
repo = args[1] repo = args[1]
if repo.ui.has_section(_NARROWACL_SECTION): if repo.ui.has_section(_NARROWACL_SECTION):
kwargs = exchange.applynarrowacl(repo, kwargs) kwargs = exchange.applynarrowacl(repo, kwargs)
kwargs['narrow_acl'] = True
martinvonzUnsubmitted
Not Done

maybe this line belongs on exchange.py:1903?

martinvonz: maybe this line belongs on exchange.py:1903?
if (kwargs.get(r'narrow', False) and if (kwargs.get(r'narrow', False) and
repo.ui.configbool('experimental', 'narrowservebrokenellipses')): repo.ui.configbool('experimental', 'narrowservebrokenellipses')):
getbundlechangegrouppart_narrow(*args, **kwargs) getbundlechangegrouppart_narrow(*args, **kwargs)
else: else:
origcgfn(*args, **kwargs) origcgfn(*args, **kwargs)
exchange.getbundle2partsmapping['changegroup'] = wrappedcgfn exchange.getbundle2partsmapping['changegroup'] = wrappedcgfn
# Extend changegroup receiver so client can fixup after widen requests. # Extend changegroup receiver so client can fixup after widen requests.
origcghandler = bundle2.parthandlermapping['changegroup'] origcghandler = bundle2.parthandlermapping['changegroup']
def wrappedcghandler(op, inpart): def wrappedcghandler(op, inpart):
origcghandler(op, inpart) origcghandler(op, inpart)
if util.safehasattr(op, '_widen_bundle'): if util.safehasattr(op, '_widen_bundle'):
handlechangegroup_widen(op, inpart) handlechangegroup_widen(op, inpart)
wrappedcghandler.params = origcghandler.params wrappedcghandler.params = origcghandler.params
bundle2.parthandlermapping['changegroup'] = wrappedcghandler bundle2.parthandlermapping['changegroup'] = wrappedcghandler

mercurial/exchange.py

part.addparam('version', version) part.addparam('version', version)
part.addparam('nbchanges', '%d' % len(outgoing.missing), part.addparam('nbchanges', '%d' % len(outgoing.missing),
mandatory=False) mandatory=False)
if 'treemanifest' in repo.requirements: if 'treemanifest' in repo.requirements:
part.addparam('treemanifest', '1') part.addparam('treemanifest', '1')
if kwargs.get(r'narrow', False) and (include or exclude): if (kwargs.get(r'narrow', False) and kwargs.get('narrow_acl', False)
and (include or exclude)):
narrowspecpart = bundler.newpart('narrow:spec') narrowspecpart = bundler.newpart('narrow:spec')
if include: if include:
narrowspecpart.addparam( narrowspecpart.addparam(
'include', '\n'.join(include), mandatory=True) 'include', '\n'.join(include), mandatory=True)
if exclude: if exclude:
narrowspecpart.addparam( narrowspecpart.addparam(
'exclude', '\n'.join(exclude), mandatory=True) 'exclude', '\n'.join(exclude), mandatory=True)

tests/test-narrow-rebase.t

$ hg rebase -d 'desc("modify outside/f1")' $ hg rebase -d 'desc("modify outside/f1")'
rebasing 7:c2f36d04e05d "modify inside/f2" (tip) rebasing 7:c2f36d04e05d "modify inside/f2" (tip)
saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-rebase.hg (glob) saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-rebase.hg (glob)
Rebase interrupts on conflicting changes outside narrow spec Rebase interrupts on conflicting changes outside narrow spec
$ hg update -q 'desc("conflicting outside/f1")' $ hg update -q 'desc("conflicting outside/f1")'
$ hg phase -f -d . $ hg phase -f -d .
no phases changed
$ hg rebase -d 'desc("modify outside/f1")' $ hg rebase -d 'desc("modify outside/f1")'
rebasing 4:707c035aadb6 "conflicting outside/f1" rebasing 4:707c035aadb6 "conflicting outside/f1"
abort: conflict in file 'outside/f1' is outside narrow clone abort: conflict in file 'outside/f1' is outside narrow clone
[255] [255]

tests/test-narrow-widen-no-ellipsis.t

$ hg pull $ hg pull
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 5 changesets with 4 changes to 2 files added 5 changesets with 4 changes to 2 files
new changesets *:* (glob) new changesets *:* (glob)
3 local changesets published
(run 'hg update' to get a working copy) (run 'hg update' to get a working copy)
$ hg update -r 'desc("add wider")' $ hg update -r 'desc("add wider")'
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat widest/f $ cat widest/f
widest v2 widest v2
$ hg update -r 'desc("update inside")' $ hg update -r 'desc("update inside")'