This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4521

narrow: add narrow and ellipses as server capabilities
ClosedPublic

Authored by pulkit on Sep 10 2018, 11:13 AM.

Details

Reviewers
durin42
martinvonz
Group Reviewers
hg-reviewers
Commits
rHGc90514043eaa: narrow: add narrow and ellipses as server capabilities
Summary

Right now we don't have a way to differentiate between whether a server can
serve ellipsis or not. The way we check whether a server is narrow enabled is by
checking bundle2 capability which does not scale outside of bundle2 world.

The goal is to use have wireprotocol commands just like remotefilelog for
widening the narrow clone, atleast in non-ellipses cases. Having a server
capability will help there as we can't rely on bundle2 capability there.

Also having a server capability is neat than having a bundle2 capability.

There are lot of things we can optimize locally on the client side by knowing
before hand that whether the server supports ellipses or not.

This will also help us in making sure that a client ellipses repo does not
communicate with a server repo without ellipses.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

pulkit created this revision.Sep 10 2018, 11:13 AM
martinvonz added a subscriber: martinvonz.Sep 10 2018, 1:47 PM

checking bundle2 capability which does not scale outside of bundle2 world.

Is your goal to use narrowness without ellipsis nodes with a changegroup1 server? Can we split this patch up so it just splits up the "narrow" capability into "narrow" and "ellipsis" and then we can consider cg1 support in a separate patch?

hgext/narrow/narrowbundle2.py
35

I'd rather not include "narrow" in the name. It doesn't necessarily require narrowness. You can imagine using ellipsis nodes to achieve a shallow clone that still has all paths.

martinvonz requested changes to this revision.Sep 10 2018, 1:48 PM
This revision now requires changes to proceed.Sep 10 2018, 1:48 PM
pulkit added a comment.Sep 10 2018, 2:16 PM
In D4521#69005, @martinvonz wrote:

checking bundle2 capability which does not scale outside of bundle2 world.

Is your goal to use narrowness without ellipsis nodes with a changegroup1 server? Can we split this patch up so it just splits up the "narrow" capability into "narrow" and "ellipsis" and then we can consider cg1 support in a separate patch?

No, that's not a goal. The goal is to make a wireprotocol command and prevent using pull to exchange data atleast for non-ellipsis cases.

martinvonz added a comment.Sep 10 2018, 5:06 PM
In D4521#69012, @pulkit wrote:
In D4521#69005, @martinvonz wrote:

checking bundle2 capability which does not scale outside of bundle2 world.

Is your goal to use narrowness without ellipsis nodes with a changegroup1 server? Can we split this patch up so it just splits up the "narrow" capability into "narrow" and "ellipsis" and then we can consider cg1 support in a separate patch?

No, that's not a goal. The goal is to make a wireprotocol command and prevent using pull to exchange data atleast for non-ellipsis cases.

Ah, that makes sense. I don't know why we made it a bundle2 capability. Could you update the commit message to clarify that the point is that we want the capability to apply to other commands?

Also note that I left a question for you about NELLIPSESCAP.

pulkit edited the summary of this revision. (Show Details)Sep 11 2018, 5:56 AM
pulkit updated this revision to Diff 10875.
martinvonz accepted this revision.Sep 11 2018, 11:47 AM
martinvonz added inline comments.
hgext/narrow/narrowbundle2.py
51

Should this perhaps be ELLIPSISCAP? I don't think the bundle is currently affected by the narrowness. OTOH, perhaps there should be some "narrowspec" part in the bundle to indicate which paths it contains? Either way, that can be changed in a later patch (if at all).

This revision is now accepted and ready to land.Sep 11 2018, 11:47 AM
Closed by commit rHGc90514043eaa: narrow: add narrow and ellipses as server capabilities (authored by pulkit). · Explain WhySep 11 2018, 1:17 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 10880

hgext/narrow/narrowbundle2.py

util, util,
wireprototypes, wireprototypes,
) )
from mercurial.utils import ( from mercurial.utils import (
stringutil, stringutil,
) )
NARROWCAP = 'narrow' NARROWCAP = 'narrow'
ELLIPSESCAP = 'ellipses'
martinvonzUnsubmitted
Not Done

I'd rather not include "narrow" in the name. It doesn't necessarily require narrowness. You can imagine using ellipsis nodes to achieve a shallow clone that still has all paths.

martinvonz: I'd rather not include "narrow" in the name. It doesn't necessarily require narrowness. You can…
_NARROWACL_SECTION = 'narrowhgacl' _NARROWACL_SECTION = 'narrowhgacl'
_CHANGESPECPART = NARROWCAP + ':changespec' _CHANGESPECPART = NARROWCAP + ':changespec'
_SPECPART = NARROWCAP + ':spec' _SPECPART = NARROWCAP + ':spec'
_SPECPART_INCLUDE = 'include' _SPECPART_INCLUDE = 'include'
_SPECPART_EXCLUDE = 'exclude' _SPECPART_EXCLUDE = 'exclude'
_KILLNODESIGNAL = 'KILL' _KILLNODESIGNAL = 'KILL'
_DONESIGNAL = 'DONE' _DONESIGNAL = 'DONE'
_ELIDEDCSHEADER = '>20s20s20sl' # cset id, p1, p2, len(text) _ELIDEDCSHEADER = '>20s20s20sl' # cset id, p1, p2, len(text)
_ELIDEDMFHEADER = '>20s20s20s20sl' # manifest id, p1, p2, link id, len(text) _ELIDEDMFHEADER = '>20s20s20s20sl' # manifest id, p1, p2, link id, len(text)
_CSHEADERSIZE = struct.calcsize(_ELIDEDCSHEADER) _CSHEADERSIZE = struct.calcsize(_ELIDEDCSHEADER)
_MFHEADERSIZE = struct.calcsize(_ELIDEDMFHEADER) _MFHEADERSIZE = struct.calcsize(_ELIDEDMFHEADER)
# When advertising capabilities, always include narrow clone support. # When advertising capabilities, always include narrow clone support.
def getrepocaps_narrow(orig, repo, **kwargs): def getrepocaps_narrow(orig, repo, **kwargs):
caps = orig(repo, **kwargs) caps = orig(repo, **kwargs)
caps[NARROWCAP] = ['v0'] caps[NARROWCAP] = ['v0']
martinvonzUnsubmitted
Not Done

Should this perhaps be ELLIPSISCAP? I don't think the bundle is currently affected by the narrowness. OTOH, perhaps there should be some "narrowspec" part in the bundle to indicate which paths it contains? Either way, that can be changed in a later patch (if at all).

martinvonz: Should this perhaps be ELLIPSISCAP? I don't think the bundle is currently affected by the…
return caps return caps
def getbundlechangegrouppart_widen(bundler, repo, source, bundlecaps=None, def getbundlechangegrouppart_widen(bundler, repo, source, bundlecaps=None,
b2caps=None, heads=None, common=None, b2caps=None, heads=None, common=None,
**kwargs): **kwargs):
"""Handling changegroup changegroup generation on the server when user """Handling changegroup changegroup generation on the server when user
is widening their narrowspec""" is widening their narrowspec"""

hgext/narrow/narrowwirepeer.py

# narrowwirepeer.py - passes narrow spec with unbundle command # narrowwirepeer.py - passes narrow spec with unbundle command
# #
# Copyright 2017 Google, Inc. # Copyright 2017 Google, Inc.
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
from mercurial.i18n import _ from mercurial.i18n import _
from mercurial import ( from mercurial import (
error, error,
extensions, extensions,
hg, hg,
narrowspec, narrowspec,
node, node,
wireprotov1server,
) )
from . import narrowbundle2
def uisetup(): def uisetup():
def peersetup(ui, peer): def peersetup(ui, peer):
# We must set up the expansion before reposetup below, since it's used # We must set up the expansion before reposetup below, since it's used
# at clone time before we have a repo. # at clone time before we have a repo.
class expandingpeer(peer.__class__): class expandingpeer(peer.__class__):
def expandnarrow(self, narrow_include, narrow_exclude, nodes): def expandnarrow(self, narrow_include, narrow_exclude, nodes):
ui.status(_("expanding narrowspec\n")) ui.status(_("expanding narrowspec\n"))
if not self.capable('exp-expandnarrow'): if not self.capable('exp-expandnarrow'):
raise error.Abort( raise error.Abort(
'peer does not support expanding narrowspecs') 'peer does not support expanding narrowspecs')
hex_nodes = (node.hex(n) for n in nodes) hex_nodes = (node.hex(n) for n in nodes)
new_narrowspec = self._call( new_narrowspec = self._call(
'expandnarrow', 'expandnarrow',
includepats=','.join(narrow_include), includepats=','.join(narrow_include),
excludepats=','.join(narrow_exclude), excludepats=','.join(narrow_exclude),
nodes=','.join(hex_nodes)) nodes=','.join(hex_nodes))
return narrowspec.parseserverpatterns(new_narrowspec) return narrowspec.parseserverpatterns(new_narrowspec)
peer.__class__ = expandingpeer peer.__class__ = expandingpeer
hg.wirepeersetupfuncs.append(peersetup) hg.wirepeersetupfuncs.append(peersetup)
extensions.wrapfunction(wireprotov1server, '_capabilities', addnarrowcap)
def addnarrowcap(orig, repo, proto):
"""add the narrow capability to the server"""
caps = orig(repo, proto)
caps.append(narrowbundle2.NARROWCAP)
if repo.ui.configbool('experimental', 'narrowservebrokenellipses'):
caps.append(narrowbundle2.ELLIPSESCAP)
return caps
def reposetup(repo): def reposetup(repo):
def wirereposetup(ui, peer): def wirereposetup(ui, peer):
def wrapped(orig, cmd, *args, **kwargs): def wrapped(orig, cmd, *args, **kwargs):
if cmd == 'unbundle': if cmd == 'unbundle':
# TODO: don't blindly add include/exclude wireproto # TODO: don't blindly add include/exclude wireproto
# arguments to unbundle. # arguments to unbundle.
include, exclude = repo.narrowpats include, exclude = repo.narrowpats
kwargs[r"includepats"] = ','.join(include) kwargs[r"includepats"] = ','.join(include)
kwargs[r"excludepats"] = ','.join(exclude) kwargs[r"excludepats"] = ','.join(exclude)
return orig(cmd, *args, **kwargs) return orig(cmd, *args, **kwargs)
extensions.wrapfunction(peer, '_calltwowaystream', wrapped) extensions.wrapfunction(peer, '_calltwowaystream', wrapped)
hg.wirepeersetupfuncs.append(wirereposetup) hg.wirepeersetupfuncs.append(wirereposetup)

tests/test-narrow-clone-non-narrow-server.t

> else: > else:
> import urllib > import urllib
> unquote = urllib.unquote_plus > unquote = urllib.unquote_plus
> print(unquote(list(sys.stdin)[1])) > print(unquote(list(sys.stdin)[1]))
> EOF > EOF
$ echo hello | hg -R . serve --stdio | \ $ echo hello | hg -R . serve --stdio | \
> $PYTHON unquote.py | grep narrow > $PYTHON unquote.py | grep narrow
narrow=v0 narrow=v0
rev-branch-cache changegroupsubset getbundle known lookup narrow protocaps pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
$ cd .. $ cd ..
$ hg clone --narrow --include f1 http://localhost:$HGPORT1/ narrowclone $ hg clone --narrow --include f1 http://localhost:$HGPORT1/ narrowclone
requesting all changes requesting all changes
abort: server doesn't support narrow clones abort: server doesn't support narrow clones
[255] [255]