This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4783

narrow: add 'narrowstream' server capability to use streamclones with narrow
AbandonedPublic

Authored by pulkit on Sep 27 2018, 4:27 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

This patch adds a new server capability "narrowstream" which will dictate
whether the server is capable of serving narrow-stream clones or not.

This also add support for getting include and exclude arguments as part of
stream_out wireprotocol call. We don't use those arguments yet, but we will use
in future.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

pulkit created this revision.Sep 27 2018, 4:27 PM
pulkit added a comment.Sep 27 2018, 4:30 PM

This series works on streamclone v1 right now because I ran into that codepath first and also that's what we use right now. I looked into the code for v2 streamclone and once I get narrow clones working for v1, I will make sure we support v2.

indygreg added a subscriber: indygreg.Oct 1 2018, 1:24 PM

Since this feature will require Mercurial 4.8 and since stream clone v2 is enabled by default in 4.8, I have a moderate preference for not changing the stream_out command to support narrow stream clones. i.e. why introduce complexity on that command when it will likely never be used.

mercurial/wireprotov1server.py
550–551

This will need a decodelist().

pulkit added a comment.Jan 29 2019, 6:57 AM

Abandoning this stack as narrow stream clones were implemented using streamclone v2.

pulkit abandoned this revision.Jan 29 2019, 6:58 AM

Revision Contents
Changeset List

Diff 11455

mercurial/configitems.py

default=False, default=False,
) )
coreconfigitem('server', 'preferuncompressed', coreconfigitem('server', 'preferuncompressed',
default=False, default=False,
) )
coreconfigitem('server', 'streamunbundle', coreconfigitem('server', 'streamunbundle',
default=False, default=False,
) )
coreconfigitem('server', 'stream-narrow-clones',
default=False,
)
coreconfigitem('server', 'uncompressed', coreconfigitem('server', 'uncompressed',
default=True, default=True,
) )
coreconfigitem('server', 'uncompressedallowsecret', coreconfigitem('server', 'uncompressedallowsecret',
default=False, default=False,
) )
coreconfigitem('server', 'validate', coreconfigitem('server', 'validate',
default=False, default=False,

mercurial/help/config.txt

If not set (the default), built-in defaults are used. Run If not set (the default), built-in defaults are used. Run
:hg:`debuginstall` to list available compression engines and their :hg:`debuginstall` to list available compression engines and their
default wire protocol priority. default wire protocol priority.
Older Mercurial clients only support zlib compression and this setting Older Mercurial clients only support zlib compression and this setting
has no effect for legacy clients. has no effect for legacy clients.
``stream-narrow-clones``
Whether the server supports streaming narrow clones or not.
(default: False)
``uncompressed`` ``uncompressed``
Whether to allow clients to clone a repository using the Whether to allow clients to clone a repository using the
uncompressed streaming protocol. This transfers about 40% more uncompressed streaming protocol. This transfers about 40% more
data than a regular clone, but uses less memory and CPU on both data than a regular clone, but uses less memory and CPU on both
server and client. Over a LAN (100 Mbps or better) or a very fast server and client. Over a LAN (100 Mbps or better) or a very fast
WAN, an uncompressed streaming clone is a lot faster (~10x) than a WAN, an uncompressed streaming clone is a lot faster (~10x) than a
regular clone. Over most WAN connections (anything slower than regular clone. Over most WAN connections (anything slower than
about 6 Mbps), uncompressed streaming is slower, because of the about 6 Mbps), uncompressed streaming is slower, because of the

mercurial/wireprotov1server.py

# otherwise, add 'streamreqs' detailing our local revlog format # otherwise, add 'streamreqs' detailing our local revlog format
else: else:
caps.append('streamreqs=%s' % ','.join(sorted(requiredformats))) caps.append('streamreqs=%s' % ','.join(sorted(requiredformats)))
if repo.ui.configbool('experimental', 'bundle2-advertise'): if repo.ui.configbool('experimental', 'bundle2-advertise'):
capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role='server')) capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role='server'))
caps.append('bundle2=' + urlreq.quote(capsblob)) caps.append('bundle2=' + urlreq.quote(capsblob))
caps.append('unbundle=%s' % ','.join(bundle2.bundlepriority)) caps.append('unbundle=%s' % ','.join(bundle2.bundlepriority))
if repo.ui.configbool('server', 'stream-narrow-clones'):
caps.append('narrowstream')
return proto.addcapabilities(repo, caps) return proto.addcapabilities(repo, caps)
# If you are writing an extension and consider wrapping this function. Wrap # If you are writing an extension and consider wrapping this function. Wrap
# `_capabilities` instead. # `_capabilities` instead.
@wireprotocommand('capabilities', permission='pull') @wireprotocommand('capabilities', permission='pull')
def capabilities(repo, proto): def capabilities(repo, proto):
caps = _capabilities(repo, proto) caps = _capabilities(repo, proto)
return wireprototypes.bytesresponse(' '.join(sorted(caps))) return wireprototypes.bytesresponse(' '.join(sorted(caps)))
with proto.mayberedirectstdio() as output: with proto.mayberedirectstdio() as output:
r = repo.pushkey(encoding.tolocal(namespace), encoding.tolocal(key), r = repo.pushkey(encoding.tolocal(namespace), encoding.tolocal(key),
encoding.tolocal(old), new) or False encoding.tolocal(old), new) or False
output = output.getvalue() if output else '' output = output.getvalue() if output else ''
return wireprototypes.bytesresponse('%d\n%s' % (int(r), output)) return wireprototypes.bytesresponse('%d\n%s' % (int(r), output))
@wireprotocommand('stream_out', permission='pull') @wireprotocommand('stream_out', '*', permission='pull')
def stream(repo, proto): def stream(repo, proto, args):
'''If the server supports streaming clone, it advertises the "stream" '''If the server supports streaming clone, it advertises the "stream"
capability with a value representing the version and flags of the repo capability with a value representing the version and flags of the repo
it is serving. Client checks to see if it understands the format. it is serving. Client checks to see if it understands the format.
''' '''
includes = None
excludes = None
if 'narrowstream' in _capabilities(repo, proto):
includes = args.get('includes')
excludes = args.get('excludes')
indygregUnsubmitted
Not Done

This will need a decodelist().

indygreg: This will need a `decodelist()`.
return wireprototypes.streamreslegacy( return wireprototypes.streamreslegacy(
streamclone.generatev1wireproto(repo)) streamclone.generatev1wireproto(repo))
@wireprotocommand('unbundle', 'heads', permission='push') @wireprotocommand('unbundle', 'heads', permission='push')
def unbundle(repo, proto, heads): def unbundle(repo, proto, heads):
their_heads = wireprototypes.decodelist(heads) their_heads = wireprototypes.decodelist(heads)
with proto.mayberedirectstdio() as output: with proto.mayberedirectstdio() as output:

tests/test-http-bundle1.t

"GET /?cmd=capabilities HTTP/1.1" 401 - "GET /?cmd=capabilities HTTP/1.1" 401 -
"GET /?cmd=capabilities HTTP/1.1" 200 - "GET /?cmd=capabilities HTTP/1.1" 200 -
"GET /?cmd=lookup HTTP/1.1" 200 - x-hgarg-1:key=tip x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull "GET /?cmd=lookup HTTP/1.1" 200 - x-hgarg-1:key=tip x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull
"GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull
"GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull
"GET /?cmd=capabilities HTTP/1.1" 401 - (no-reposimplestore !) "GET /?cmd=capabilities HTTP/1.1" 401 - (no-reposimplestore !)
"GET /?cmd=capabilities HTTP/1.1" 200 - (no-reposimplestore !) "GET /?cmd=capabilities HTTP/1.1" 200 - (no-reposimplestore !)
"GET /?cmd=branchmap HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !) "GET /?cmd=branchmap HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !)
"GET /?cmd=stream_out HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !) "GET /?cmd=stream_out HTTP/1.1" 200 - x-hgarg-1:excludes=&includes= x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull
"GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !) "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !)
"GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D5fed3813f7f5e1824344fdc9cf8f63bb662c292d x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !) "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D5fed3813f7f5e1824344fdc9cf8f63bb662c292d x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !)
"GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !) "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !)
"GET /?cmd=capabilities HTTP/1.1" 401 - (no-reposimplestore !) "GET /?cmd=capabilities HTTP/1.1" 401 - (no-reposimplestore !)
"GET /?cmd=capabilities HTTP/1.1" 200 - (no-reposimplestore !) "GET /?cmd=capabilities HTTP/1.1" 200 - (no-reposimplestore !)
"GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !) "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (no-reposimplestore !)
"GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull
"GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:common=0000000000000000000000000000000000000000&heads=5fed3813f7f5e1824344fdc9cf8f63bb662c292d x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:common=0000000000000000000000000000000000000000&heads=5fed3813f7f5e1824344fdc9cf8f63bb662c292d x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull