This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3200

wireproto: implement capabilities for wire protocol v2
ClosedPublic

Authored by indygreg on Apr 9 2018, 2:58 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGdf4985497986: wireproto: implement capabilities for wire protocol v2
Summary

The capabilities mechanism for wire protocol version 2 represents a
clean break from version 1.

Instead of effectively exchanging a set of capabilities, we're
exchanging a rich data structure.

This data structure currently contains information about
every available command, including its accepted arguments. It also
contains information about supported compression formats.

Exposing information about supported commands will allow clients
to automatically generate bindings to the server. Clients will be
able to do things like detect when they are attempting to run a
command that isn't known to the server. Exposing the required
permissions to run a command can be used by clients to determine if
they have privileges to call a command before actually calling it.
We could potentially even have clients send credentials
preemptively without waiting for the server to deny the command
request. Lots of potential here.

The data returned by this command will likely evolve heavily. So we
shouldn't bikeshed the implementation just yet.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Apr 9 2018, 2:58 PM
durin42 accepted this revision.Apr 9 2018, 6:27 PM
This revision is now accepted and ready to land.Apr 9 2018, 6:27 PM
Closed by commit rHGdf4985497986: wireproto: implement capabilities for wire protocol v2 (authored by indygreg). · Explain WhyApr 11 2018, 12:34 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 7979

mercurial/help/internals/wireprotocol.txt

Obtain heads in named branches. Obtain heads in named branches.
Receives no arguments. Receives no arguments.
The response is a map with bytestring keys defining the branch name. The response is a map with bytestring keys defining the branch name.
Values are arrays of bytestring defining raw changeset nodes. Values are arrays of bytestring defining raw changeset nodes.
capabilities
------------
Obtain the server's capabilities.
Receives no arguments.
This command is typically called only as part of the handshake during
initial connection establishment.
The response is a map with bytestring keys defining server information.
The defined keys are:
commands
A map defining available wire protocol commands on this server.
Keys in the map are the names of commands that can be invoked. Values
are maps defining information about that command. The bytestring keys
are:
args
An array of argument names accepted by this command.
permissions
An array of permissions required to execute this command.
compression
An array of maps defining available compression format support.
The array is sorted from most preferred to least preferred.
Each entry has the following bytestring keys:
name
Name of the compression engine. e.g. ``zstd`` or ``zlib``.
heads heads
----- -----
Obtain DAG heads in the repository. Obtain DAG heads in the repository.
The command accepts the following arguments: The command accepts the following arguments:
publiconly (optional) publiconly (optional)

mercurial/wireproto.py

capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role='server')) capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role='server'))
caps.append('bundle2=' + urlreq.quote(capsblob)) caps.append('bundle2=' + urlreq.quote(capsblob))
caps.append('unbundle=%s' % ','.join(bundle2.bundlepriority)) caps.append('unbundle=%s' % ','.join(bundle2.bundlepriority))
return proto.addcapabilities(repo, caps) return proto.addcapabilities(repo, caps)
# If you are writing an extension and consider wrapping this function. Wrap # If you are writing an extension and consider wrapping this function. Wrap
# `_capabilities` instead. # `_capabilities` instead.
@wireprotocommand('capabilities', permission='pull') @wireprotocommand('capabilities', permission='pull',
transportpolicy=POLICY_V1_ONLY)
def capabilities(repo, proto): def capabilities(repo, proto):
caps = _capabilities(repo, proto) caps = _capabilities(repo, proto)
return wireprototypes.bytesresponse(' '.join(sorted(caps))) return wireprototypes.bytesresponse(' '.join(sorted(caps)))
@wireprotocommand('changegroup', 'roots', transportpolicy=POLICY_V1_ONLY, @wireprotocommand('changegroup', 'roots', transportpolicy=POLICY_V1_ONLY,
permission='pull') permission='pull')
def changegroup(repo, proto, roots): def changegroup(repo, proto, roots):
nodes = decodelist(roots) nodes = decodelist(roots)
manargs, advargs)) manargs, advargs))
except error.PushRaced as exc: except error.PushRaced as exc:
bundler.newpart('error:pushraced', bundler.newpart('error:pushraced',
[('message', stringutil.forcebytestr(exc))]) [('message', stringutil.forcebytestr(exc))])
return wireprototypes.streamreslegacy(gen=bundler.getchunks()) return wireprototypes.streamreslegacy(gen=bundler.getchunks())
# Wire protocol version 2 commands only past this point. # Wire protocol version 2 commands only past this point.
def _capabilitiesv2(repo, proto):
"""Obtain the set of capabilities for version 2 transports.
These capabilities are distinct from the capabilities for version 1
transports.
"""
compression = []
for engine in supportedcompengines(repo.ui, util.SERVERROLE):
compression.append({
b'name': engine.wireprotosupport().name,
})
caps = {
'commands': {},
'compression': compression,
}
for command, entry in commandsv2.items():
caps['commands'][command] = {
'args': sorted(entry.args.split()) if entry.args else [],
'permissions': [entry.permission],
}
return proto.addcapabilities(repo, caps)
@wireprotocommand('branchmap', permission='pull', @wireprotocommand('branchmap', permission='pull',
transportpolicy=POLICY_V2_ONLY) transportpolicy=POLICY_V2_ONLY)
def branchmapv2(repo, proto): def branchmapv2(repo, proto):
branchmap = {encoding.fromlocal(k): v branchmap = {encoding.fromlocal(k): v
for k, v in repo.branchmap().iteritems()} for k, v in repo.branchmap().iteritems()}
return wireprototypes.cborresponse(branchmap) return wireprototypes.cborresponse(branchmap)
@wireprotocommand('capabilities', permission='pull',
transportpolicy=POLICY_V2_ONLY)
def capabilitiesv2(repo, proto):
caps = _capabilitiesv2(repo, proto)
return wireprototypes.cborresponse(caps)
@wireprotocommand('heads', args='publiconly', permission='pull', @wireprotocommand('heads', args='publiconly', permission='pull',
transportpolicy=POLICY_V2_ONLY) transportpolicy=POLICY_V2_ONLY)
def headsv2(repo, proto, publiconly=False): def headsv2(repo, proto, publiconly=False):
if publiconly: if publiconly:
repo = repo.filtered('immutable') repo = repo.filtered('immutable')
return wireprototypes.cborresponse(repo.heads()) return wireprototypes.cborresponse(repo.heads())

mercurial/wireprotoserver.py

class sshv2protocolhandler(sshv1protocolhandler): class sshv2protocolhandler(sshv1protocolhandler):
"""Protocol handler for version 2 of the SSH protocol.""" """Protocol handler for version 2 of the SSH protocol."""
@property @property
def name(self): def name(self):
return wireprototypes.SSHV2 return wireprototypes.SSHV2
def addcapabilities(self, repo, caps):
return caps
def _runsshserver(ui, repo, fin, fout, ev): def _runsshserver(ui, repo, fin, fout, ev):
# This function operates like a state machine of sorts. The following # This function operates like a state machine of sorts. The following
# states are defined: # states are defined:
# #
# protov1-serving # protov1-serving
# Server is in protocol version 1 serving mode. Commands arrive on # Server is in protocol version 1 serving mode. Commands arrive on
# new lines. These commands are processed in this state, one command # new lines. These commands are processed in this state, one command
# after the other. # after the other.

tests/test-ssh-proto.t

o> upgraded this-is-some-token exp-ssh-v2-0001\n o> upgraded this-is-some-token exp-ssh-v2-0001\n
o> readline() -> 4: o> readline() -> 4:
o> 412\n o> 412\n
o> readline() -> 413: o> readline() -> 413:
o> capabilities: batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset getbundle known lookup protocaps pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash\n o> capabilities: batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset getbundle known lookup protocaps pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash\n
i> write(6) -> 6: i> write(6) -> 6:
i> hello\n i> hello\n
o> readline() -> 4: o> readline() -> 4:
o> 403\n o> 397\n
o> readline() -> 403: o> readline() -> 397:
o> capabilities: batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset getbundle known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash\n o> capabilities: branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset getbundle known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash\n
Multiple upgrades is not allowed Multiple upgrades is not allowed
$ hg debugwireproto --localssh --peer raw << EOF $ hg debugwireproto --localssh --peer raw << EOF
> raw > raw
> upgrade this-is-some-token proto=exp-ssh-v2-0001\n > upgrade this-is-some-token proto=exp-ssh-v2-0001\n
> hello\n > hello\n
> between\n > between\n

tests/test-wireproto-command-capabilities.t

  • This file was added.
$ . $TESTDIR/wireprotohelpers.sh
$ hg init server
$ enablehttpv2 server
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
$ cat hg.pid > $DAEMON_PIDS
capabilities request returns an array of capability strings
$ sendhttpv2peer << EOF
> command capabilities
> EOF
creating http peer for wire protocol version 2
sending capabilities command
s> POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-exp-framing-0003\r\n
s> content-type: application/mercurial-exp-framing-0003\r\n
s> content-length: 27\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> \x13\x00\x00\x01\x00\x01\x01\x11\xa1DnameLcapabilities
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-exp-framing-0003\r\n
s> Transfer-Encoding: chunked\r\n
s> \r\n
s> *\r\n (glob)
s> *\x00\x01\x00\x02\x01F (glob)
s> \xa2Hcommands\xabEheads\xa2Dargs\x81JpubliconlyKpermissions\x81DpullEknown\xa2Dargs\x81EnodesKpermissions\x81DpullFlookup\xa2Dargs\x81CkeyKpermissions\x81DpullGpushkey\xa2Dargs\x84CkeyInamespaceCnewColdKpermissions\x81DpushHlistkeys\xa2Dargs\x81InamespaceKpermissions\x81DpullHunbundle\xa2Dargs\x81EheadsKpermissions\x81DpushIbranchmap\xa2Dargs\x80Kpermissions\x81DpullIgetbundle\xa2Dargs\x81A*Kpermissions\x81DpullJstream_out\xa2Dargs\x80Kpermissions\x81DpullLcapabilities\xa2Dargs\x80Kpermissions\x81DpullLclonebundles\xa2Dargs\x80Kpermissions\x81DpullKcompression\x82\xa1DnameDzstd\xa1DnameDzlib
s> \r\n
received frame(size=*; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos|cbor) (glob)
s> 0\r\n
s> \r\n
response: [{b'commands': {b'branchmap': {b'args': [], b'permissions': [b'pull']}, b'capabilities': {b'args': [], b'permissions': [b'pull']}, b'clonebundles': {b'args': [], b'permissions': [b'pull']}, b'getbundle': {b'args': [b'*'], b'permissions': [b'pull']}, b'heads': {b'args': [b'publiconly'], b'permissions': [b'pull']}, b'known': {b'args': [b'nodes'], b'permissions': [b'pull']}, b'listkeys': {b'args': [b'namespace'], b'permissions': [b'pull']}, b'lookup': {b'args': [b'key'], b'permissions': [b'pull']}, b'pushkey': {b'args': [b'key', b'namespace', b'new', b'old'], b'permissions': [b'push']}, b'stream_out': {b'args': [], b'permissions': [b'pull']}, b'unbundle': {b'args': [b'heads'], b'permissions': [b'push']}}, b'compression': [{b'name': b'zstd'}, {b'name': b'zlib'}]}]
$ cat error.log