This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3177

wireproto: crude support for version 2 HTTP peer
ClosedPublic

Authored by indygreg on Apr 7 2018, 1:53 AM.

Details

Reviewers
yuja
Group Reviewers
hg-reviewers
Commits
rHG61e405fb6372: wireproto: crude support for version 2 HTTP peer
Summary

As part of implementing the server-side bits of the wire protocol
command handlers for version 2, we want a way to easily test those
commands. Currently, we use the "httprequest" action of `hg
debugwireproto`. But this requires explicitly specifying the HTTP
request headers, low-level frame details, and the data structure
to encode with CBOR. That's a lot of boilerplate and a lot of it can
change as the wire protocol evolves.

hg debugwireproto has a mechanism to issue commands via the peer
interface. That is *much* easier to use and we prefer to test with
that going forward.

This commit implements enough parts of the peer API to send basic
requests via the HTTP version 2 transport.

The peer code is super hacky. Again, the goal is to facilitate
server testing, not robustly implement a client. The client code
will receive love at a later time.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Apr 7 2018, 1:53 AM
yuja accepted this revision.Apr 9 2018, 10:58 AM
This revision is now accepted and ready to land.Apr 9 2018, 10:58 AM
yuja added inline comments.Apr 9 2018, 11:07 AM
mercurial/debugcommands.py
2962

My patch broke this, so fixed in flight. :)

Closed by commit rHG61e405fb6372: wireproto: crude support for version 2 HTTP peer (authored by indygreg). · Explain WhyApr 9 2018, 11:13 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 7894

mercurial/debugcommands.py

peers and to debug the raw data being exchanged. peers and to debug the raw data being exchanged.
``--localssh`` will start an SSH server against the current repository ``--localssh`` will start an SSH server against the current repository
and connect to that. By default, the connection will perform a handshake and connect to that. By default, the connection will perform a handshake
and establish an appropriate peer instance. and establish an appropriate peer instance.
``--peer`` can be used to bypass the handshake protocol and construct a ``--peer`` can be used to bypass the handshake protocol and construct a
peer instance using the specified class type. Valid values are ``raw``, peer instance using the specified class type. Valid values are ``raw``,
``ssh1``, and ``ssh2``. ``raw`` instances only allow sending raw data ``http2``, ``ssh1``, and ``ssh2``. ``raw`` instances only allow sending
payloads and don't support higher-level command actions. raw data payloads and don't support higher-level command actions.
``--noreadstderr`` can be used to disable automatic reading from stderr ``--noreadstderr`` can be used to disable automatic reading from stderr
of the peer (for SSH connections only). Disabling automatic reading of of the peer (for SSH connections only). Disabling automatic reading of
stderr is useful for making output more deterministic. stderr is useful for making output more deterministic.
Commands are issued via a mini language which is specified via stdin. Commands are issued via a mini language which is specified via stdin.
The language consists of individual actions to perform. An action is The language consists of individual actions to perform. An action is
defined by a block. A block is defined as a line with no leading defined by a block. A block is defined as a line with no leading
string. string.
Arguments to the command are defined as lines in this block. The format of Arguments to the command are defined as lines in this block. The format of
each line is ``<key> <value>``. e.g.:: each line is ``<key> <value>``. e.g.::
command listkeys command listkeys
namespace bookmarks namespace bookmarks
Values are interpreted as Python b'' literals. This allows encoding If the value begins with ``eval:``, it will be interpreted as a Python
special byte sequences via backslash escaping. literal expression. Otherwise values are interpreted as Python b'' literals.
This allows sending complex types and encoding special byte sequences via
backslash escaping.
The following arguments have special meaning: The following arguments have special meaning:
``PUSHFILE`` ``PUSHFILE``
When defined, the *push* mechanism of the peer will be used instead When defined, the *push* mechanism of the peer will be used instead
of the static request-response mechanism and the content of the of the static request-response mechanism and the content of the
file specified in the value of this argument will be sent as the file specified in the value of this argument will be sent as the
command payload. command payload.
resulting object is fed into a CBOR encoder. Otherwise it is interpreted resulting object is fed into a CBOR encoder. Otherwise it is interpreted
as a Python byte string literal. as a Python byte string literal.
""" """
opts = pycompat.byteskwargs(opts) opts = pycompat.byteskwargs(opts)
if opts['localssh'] and not repo: if opts['localssh'] and not repo:
raise error.Abort(_('--localssh requires a repository')) raise error.Abort(_('--localssh requires a repository'))
if opts['peer'] and opts['peer'] not in ('raw', 'ssh1', 'ssh2'): if opts['peer'] and opts['peer'] not in ('raw', 'http2', 'ssh1', 'ssh2'):
raise error.Abort(_('invalid value for --peer'), raise error.Abort(_('invalid value for --peer'),
hint=_('valid values are "raw", "ssh1", and "ssh2"')) hint=_('valid values are "raw", "ssh1", and "ssh2"'))
if path and opts['localssh']: if path and opts['localssh']:
raise error.Abort(_('cannot specify --localssh with an explicit ' raise error.Abort(_('cannot specify --localssh with an explicit '
'path')) 'path'))
if ui.interactive(): if ui.interactive():
# We bypass hg.peer() so we can proxy the sockets. # We bypass hg.peer() so we can proxy the sockets.
# TODO consider not doing this because we skip # TODO consider not doing this because we skip
# ``hg.wirepeersetupfuncs`` and potentially other useful functionality. # ``hg.wirepeersetupfuncs`` and potentially other useful functionality.
u = util.url(path) u = util.url(path)
if u.scheme != 'http': if u.scheme != 'http':
raise error.Abort(_('only http:// paths are currently supported')) raise error.Abort(_('only http:// paths are currently supported'))
url, authinfo = u.authinfo() url, authinfo = u.authinfo()
openerargs = {} openerargs = {
r'useragent': b'Mercurial debugwireproto',
}
# Turn pipes/sockets into observers so we can log I/O. # Turn pipes/sockets into observers so we can log I/O.
if ui.verbose: if ui.verbose:
openerargs = { openerargs.update({
r'loggingfh': ui, r'loggingfh': ui,
r'loggingname': b's', r'loggingname': b's',
r'loggingopts': { r'loggingopts': {
r'logdata': True, r'logdata': True,
r'logdataapis': False, r'logdataapis': False,
}, },
} })
if ui.debugflag: if ui.debugflag:
openerargs[r'loggingopts'][r'logdataapis'] = True openerargs[r'loggingopts'][r'logdataapis'] = True
# Don't send default headers when in raw mode. This allows us to # Don't send default headers when in raw mode. This allows us to
# bypass most of the behavior of our URL handling code so we can # bypass most of the behavior of our URL handling code so we can
# have near complete control over what's sent on the wire. # have near complete control over what's sent on the wire.
if opts['peer'] == 'raw': if opts['peer'] == 'raw':
openerargs[r'sendaccept'] = False openerargs[r'sendaccept'] = False
opener = urlmod.opener(ui, authinfo, **openerargs) opener = urlmod.opener(ui, authinfo, **openerargs)
if opts['peer'] == 'raw': if opts['peer'] == 'http2':
ui.write(_('creating http peer for wire protocol version 2\n'))
peer = httppeer.httpv2peer(ui, path, opener)
elif opts['peer'] == 'raw':
ui.write(_('using raw connection to peer\n')) ui.write(_('using raw connection to peer\n'))
peer = None peer = None
elif opts['peer']: elif opts['peer']:
raise error.Abort(_('--peer %s not supported with HTTP peers') % raise error.Abort(_('--peer %s not supported with HTTP peers') %
opts['peer']) opts['peer'])
else: else:
peer = httppeer.httppeer(ui, path, url, opener) peer = httppeer.httppeer(ui, path, url, opener)
peer._fetchcaps() peer._fetchcaps()
# We need to allow empty values. # We need to allow empty values.
fields = line.lstrip().split(' ', 1) fields = line.lstrip().split(' ', 1)
if len(fields) == 1: if len(fields) == 1:
key = fields[0] key = fields[0]
value = '' value = ''
else: else:
key, value = fields key, value = fields
args[key] = stringutil.unescapestr(value) if value.startswith('eval:'):
value = stringutil.evalpythonliteral(value[5:])
yujaUnsubmitted
Not Done

My patch broke this, so fixed in flight. :)

yuja: My patch broke this, so fixed in flight. :)
else:
value = stringutil.unescapestr(value)
args[key] = value
if batchedcommands is not None: if batchedcommands is not None:
batchedcommands.append((command, args)) batchedcommands.append((command, args))
continue continue
ui.status(_('sending %s command\n') % command) ui.status(_('sending %s command\n') % command)
if 'PUSHFILE' in args: if 'PUSHFILE' in args:

mercurial/httppeer.py

import errno import errno
import io import io
import os import os
import socket import socket
import struct import struct
import tempfile import tempfile
from .i18n import _ from .i18n import _
from .thirdparty import (
cbor,
)
from . import ( from . import (
bundle2, bundle2,
error, error,
httpconnection, httpconnection,
pycompat, pycompat,
statichttprepo, statichttprepo,
url as urlmod, url as urlmod,
util, util,
wireproto, wireproto,
wireprotoframing,
wireprotoserver,
) )
httplib = util.httplib httplib = util.httplib
urlerr = util.urlerr urlerr = util.urlerr
urlreq = util.urlreq urlreq = util.urlreq
def encodevalueinheaders(value, header, limit): def encodevalueinheaders(value, header, limit):
"""Encode a string value into multiple HTTP headers. """Encode a string value into multiple HTTP headers.
os.unlink(filename) os.unlink(filename)
def _callcompressable(self, cmd, **args): def _callcompressable(self, cmd, **args):
return self._callstream(cmd, _compressible=True, **args) return self._callstream(cmd, _compressible=True, **args)
def _abort(self, exception): def _abort(self, exception):
raise exception raise exception
# TODO implement interface for version 2 peers
class httpv2peer(object):
def __init__(self, ui, repourl, opener):
self.ui = ui
if repourl.endswith('/'):
repourl = repourl[:-1]
self.url = repourl
self._opener = opener
# This is an its own attribute to facilitate extensions overriding
# the default type.
self._requestbuilder = urlreq.request
def close(self):
pass
# TODO require to be part of a batched primitive, use futures.
def _call(self, name, **args):
"""Call a wire protocol command with arguments."""
# TODO permissions should come from capabilities results.
permission = wireproto.commandsv2[name].permission
if permission not in ('push', 'pull'):
raise error.ProgrammingError('unknown permission type: %s' %
permission)
permission = {
'push': 'rw',
'pull': 'ro',
}[permission]
url = '%s/api/%s/%s/%s' % (self.url, wireprotoserver.HTTPV2, permission,
name)
# TODO modify user-agent to reflect v2.
headers = {
r'Accept': wireprotoserver.FRAMINGTYPE,
r'Content-Type': wireprotoserver.FRAMINGTYPE,
}
# TODO this should be part of a generic peer for the frame-based
# protocol.
stream = wireprotoframing.stream(1)
frames = wireprotoframing.createcommandframes(stream, 1,
name, args)
body = b''.join(map(bytes, frames))
req = self._requestbuilder(pycompat.strurl(url), body, headers)
req.add_unredirected_header(r'Content-Length', r'%d' % len(body))
# TODO unify this code with httppeer.
try:
res = self._opener.open(req)
except urlerr.httperror as e:
if e.code == 401:
raise error.Abort(_('authorization failed'))
raise
except httplib.HTTPException as e:
self.ui.traceback()
raise IOError(None, e)
# TODO validate response type, wrap response to handle I/O errors.
# TODO more robust frame receiver.
results = []
while True:
frame = wireprotoframing.readframe(res)
if frame is None:
break
self.ui.note(_('received %r\n') % frame)
if frame.typeid == wireprotoframing.FRAME_TYPE_BYTES_RESPONSE:
if frame.flags & wireprotoframing.FLAG_BYTES_RESPONSE_CBOR:
payload = util.bytesio(frame.payload)
decoder = cbor.CBORDecoder(payload)
while payload.tell() + 1 < len(frame.payload):
results.append(decoder.decode())
else:
results.append(frame.payload)
else:
error.ProgrammingError('unhandled frame type: %d' %
frame.typeid)
return results
def makepeer(ui, path): def makepeer(ui, path):
u = util.url(path) u = util.url(path)
if u.query or u.fragment: if u.query or u.fragment:
raise error.Abort(_('unsupported URL component: "%s"') % raise error.Abort(_('unsupported URL component: "%s"') %
(u.query or u.fragment)) (u.query or u.fragment))
# urllib cannot handle URLs with embedded user or passwd. # urllib cannot handle URLs with embedded user or passwd.
url, authinfo = u.authinfo() url, authinfo = u.authinfo()

tests/test-http-api-httpv2.t

s> Transfer-Encoding: chunked\r\n s> Transfer-Encoding: chunked\r\n
s> \r\n s> \r\n
s> 25\r\n s> 25\r\n
s> \x1d\x00\x00\x01\x00\x02\x01Bcustomreadonly bytes response s> \x1d\x00\x00\x01\x00\x02\x01Bcustomreadonly bytes response
s> \r\n s> \r\n
s> 0\r\n s> 0\r\n
s> \r\n s> \r\n
$ sendhttpv2peer << EOF
> command customreadonly
> EOF
creating http peer for wire protocol version 2
sending customreadonly command
s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-exp-framing-0003\r\n
s> content-type: application/mercurial-exp-framing-0003\r\n
s> content-length: 29\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> \x15\x00\x00\x01\x00\x01\x01\x11\xa1DnameNcustomreadonly
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-exp-framing-0003\r\n
s> Transfer-Encoding: chunked\r\n
s> \r\n
s> 25\r\n
s> \x1d\x00\x00\x01\x00\x02\x01B
s> customreadonly bytes response
s> \r\n
received frame(size=29; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos)
s> 0\r\n
s> \r\n
response: [b'customreadonly bytes response']
Request to read-write command fails because server is read-only by default Request to read-write command fails because server is read-only by default
GET to read-write request yields 405 GET to read-write request yields 405
$ sendhttpraw << EOF $ sendhttpraw << EOF
> httprequest GET api/$HTTPV2/rw/customreadonly > httprequest GET api/$HTTPV2/rw/customreadonly
> user-agent: test > user-agent: test
> EOF > EOF

tests/test-http-protocol.t

$ hg --verbose debugwireproto http://$LOCALIP:$HGPORT << EOF $ hg --verbose debugwireproto http://$LOCALIP:$HGPORT << EOF
> command listkeys > command listkeys
> namespace namespaces > namespace namespaces
> EOF > EOF
s> GET /?cmd=capabilities HTTP/1.1\r\n s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-0.1\r\n s> accept: application/mercurial-0.1\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: mercurial/proto-1.0 (Mercurial *)\r\n (glob) s> user-agent: Mercurial debugwireproto\r\n
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 Script output follows\r\n s> HTTP/1.1 200 Script output follows\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-0.1\r\n s> Content-Type: application/mercurial-0.1\r\n
s> Content-Length: *\r\n (glob) s> Content-Length: *\r\n (glob)
s> \r\n s> \r\n
s> batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash s> batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
sending listkeys command sending listkeys command
s> GET /?cmd=listkeys HTTP/1.1\r\n s> GET /?cmd=listkeys HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> vary: X-HgArg-1,X-HgProto-1\r\n s> vary: X-HgArg-1,X-HgProto-1\r\n
s> x-hgarg-1: namespace=namespaces\r\n s> x-hgarg-1: namespace=namespaces\r\n
s> x-hgproto-1: 0.1 0.2 comp=$USUAL_COMPRESSIONS$\r\n s> x-hgproto-1: 0.1 0.2 comp=$USUAL_COMPRESSIONS$\r\n
s> accept: application/mercurial-0.1\r\n s> accept: application/mercurial-0.1\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: mercurial/proto-1.0 (Mercurial *)\r\n (glob) s> user-agent: Mercurial debugwireproto\r\n
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 Script output follows\r\n s> HTTP/1.1 200 Script output follows\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-0.1\r\n s> Content-Type: application/mercurial-0.1\r\n
s> Content-Length: 30\r\n s> Content-Length: 30\r\n
s> \r\n s> \r\n

tests/wireprotohelpers.sh

HTTPV2=exp-http-v2-0001 HTTPV2=exp-http-v2-0001
MEDIATYPE=application/mercurial-exp-framing-0003 MEDIATYPE=application/mercurial-exp-framing-0003
sendhttpraw() { sendhttpraw() {
hg --verbose debugwireproto --peer raw http://$LOCALIP:$HGPORT/ hg --verbose debugwireproto --peer raw http://$LOCALIP:$HGPORT/
} }
sendhttpv2peer() {
hg --verbose debugwireproto --peer http2 http://$LOCALIP:$HGPORT/
}
cat > dummycommands.py << EOF cat > dummycommands.py << EOF
from mercurial import ( from mercurial import (
wireprototypes, wireprototypes,
wireproto, wireproto,
) )
@wireproto.wireprotocommand('customreadonly', permission='pull') @wireproto.wireprotocommand('customreadonly', permission='pull')
def customreadonly(repo, proto): def customreadonly(repo, proto):