( )⚙ D2836 wireproto: define permissions-based routing of HTTPv2 wire protocol

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2836

wireproto: define permissions-based routing of HTTPv2 wire protocol
ClosedPublic

Authored by indygreg on Mar 12 2018, 10:16 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGfddcb51b5084: wireproto: define permissions-based routing of HTTPv2 wire protocol
Summary

Now that we have a scaffolding for serving version 2 of the HTTP
protocol, let's start implementing it.

A good place to start is URL routing and basic request processing
semantics. We can focus on content types, capabilities detect, etc
later.

Version 2 of the HTTP wire protocol encodes the needed permissions
of the request in the URL path. The reasons for this are documented
in the added documentation. In short, a) it makes it really easy and
fail proof for server administrators to implement path-based
authentication and b) it will enable clients to realize very early in
a server exchange that authentication will be required to complete
the operation. This latter point avoids all kinds of complexity and
problems, like dealing with Expect: 100-continue and clients finding
out later during hg push that they need to provide authentication.
This will avoid the current badness where clients send a full bundle,
get an HTTP 403, provide authentication, then retransmit the bundle.

In order to implement command checking, we needed to implement a
protocol handler for the new wire protocol. Our handler is just
small enough to run the code we've implemented.

Tests for the defined functionality have been added.

I very much want to refactor the permissions checking code and define
a better response format. But this can be done later. Nothing is
covered by backwards compatibility at this point.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Mar 12 2018, 10:16 PM
indygreg updated this revision to Diff 6999.Mar 13 2018, 3:11 PM
indygreg updated this revision to Diff 7010.Mar 13 2018, 9:16 PM
indygreg updated this revision to Diff 7014.Mar 13 2018, 10:49 PM
indygreg planned changes to this revision.Mar 19 2018, 7:38 PM

I'm going to refactor tests for this to deal with an issue I had testing things on my MBP.

indygreg updated this revision to Diff 7135.Mar 19 2018, 7:59 PM
durin42 accepted this revision.Mar 21 2018, 3:18 PM
This revision is now accepted and ready to land.Mar 21 2018, 3:18 PM
indygreg updated this revision to Diff 7195.Mar 21 2018, 6:19 PM
durin42 accepted this revision.Mar 21 2018, 6:21 PM
Closed by commit rHGfddcb51b5084: wireproto: define permissions-based routing of HTTPv2 wire protocol (authored by indygreg). · Explain WhyMar 21 2018, 8:57 PM
This revision was automatically updated to reflect the committed changes.
yuja added a subscriber: yuja.Mar 23 2018, 11:35 AM
yuja added inline comments.
mercurial/wireprotoserver.py
284

Just a nit: No _() is needed because we don't know client language nor character encoding.

Revision Contents
Changeset List

Diff 7135

mercurial/debugcommands.py

body = fh.read() body = fh.read()
else: else:
raise error.Abort(_('unknown argument to httprequest: %s') % raise error.Abort(_('unknown argument to httprequest: %s') %
line) line)
url = path + httppath url = path + httppath
req = urlmod.urlreq.request(pycompat.strurl(url), body, headers) req = urlmod.urlreq.request(pycompat.strurl(url), body, headers)
# urllib.Request insists on using has_data() as a proxy for
# determining the request method. Override that to use our
# explicitly requested method.
req.get_method = lambda: method
try: try:
opener.open(req).read() opener.open(req).read()
except util.urlerr.urlerror as e: except util.urlerr.urlerror as e:
e.read() e.read()
elif action == 'close': elif action == 'close':
peer.close() peer.close()
elif action == 'readavailable': elif action == 'readavailable':

mercurial/help/internals/wireprotocol.txt

``application/mercurial-0.*`` media type and the HTTP response body contains ``application/mercurial-0.*`` media type and the HTTP response body contains
the raw string value (after compression decoding, if used). A the raw string value (after compression decoding, if used). A
``Content-Length`` header is typically issued, but not required. ``Content-Length`` header is typically issued, but not required.
A command returning a ``stream`` response issues a A command returning a ``stream`` response issues a
``application/mercurial-0.*`` media type and the HTTP response is typically ``application/mercurial-0.*`` media type and the HTTP response is typically
using *chunked transfer* (``Transfer-Encoding: chunked``). using *chunked transfer* (``Transfer-Encoding: chunked``).
HTTP Version 2 Transport
------------------------
**Experimental - feature under active development**
Version 2 of the HTTP protocol is exposed under the ``/api/*`` URL space.
It's final API name is not yet formalized.
Commands are triggered by sending HTTP requests against URLs of the
form ``<permission>/<command>``, where ``<permission>`` is ``ro`` or
``rw``, meaning read-only and read-write, respectively and ``<command>``
is a named wire protocol command.
Commands that modify repository state in meaningful ways MUST NOT be
exposed under the ``ro`` URL prefix. All available commands MUST be
available under the ``rw`` URL prefix.
Server adminstrators MAY implement blanket HTTP authentication keyed
off the URL prefix. For example, a server may require authentication
for all ``rw/*`` URLs and let unauthenticated requests to ``ro/*``
URL proceed. A server MAY issue an HTTP 401, 403, or 407 response
in accordance with RFC 7235. Clients SHOULD recognize the HTTP Basic
(RFC 7617) and Digest (RFC 7616) authentication schemes. Clients SHOULD
make an attempt to recognize unknown schemes using the
``WWW-Authenticate`` response header on a 401 response, as defined by
RFC 7235.
Read-only commands are accessible under ``rw/*`` URLs so clients can
signal the intent of the operation very early in the connection
lifecycle. For example, a ``push`` operation - which consists of
various read-only commands mixed with at least one read-write command -
can perform all commands against ``rw/*`` URLs so that any server-side
authentication requirements are discovered upon attempting the first
command - not potentially several commands into the exchange. This
allows clients to fail faster or prompt for credentials as soon as the
exchange takes place. This provides a better end-user experience.
Requests to unknown commands or URLS result in an HTTP 404.
TODO formally define response type, how error is communicated, etc.
SSH Protocol SSH Protocol
============ ============
Handshake Handshake
--------- ---------
For all clients, the handshake consists of the client sending 1 or more For all clients, the handshake consists of the client sending 1 or more
commands to the server using version 1 of the transport. Servers respond commands to the server using version 1 of the transport. Servers respond

mercurial/wireprotoserver.py

res.headers[b'Content-Type'] = b'text/plain' res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('API %s not enabled\n') % proto) res.setbodybytes(_('API %s not enabled\n') % proto)
return return
API_HANDLERS[proto]['handler'](rctx, req, res, checkperm, API_HANDLERS[proto]['handler'](rctx, req, res, checkperm,
req.dispatchparts[2:]) req.dispatchparts[2:])
def _handlehttpv2request(rctx, req, res, checkperm, urlparts): def _handlehttpv2request(rctx, req, res, checkperm, urlparts):
from .hgweb import common as hgwebcommon
# URL space looks like: <permissions>/<command>, where <permission> can
# be ``ro`` or ``rw`` to signal read-only or read-write, respectively.
# Root URL does nothing meaningful... yet.
if not urlparts:
res.status = b'200 OK'
res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('HTTP version 2 API handler'))
yujaUnsubmitted
Not Done

Just a nit: No _() is needed because we don't know client language nor character encoding.

yuja: Just a nit: No `_()` is needed because we don't know client language nor character encoding.
return
if len(urlparts) == 1:
res.status = b'404 Not Found'
res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('do not know how to process %s\n') %
req.dispatchpath)
return
permission, command = urlparts[0:2]
if permission not in (b'ro', b'rw'):
res.status = b'404 Not Found'
res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('unknown permission: %s') % permission)
return
# At some point we'll want to use our own API instead of recycling the
# behavior of version 1 of the wire protocol...
# TODO return reasonable responses - not responses that overload the
# HTTP status line message for error reporting.
try:
checkperm(rctx, req, 'pull' if permission == b'ro' else 'push')
except hgwebcommon.ErrorResponse as e:
res.status = hgwebcommon.statusmessage(e.code, pycompat.bytestr(e))
for k, v in e.headers:
res.headers[k] = v
res.setbodybytes('permission denied')
return
if command not in wireproto.commands:
res.status = b'404 Not Found'
res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('unknown wire protocol command: %s\n') % command)
return
repo = rctx.repo
ui = repo.ui
proto = httpv2protocolhandler(req, ui)
if not wireproto.commands.commandavailable(command, proto):
res.status = b'404 Not Found'
res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('invalid wire protocol command: %s') % command)
return
# We don't do anything meaningful yet.
res.status = b'200 OK' res.status = b'200 OK'
res.headers[b'Content-Type'] = b'text/plain' res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(b'/'.join(urlparts) + b'\n') res.setbodybytes(b'/'.join(urlparts) + b'\n')
# Maps API name to metadata so custom API can be registered. # Maps API name to metadata so custom API can be registered.
API_HANDLERS = { API_HANDLERS = {
HTTPV2: { HTTPV2: {
'config': ('experimental', 'web.api.http-v2'), 'config': ('experimental', 'web.api.http-v2'),
'handler': _handlehttpv2request, 'handler': _handlehttpv2request,
}, },
} }
class httpv2protocolhandler(wireprototypes.baseprotocolhandler):
def __init__(self, req, ui):
self._req = req
self._ui = ui
@property
def name(self):
return HTTPV2
def getargs(self, args):
raise NotImplementedError
def forwardpayload(self, fp):
raise NotImplementedError
@contextlib.contextmanager
def mayberedirectstdio(self):
raise NotImplementedError
def client(self):
raise NotImplementedError
def addcapabilities(self, repo, caps):
raise NotImplementedError
def checkperm(self, perm):
raise NotImplementedError
def _httpresponsetype(ui, req, prefer_uncompressed): def _httpresponsetype(ui, req, prefer_uncompressed):
"""Determine the appropriate response type and compression settings. """Determine the appropriate response type and compression settings.
Returns a tuple of (mediatype, compengine, engineopts). Returns a tuple of (mediatype, compengine, engineopts).
""" """
# Determine the response media type and compression engine based # Determine the response media type and compression engine based
# on the request parameters. # on the request parameters.
protocaps = decodevaluefromheaders(req, 'X-HgProto').split(' ') protocaps = decodevaluefromheaders(req, 'X-HgProto').split(' ')

tests/test-http-api-httpv2.t

$ HTTPV2=exp-http-v2-0001
$ send() { $ send() {
> hg --verbose debugwireproto --peer raw http://$LOCALIP:$HGPORT/ > hg --verbose debugwireproto --peer raw http://$LOCALIP:$HGPORT/
> } > }
$ cat > dummycommands.py << EOF
> from mercurial import wireprototypes, wireproto
> @wireproto.wireprotocommand('customreadonly', permission='pull')
> def customreadonly(repo, proto):
> return wireprototypes.bytesresponse(b'customreadonly bytes response')
> @wireproto.wireprotocommand('customreadwrite', permission='push')
> def customreadwrite(repo, proto):
> return wireprototypes.bytesresponse(b'customreadwrite bytes response')
> EOF
$ cat >> $HGRCPATH << EOF
> [extensions]
> dummycommands = $TESTTMP/dummycommands.py
> EOF
$ hg init server $ hg init server
$ cat > server/.hg/hgrc << EOF $ cat > server/.hg/hgrc << EOF
> [experimental] > [experimental]
> web.apiserver = true > web.apiserver = true
> EOF > EOF
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid $ hg -R server serve -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS $ cat hg.pid > $DAEMON_PIDS
HTTP v2 protocol not enabled by default HTTP v2 protocol not enabled by default
$ send << EOF $ send << EOF
> httprequest GET api/exp-http-v2-0001 > httprequest GET api/$HTTPV2
> user-agent: test > user-agent: test
> EOF > EOF
using raw connection to peer using raw connection to peer
s> GET /api/exp-http-v2-0001 HTTP/1.1\r\n s> GET /api/exp-http-v2-0001 HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n s> \r\n
> [experimental] > [experimental]
> web.apiserver = true > web.apiserver = true
> web.api.http-v2 = true > web.api.http-v2 = true
> EOF > EOF
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid $ hg -R server serve -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS $ cat hg.pid > $DAEMON_PIDS
Requests simply echo their path (for now) Request to read-only command works out of the box
$ send << EOF $ send << EOF
> httprequest GET api/exp-http-v2-0001/path1/path2 > httprequest GET api/$HTTPV2/ro/customreadonly
> user-agent: test > user-agent: test
> EOF > EOF
using raw connection to peer using raw connection to peer
s> GET /api/exp-http-v2-0001/path1/path2 HTTP/1.1\r\n s> GET /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n s> Content-Type: text/plain\r\n
s> Content-Length: 12\r\n s> Content-Length: 18\r\n
s> \r\n
s> ro/customreadonly\n
Request to unknown command yields 404
$ send << EOF
> httprequest GET api/$HTTPV2/ro/badcommand
> user-agent: test
> EOF
using raw connection to peer
s> GET /api/exp-http-v2-0001/ro/badcommand HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 404 Not Found\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n
s> Content-Length: 42\r\n
s> \r\n
s> unknown wire protocol command: badcommand\n
Request to read-write command fails because server is read-only by default
GET to read-write request not allowed
$ send << EOF
> httprequest GET api/$HTTPV2/rw/customreadonly
> user-agent: test
> EOF
using raw connection to peer
s> GET /api/exp-http-v2-0001/rw/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 405 push requires POST request\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Length: 17\r\n
s> \r\n
s> permission denied
Even for unknown commands
$ send << EOF
> httprequest GET api/$HTTPV2/rw/badcommand
> user-agent: test
> EOF
using raw connection to peer
s> GET /api/exp-http-v2-0001/rw/badcommand HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 405 push requires POST request\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Length: 17\r\n
s> \r\n
s> permission denied
SSL required by default
$ send << EOF
> httprequest POST api/$HTTPV2/rw/customreadonly
> user-agent: test
> EOF
using raw connection to peer
s> POST /api/exp-http-v2-0001/rw/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 403 ssl required\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Length: 17\r\n
s> \r\n
s> permission denied
Restart server to allow non-ssl read-write operations
$ killdaemons.py
$ cat > server/.hg/hgrc << EOF
> [experimental]
> web.apiserver = true
> web.api.http-v2 = true
> [web]
> push_ssl = false
> EOF
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS
Server insists on POST for read-write commands
$ send << EOF
> httprequest GET api/$HTTPV2/rw/customreadonly
> user-agent: test
> EOF
using raw connection to peer
s> GET /api/exp-http-v2-0001/rw/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 405 push requires POST request\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Length: 17\r\n
s> \r\n
s> permission denied
$ killdaemons.py
$ cat > server/.hg/hgrc << EOF
> [experimental]
> web.apiserver = true
> web.api.http-v2 = true
> [web]
> push_ssl = false
> allow-push = *
> EOF
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS
Authorized request for valid read-write command works
$ send << EOF
> httprequest POST api/$HTTPV2/rw/customreadonly
> user-agent: test
> EOF
using raw connection to peer
s> POST /api/exp-http-v2-0001/rw/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n
s> Content-Length: 18\r\n
s> \r\n
s> rw/customreadonly\n
Authorized request for unknown command is rejected
$ send << EOF
> httprequest POST api/$HTTPV2/rw/badcommand
> user-agent: test
> EOF
using raw connection to peer
s> POST /api/exp-http-v2-0001/rw/badcommand HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 404 Not Found\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n
s> Content-Length: 42\r\n
s> \r\n s> \r\n
s> path1/path2\n s> unknown wire protocol command: badcommand\n