This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2850

wireproto: define content negotiation for HTTPv2
ClosedPublic

Authored by indygreg on Mar 13 2018, 9:16 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG37d7a1d18b97: wireproto: define content negotiation for HTTPv2
Summary

HTTP messages communicate their media types and what media types
they can understand via the Content-Type and Accept header,
respectively.

While I don't want the wire protocol to lean too heavily on HTTP
because I'm aiming for the wire protocol to be as transport
agnostic as possible, it is nice to play by the spec if possible.

This commit defines our media negotiation mechanism for version
2 of the HTTP protocol. Essentially, we mandate the use of a
new media type and how clients and servers should react to
various headers or lack thereof.

The name of the media type is a placeholder. We purposefully don't
yet define the format of the new media type because that's a lot
of work.

I feel pretty strongly that we should use Content-Type. I feel
less strongly about Accept. I think it is reasonable for servers
to return the media type that was submitted to them. So we may
strike this header before the protocol is finished...

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 13 2018, 9:16 PM
indygreg updated this revision to Diff 7138.Mar 19 2018, 7:59 PM
durin42 accepted this revision.Mar 21 2018, 3:18 PM
This revision is now accepted and ready to land.Mar 21 2018, 3:18 PM
indygreg updated this revision to Diff 7198.Mar 21 2018, 6:19 PM
durin42 accepted this revision.Mar 21 2018, 6:22 PM
Closed by commit rHG37d7a1d18b97: wireproto: define content negotiation for HTTPv2 (authored by indygreg). · Explain WhyMar 21 2018, 8:58 PM
This revision was automatically updated to reflect the committed changes.
yuja added a subscriber: yuja.Mar 24 2018, 9:31 AM

We might have to handle multiple Accept headers correctly, if a proxy
server MAY modify them. I don't know if that could happen, though.

Revision Contents
Changeset List

Diff 7226

mercurial/help/internals/wireprotocol.txt

authentication requirements are discovered upon attempting the first authentication requirements are discovered upon attempting the first
command - not potentially several commands into the exchange. This command - not potentially several commands into the exchange. This
allows clients to fail faster or prompt for credentials as soon as the allows clients to fail faster or prompt for credentials as soon as the
exchange takes place. This provides a better end-user experience. exchange takes place. This provides a better end-user experience.
Requests to unknown commands or URLS result in an HTTP 404. Requests to unknown commands or URLS result in an HTTP 404.
TODO formally define response type, how error is communicated, etc. TODO formally define response type, how error is communicated, etc.
HTTP request and response bodies use the *TBD Protocol* for media exchange.
Clients and servers MUST advertise the ``TBD`` media type via the
``Content-Type`` request and response headers. In addition, clients MUST
advertise this media type value in their ``Accept`` request header in all
requests.
Servers receiving requests without an ``Accept`` header SHOULD respond with
an HTTP 406.
Servers receiving requests with an invalid ``Content-Type`` header SHOULD
respond with an HTTP 415.
SSH Protocol SSH Protocol
============ ============
Handshake Handshake
--------- ---------
For all clients, the handshake consists of the client sending 1 or more For all clients, the handshake consists of the client sending 1 or more
commands to the server using version 1 of the transport. Servers respond commands to the server using version 1 of the transport. Servers respond

mercurial/wireprotoserver.py

urlerr = util.urlerr urlerr = util.urlerr
urlreq = util.urlreq urlreq = util.urlreq
HTTP_OK = 200 HTTP_OK = 200
HGTYPE = 'application/mercurial-0.1' HGTYPE = 'application/mercurial-0.1'
HGTYPE2 = 'application/mercurial-0.2' HGTYPE2 = 'application/mercurial-0.2'
HGERRTYPE = 'application/hg-error' HGERRTYPE = 'application/hg-error'
HTTPV2TYPE = 'application/mercurial-tbd'
HTTPV2 = wireprototypes.HTTPV2 HTTPV2 = wireprototypes.HTTPV2
SSHV1 = wireprototypes.SSHV1 SSHV1 = wireprototypes.SSHV1
SSHV2 = wireprototypes.SSHV2 SSHV2 = wireprototypes.SSHV2
def decodevaluefromheaders(req, headerprefix): def decodevaluefromheaders(req, headerprefix):
"""Decode a long value from multiple HTTP request headers. """Decode a long value from multiple HTTP request headers.
proto = httpv2protocolhandler(req, ui) proto = httpv2protocolhandler(req, ui)
if not wireproto.commands.commandavailable(command, proto): if not wireproto.commands.commandavailable(command, proto):
res.status = b'404 Not Found' res.status = b'404 Not Found'
res.headers[b'Content-Type'] = b'text/plain' res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('invalid wire protocol command: %s') % command) res.setbodybytes(_('invalid wire protocol command: %s') % command)
return return
if req.headers.get(b'Accept') != HTTPV2TYPE:
res.status = b'406 Not Acceptable'
res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('client MUST specify Accept header with value: %s\n')
% HTTPV2TYPE)
return
if (b'Content-Type' in req.headers
and req.headers[b'Content-Type'] != HTTPV2TYPE):
res.status = b'415 Unsupported Media Type'
# TODO we should send a response with appropriate media type,
# since client does Accept it.
res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(_('client MUST send Content-Type header with '
'value: %s\n') % HTTPV2TYPE)
return
# We don't do anything meaningful yet. # We don't do anything meaningful yet.
res.status = b'200 OK' res.status = b'200 OK'
res.headers[b'Content-Type'] = b'text/plain' res.headers[b'Content-Type'] = b'text/plain'
res.setbodybytes(b'/'.join(urlparts) + b'\n') res.setbodybytes(b'/'.join(urlparts) + b'\n')
# Maps API name to metadata so custom API can be registered. # Maps API name to metadata so custom API can be registered.
API_HANDLERS = { API_HANDLERS = {
HTTPV2: { HTTPV2: {

tests/test-http-api-httpv2.t

$ HTTPV2=exp-http-v2-0001 $ HTTPV2=exp-http-v2-0001
$ MEDIATYPE=application/mercurial-tbd
$ send() { $ send() {
> hg --verbose debugwireproto --peer raw http://$LOCALIP:$HGPORT/ > hg --verbose debugwireproto --peer raw http://$LOCALIP:$HGPORT/
> } > }
$ cat > dummycommands.py << EOF $ cat > dummycommands.py << EOF
> from mercurial import wireprototypes, wireproto > from mercurial import wireprototypes, wireproto
> @wireproto.wireprotocommand('customreadonly', permission='pull') > @wireproto.wireprotocommand('customreadonly', permission='pull')
> [experimental] > [experimental]
> web.apiserver = true > web.apiserver = true
> web.api.http-v2 = true > web.api.http-v2 = true
> EOF > EOF
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid $ hg -R server serve -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS $ cat hg.pid > $DAEMON_PIDS
Request to read-only command works out of the box Request to unknown command yields 404
$ send << EOF
> httprequest POST api/$HTTPV2/ro/badcommand
> user-agent: test
> EOF
using raw connection to peer
s> POST /api/exp-http-v2-0001/ro/badcommand HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 404 Not Found\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n
s> Content-Length: 42\r\n
s> \r\n
s> unknown wire protocol command: badcommand\n
GET to read-only command yields a 405
$ send << EOF
> httprequest GET api/$HTTPV2/ro/customreadonly
> user-agent: test
> EOF
using raw connection to peer
s> GET /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 405 Method Not Allowed\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Allow: POST\r\n
s> Content-Length: 30\r\n
s> \r\n
s> commands require POST requests
Missing Accept header results in 406
$ send << EOF $ send << EOF
> httprequest POST api/$HTTPV2/ro/customreadonly > httprequest POST api/$HTTPV2/ro/customreadonly
> user-agent: test > user-agent: test
> EOF > EOF
using raw connection to peer using raw connection to peer
s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> user-agent: test\r\n s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n s> HTTP/1.1 406 Not Acceptable\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n s> Content-Type: text/plain\r\n
s> Content-Length: 18\r\n s> Content-Length: 72\r\n
s> \r\n s> \r\n
s> ro/customreadonly\n s> client MUST specify Accept header with value: application/mercurial-tbd\n
Request to unknown command yields 404 Bad Accept header results in 406
$ send << EOF $ send << EOF
> httprequest POST api/$HTTPV2/ro/badcommand > httprequest POST api/$HTTPV2/ro/customreadonly
> accept: invalid
> user-agent: test > user-agent: test
> EOF > EOF
using raw connection to peer using raw connection to peer
s> POST /api/exp-http-v2-0001/ro/badcommand HTTP/1.1\r\n s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> accept: invalid\r\n
s> user-agent: test\r\n s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 404 Not Found\r\n s> HTTP/1.1 406 Not Acceptable\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n s> Content-Type: text/plain\r\n
s> Content-Length: 42\r\n s> Content-Length: 72\r\n
s> \r\n s> \r\n
s> unknown wire protocol command: badcommand\n s> client MUST specify Accept header with value: application/mercurial-tbd\n
GET to read-only command yields a 405 Bad Content-Type header results in 415
$ send << EOF $ send << EOF
> httprequest GET api/$HTTPV2/ro/customreadonly > httprequest POST api/$HTTPV2/ro/customreadonly
> accept: $MEDIATYPE
> user-agent: test > user-agent: test
> content-type: badmedia
> EOF > EOF
using raw connection to peer using raw connection to peer
s> GET /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-tbd\r\n
s> content-type: badmedia\r\n
s> user-agent: test\r\n s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 405 Method Not Allowed\r\n s> HTTP/1.1 415 Unsupported Media Type\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Allow: POST\r\n s> Content-Type: text/plain\r\n
s> Content-Length: 30\r\n s> Content-Length: 75\r\n
s> \r\n s> \r\n
s> commands require POST requests s> client MUST send Content-Type header with value: application/mercurial-tbd\n
Request to read-only command works out of the box
$ send << EOF
> httprequest POST api/$HTTPV2/ro/customreadonly
> accept: $MEDIATYPE
> content-type: $MEDIATYPE
> user-agent: test
> EOF
using raw connection to peer
s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-tbd\r\n
s> content-type: application/mercurial-tbd\r\n
s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n
s> Content-Length: 18\r\n
s> \r\n
s> ro/customreadonly\n
Request to read-write command fails because server is read-only by default Request to read-write command fails because server is read-only by default
GET to read-write request yields 405 GET to read-write request yields 405
$ send << EOF $ send << EOF
> httprequest GET api/$HTTPV2/rw/customreadonly > httprequest GET api/$HTTPV2/rw/customreadonly
> user-agent: test > user-agent: test
$ hg -R server serve -p $HGPORT -d --pid-file hg.pid $ hg -R server serve -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS $ cat hg.pid > $DAEMON_PIDS
Authorized request for valid read-write command works Authorized request for valid read-write command works
$ send << EOF $ send << EOF
> httprequest POST api/$HTTPV2/rw/customreadonly > httprequest POST api/$HTTPV2/rw/customreadonly
> user-agent: test > user-agent: test
> accept: $MEDIATYPE
> content-type: $MEDIATYPE
> EOF > EOF
using raw connection to peer using raw connection to peer
s> POST /api/exp-http-v2-0001/rw/customreadonly HTTP/1.1\r\n s> POST /api/exp-http-v2-0001/rw/customreadonly HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-tbd\r\n
s> content-type: application/mercurial-tbd\r\n
s> user-agent: test\r\n s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n s> Content-Type: text/plain\r\n
s> Content-Length: 18\r\n s> Content-Length: 18\r\n
s> \r\n s> \r\n
s> rw/customreadonly\n s> rw/customreadonly\n
Authorized request for unknown command is rejected Authorized request for unknown command is rejected
$ send << EOF $ send << EOF
> httprequest POST api/$HTTPV2/rw/badcommand > httprequest POST api/$HTTPV2/rw/badcommand
> user-agent: test > user-agent: test
> accept: $MEDIATYPE
> EOF > EOF
using raw connection to peer using raw connection to peer
s> POST /api/exp-http-v2-0001/rw/badcommand HTTP/1.1\r\n s> POST /api/exp-http-v2-0001/rw/badcommand HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-tbd\r\n
s> user-agent: test\r\n s> user-agent: test\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 404 Not Found\r\n s> HTTP/1.1 404 Not Found\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: text/plain\r\n s> Content-Type: text/plain\r\n
s> Content-Length: 42\r\n s> Content-Length: 42\r\n
s> \r\n s> \r\n
s> unknown wire protocol command: badcommand\n s> unknown wire protocol command: badcommand\n