This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2060

internals: refactor wire protocol documentation
ClosedPublic

Authored by indygreg on Feb 6 2018, 2:31 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG40d94ea51402: internals: refactor wire protocol documentation
Summary

Upcoming work will introduce a new version of the HTTP and SSH
transports. The differences will be significant enough to consider
them new transports. So, we now attach a version number to each
transport.

In addition, having the handshake documented after the transport
and in a single shared section made it harder to follow the flow
of the connection. The handshake documentation is now moved to the
protocol section it describes. We now have a generic section about
the purpose of the handshake, which was rewritten significantly.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Feb 6 2018, 2:31 PM
durin42 accepted this revision.Feb 7 2018, 5:04 PM
This revision is now accepted and ready to land.Feb 7 2018, 5:04 PM
Closed by commit rHG40d94ea51402: internals: refactor wire protocol documentation (authored by indygreg). · Explain WhyFeb 7 2018, 5:40 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 5300

mercurial/help/internals/wireprotocol.txt

The Mercurial wire protocol is a request-response based protocol The Mercurial wire protocol is a request-response based protocol
with multiple wire representations. with multiple wire representations.
Each request is modeled as a command name, a dictionary of arguments, and Each request is modeled as a command name, a dictionary of arguments, and
optional raw input. Command arguments and their types are intrinsic optional raw input. Command arguments and their types are intrinsic
properties of commands. So is the response type of the command. This means properties of commands. So is the response type of the command. This means
clients can't always send arbitrary arguments to servers and servers can't clients can't always send arbitrary arguments to servers and servers can't
return multiple response types. return multiple response types.
The protocol is synchronous and does not support multiplexing (concurrent The protocol is synchronous and does not support multiplexing (concurrent
commands). commands).
Transport Protocols Handshake
=================== =========
HTTP Transport It is required or common for clients to perform a *handshake* when connecting
-------------- to a server. The handshake serves the following purposes:
* Negotiating protocol/transport level options
* Allows the client to learn about server capabilities to influence
future requests
* Ensures the underlying transport channel is in a *clean* state
An important goal of the handshake is to allow clients to use more modern
wire protocol features. By default, clients must assume they are talking
to an old version of Mercurial server (possibly even the very first
implementation). So, clients should not attempt to call or utilize modern
wire protocol features until they have confirmation that the server
supports them. The handshake implementation is designed to allow both
ends to utilize the latest set of features and capabilities with as
few round trips as possible.
The handshake mechanism varies by transport and protocol and is documented
in the sections below.
HTTP Protocol
=============
Handshake
---------
The client sends a ``capabilities`` command request (``?cmd=capabilities``)
as soon as HTTP requests may be issued.
The server responds with a capabilities string, which the client parses to
learn about the server's abilities.
HTTP Version 1 Transport
------------------------
Commands are issued as HTTP/1.0 or HTTP/1.1 requests. Commands are Commands are issued as HTTP/1.0 or HTTP/1.1 requests. Commands are
sent to the base URL of the repository with the command name sent in sent to the base URL of the repository with the command name sent in
the ``cmd`` query string parameter. e.g. the ``cmd`` query string parameter. e.g.
``https://example.com/repo?cmd=capabilities``. The HTTP method is ``GET`` ``https://example.com/repo?cmd=capabilities``. The HTTP method is ``GET``
or ``POST`` depending on the command and whether there is a request or ``POST`` depending on the command and whether there is a request
body. body.
``application/mercurial-0.*`` media type and the HTTP response body contains ``application/mercurial-0.*`` media type and the HTTP response body contains
the raw string value (after compression decoding, if used). A the raw string value (after compression decoding, if used). A
``Content-Length`` header is typically issued, but not required. ``Content-Length`` header is typically issued, but not required.
A command returning a ``stream`` response issues a A command returning a ``stream`` response issues a
``application/mercurial-0.*`` media type and the HTTP response is typically ``application/mercurial-0.*`` media type and the HTTP response is typically
using *chunked transfer* (``Transfer-Encoding: chunked``). using *chunked transfer* (``Transfer-Encoding: chunked``).
SSH Transport SSH Protocol
============= ============
The SSH transport is a custom text-based protocol suitable for use over any Handshake
bi-directional stream transport. It is most commonly used with SSH. ---------
For all clients, the handshake consists of the client sending 1 or more
commands to the server using version 1 of the transport. Servers respond
to commands they know how to respond to and send an empty response (``0\n``)
for unknown commands (per standard behavior of version 1 of the transport).
Clients then typically look for a response to the newest sent command to
determine which transport version to use and what the available features for
the connection and server are.
Preceding any response from client-issued commands, the server may print
non-protocol output. It is common for SSH servers to print banners, message
of the day announcements, etc when clients connect. It is assumed that any
such *banner* output will precede any Mercurial server output. So clients
must be prepared to handle server output on initial connect that isn't
in response to any client-issued command and doesn't conform to Mercurial's
wire protocol. This *banner* output should only be on stdout. However,
some servers may send output on stderr.
Pre 0.9.1 clients issue a ``between`` command with the ``pairs`` argument
having the value
``0000000000000000000000000000000000000000-0000000000000000000000000000000000000000``.
The ``between`` command has been supported since the original Mercurial
SSH server. Requesting the empty range will return a ``\n`` string response,
which will be encoded as ``1\n\n`` (value length of ``1`` followed by a newline
followed by the value, which happens to be a newline).
For pre 0.9.1 clients and all servers, the exchange looks like::
c: between\n
c: pairs 81\n
c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
s: 1\n
s: \n
0.9.1+ clients send a ``hello`` command (with no arguments) before the
``between`` command. The response to this command allows clients to
discover server capabilities and settings.
An example exchange between 0.9.1+ clients and a ``hello`` aware server looks
like::
c: hello\n
c: between\n
c: pairs 81\n
c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
s: 324\n
s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n
s: 1\n
s: \n
And a similar scenario but with servers sending a banner on connect::
c: hello\n
c: between\n
c: pairs 81\n
c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
s: welcome to the server\n
s: if you find any issues, email someone@somewhere.com\n
s: 324\n
s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n
s: 1\n
s: \n
Note that output from the ``hello`` command is terminated by a ``\n``. This is
part of the response payload and not part of the wire protocol adding a newline
after responses. In other words, the length of the response contains the
trailing ``\n``.
SSH Version 1 Transport
-----------------------
The SSH transport (version 1) is a custom text-based protocol suitable for
use over any bi-directional stream transport. It is most commonly used with
SSH.
A SSH transport server can be started with ``hg serve --stdio``. The stdin, A SSH transport server can be started with ``hg serve --stdio``. The stdin,
stderr, and stdout file descriptors of the started process are used to exchange stderr, and stdout file descriptors of the started process are used to exchange
data. When Mercurial connects to a remote server over SSH, it actually starts data. When Mercurial connects to a remote server over SSH, it actually starts
a ``hg serve --stdio`` process on the remote server. a ``hg serve --stdio`` process on the remote server.
Commands are issued by sending the command name followed by a trailing newline Commands are issued by sending the command name followed by a trailing newline
``\n`` to the server. e.g. ``capabilities\n``. ``\n`` to the server. e.g. ``capabilities\n``.
July 2006). July 2006).
Mercurial 0.9.2 (released December 2006) added values to the capability Mercurial 0.9.2 (released December 2006) added values to the capability
indicating which bundle types the server supports receiving. This value is a indicating which bundle types the server supports receiving. This value is a
comma-delimited list. e.g. ``HG10GZ,HG10BZ,HG10UN``. The order of values comma-delimited list. e.g. ``HG10GZ,HG10BZ,HG10UN``. The order of values
reflects the priority/preference of that type, where the first value is the reflects the priority/preference of that type, where the first value is the
most preferred type. most preferred type.
Handshake Protocol
==================
While not explicitly required, it is common for clients to perform a
*handshake* when connecting to a server. The handshake accomplishes 2 things:
* Obtaining capabilities and other server features
* Flushing extra server output (e.g. SSH servers may print extra text
when connecting that may confuse the wire protocol)
This isn't a traditional *handshake* as far as network protocols go because
there is no persistent state as a result of the handshake: the handshake is
simply the issuing of commands and commands are stateless.
The canonical clients perform a capabilities lookup at connection establishment
time. This is because clients must assume a server only supports the features
of the original Mercurial server implementation until proven otherwise (from
advertised capabilities). Nearly every server running today supports features
that weren't present in the original Mercurial server implementation. Rather
than wait for a client to perform functionality that needs to consult
capabilities, it issues the lookup at connection start to avoid any delay later.
For HTTP servers, the client sends a ``capabilities`` command request as
soon as the connection is established. The server responds with a capabilities
string, which the client parses.
For SSH servers, the client sends the ``hello`` command (no arguments)
and a ``between`` command with the ``pairs`` argument having the value
``0000000000000000000000000000000000000000-0000000000000000000000000000000000000000``.
The ``between`` command has been supported since the original Mercurial
server. Requesting the empty range will return a ``\n`` string response,
which will be encoded as ``1\n\n`` (value length of ``1`` followed by a newline
followed by the value, which happens to be a newline).
The ``hello`` command was later introduced. Servers supporting it will issue
a response to that command before sending the ``1\n\n`` response to the
``between`` command. Servers not supporting ``hello`` will send an empty
response (``0\n``).
In addition to the expected output from the ``hello`` and ``between`` commands,
servers may also send other output, such as *message of the day (MOTD)*
announcements. Clients assume servers will send this output before the
Mercurial server replies to the client-issued commands. So any server output
not conforming to the expected command responses is assumed to be not related
to Mercurial and can be ignored.
Content Negotiation Content Negotiation
=================== ===================
The wire protocol has some mechanisms to help peers determine what content The wire protocol has some mechanisms to help peers determine what content
types and encoding the other side will accept. Historically, these mechanisms types and encoding the other side will accept. Historically, these mechanisms
have been built into commands themselves because most commands only send a have been built into commands themselves because most commands only send a
well-defined response type and only certain commands needed to support well-defined response type and only certain commands needed to support
functionality like compression. functionality like compression.
Currently, only the HTTP transport supports content negotiation at the protocol Currently, only the HTTP version 1 transport supports content negotiation
layer. at the protocol layer.
HTTP requests advertise supported response formats via the ``X-HgProto-<N>`` HTTP requests advertise supported response formats via the ``X-HgProto-<N>``
request header, where ``<N>`` is an integer starting at 1 allowing the logical request header, where ``<N>`` is an integer starting at 1 allowing the logical
value to span multiple headers. This value consists of a list of value to span multiple headers. This value consists of a list of
space-delimited parameters. Each parameter denotes a feature or capability. space-delimited parameters. Each parameter denotes a feature or capability.
The following parameters are defined: The following parameters are defined:
Boolean indicating whether the client attempted to use the *clone bundles* Boolean indicating whether the client attempted to use the *clone bundles*
feature before performing this request. feature before performing this request.
bookmarks bookmarks
Boolean indicating whether bookmark data is requested. Boolean indicating whether bookmark data is requested.
phases phases
Boolean indicating whether phases data is requested. Boolean indicating whether phases data is requested.
The return type on success is a ``stream`` where the value is bundle. The return type on success is a ``stream`` where the value is bundle.
On the HTTP transport, the response is zlib compressed. On the HTTP version 1 transport, the response is zlib compressed.
If an error occurs, a generic error response can be sent. If an error occurs, a generic error response can be sent.
Unless the client sends a false value for the ``cg`` argument, the returned Unless the client sends a false value for the ``cg`` argument, the returned
bundle contains a changegroup with the nodes between the specified ``common`` bundle contains a changegroup with the nodes between the specified ``common``
and ``heads`` nodes. Depending on the command arguments, the type and content and ``heads`` nodes. Depending on the command arguments, the type and content
of the returned bundle can vary significantly. of the returned bundle can vary significantly.
Accepts arguments ``namespace``, ``key``, ``old``, and ``new``, which Accepts arguments ``namespace``, ``key``, ``old``, and ``new``, which
correspond to the pushkey namespace to operate on, the key within that correspond to the pushkey namespace to operate on, the key within that
namespace to change, the old value (which may be empty), and the new value. namespace to change, the old value (which may be empty), and the new value.
All arguments are string types. All arguments are string types.
The return type is a ``string``. The value depends on the transport protocol. The return type is a ``string``. The value depends on the transport protocol.
The SSH transport sends a string encoded integer followed by a newline The SSH version 1 transport sends a string encoded integer followed by a
(``\n``) which indicates operation result. The server may send additional newline (``\n``) which indicates operation result. The server may send
output on the ``stderr`` stream that should be displayed to the user. additional output on the ``stderr`` stream that should be displayed to the
user.
The HTTP transport sends a string encoded integer followed by a newline
followed by additional server output that should be displayed to the user. The HTTP version 1 transport sends a string encoded integer followed by a
This may include output from hooks, etc. newline followed by additional server output that should be displayed to
the user. This may include output from hooks, etc.
The integer result varies by namespace. ``0`` means an error has occurred The integer result varies by namespace. ``0`` means an error has occurred
and there should be additional output to display to the user. and there should be additional output to display to the user.
stream_out stream_out
---------- ----------
Obtain *streaming clone* data. Obtain *streaming clone* data.
operation. A value of ``0`` indicates nothing changed. ``1`` means the number operation. A value of ``0`` indicates nothing changed. ``1`` means the number
of heads remained the same. Values ``2`` and larger indicate the number of of heads remained the same. Values ``2`` and larger indicate the number of
added heads minus 1. e.g. ``3`` means 2 heads were added. Negative values added heads minus 1. e.g. ``3`` means 2 heads were added. Negative values
indicate the number of fewer heads, also off by 1. e.g. ``-2`` means there indicate the number of fewer heads, also off by 1. e.g. ``-2`` means there
is 1 fewer head. is 1 fewer head.
The encoding of the ``push response`` type varies by transport. The encoding of the ``push response`` type varies by transport.
For the SSH transport, this type is composed of 2 ``string`` responses: an For the SSH version 1 transport, this type is composed of 2 ``string``
empty response (``0\n``) followed by the integer result value. e.g. responses: an empty response (``0\n``) followed by the integer result value.
``1\n2``. So the full response might be ``0\n1\n2``. e.g. ``1\n2``. So the full response might be ``0\n1\n2``.
For the HTTP transport, the response is a ``string`` type composed of an For the HTTP version 1 transport, the response is a ``string`` type composed
integer result value followed by a newline (``\n``) followed by string of an integer result value followed by a newline (``\n``) followed by string
content holding server output that should be displayed on the client (output content holding server output that should be displayed on the client (output
hooks, etc). hooks, etc).
In some cases, the server may respond with a ``bundle2`` bundle. In this In some cases, the server may respond with a ``bundle2`` bundle. In this
case, the response type is ``stream``. For the HTTP transport, the response case, the response type is ``stream``. For the HTTP version 1 transport, the
is zlib compressed. response is zlib compressed.
The server may also respond with a generic error type, which contains a string The server may also respond with a generic error type, which contains a string
indicating the failure. indicating the failure.