This is an archive of the discontinued Mercurial Phabricator instance.

wireproto: define and implement unified framing protocol for SSH
AbandonedPublic

Authored by indygreg on Mar 3 2018, 6:38 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

THIS COMMIT IS INCOMPLETE AND TESTS STILL FAIL. DO NOT COMMIT.

(This is probably one of the largest patches you'll ever see me write.
Sorry about that: it is difficult to implement all of this
functionality across multiple commits.)

The existing HTTP and SSH wire protocols suffer from a host of flaws
and shortcomings. I've been wanting to rewrite the protocol for a while
now. Supporting partial clone - which will require new wire protocol
commands and capabilities - and other advanced server functionality
will be much easier if we start from a clean slate and don't have
to be constrained by limitations of the existing wire protocol.

This commit introduces a rewrite of the wire protocol. The current
implementation is very, very, very far from what I want the final
implementation to look like. It hasn't begun to tackle things like
compression, multi part responses, various side-channels, etc.

Anyway, the goal of this initial commit is to create a clean break
from the existing SSH wire protocol so we have *something* better.
And that *something* will gradually evolve over several backwards
incompatible changes.

As the updated internals documentation states, the new protocol
attempts to be transport agnostic: all that's required to run the
protocol is a pair of unidirectional, half-duplex pipes. While
we have only implemented the protocol for SSH, we eventually want
to "tunnel" this protocol over HTTP. (For HTTP I anticipate
supporting both exchanging this protocol format via HTTP message
bodies as well as leveraging the Upgrade header to switch from HTTP
to the new frame-based protocol - thus effectively making HTTP
connections behave identically to SSH. But this is for a future
commit.) One of the differences with this protocol for SSH is
it doesn't use stderr. This is by design: by limiting ourselves
to only a pair of pipes, we make it possible to speak this protocol
across any transport channel that supports ordered message delivery,
including HTTP.

The new protocol is built on top of "frames." A frame is an atomic
unit (kind of like a TCP packet). It has a header and data payload.
The header is currently extremely simple and will evolve significantly
over time. The header records the size of the payload, the frame
type, and per-type bit flags to denote frame behavior.

The defined frame types are based on features of the existing
wire protocol. You have frames to communicate a request to run a
command: a command frame, argument frame, and command data frame.
There are also frames to represent the response to that request:
a raw data frame, an error frame, and frames for sending server
output. I anticipate that the frame types will evolve heavily over
time. It is difficult to do anything too radical at this stage because
we need to concurrently support both the old and new protocols and
the client and server APIs still need to evolve a ways before we
can do crazier things.

Because the new protocol aims to be transport agnostic, we have
generic implementations for the peer and server. The SSH peer and
server are very thin wrappers around the generic implementation.

The server is implemented as a state machine of sorts. It basically
sits around, waiting to receive a frame. Once it has received a
request to run a command, it dispatches to the wire command handler
and then sends a response. It is a very traditional half-duplex
request-response protocol, not unlike HTTP/1.1. I anticipate
evolving the protocol to support full-duplex operations. But that's
way too complicated for the initial implementation.

The new protocol is fully functional over SSH. Support for speaking
this protocol over HTTP will be added later.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Mar 3 2018, 6:38 PM
indygreg abandoned this revision.Mar 13 2018, 6:25 PM

I'll break this up and send it as multiple commits.