This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4915

wireprotov2: update stream encoding specification
ClosedPublic

Authored by indygreg on Oct 8 2018, 8:29 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGe2fe1074024c: wireprotov2: update stream encoding specification
Summary

The encoding of data within streams in the frame-based protocol is
not yet defined or implemented. This means that all data in wire
protocol version 2 is currently being sent out raw, without
compression. That's obviously not ideal.

This commit formalizes the beginnings of stream encoding support
in the protocol.

I suspect we'll change behavior substantially in the future. My goal
is to get something landed so we can use compression. We can build
out more robust support later.

Because the frame type ID changed, this is strictly BC. But existing
code wasn't using the frame. I'll bump the framing protocol version
later once code is introduced to use the new frame.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Oct 8 2018, 8:29 PM
Closed by commit rHGe2fe1074024c: wireprotov2: update stream encoding specification (authored by indygreg). · Explain WhyOct 10 2018, 10:57 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 11779

mercurial/help/internals/wireprotocolrpc.txt

they see fit, including not at all. they see fit, including not at all.
The string data describing the topic SHOULD be static strings to The string data describing the topic SHOULD be static strings to
facilitate receivers localizing that string data. The emitter facilitate receivers localizing that string data. The emitter
MUST normalize all string data to valid UTF-8 and receivers SHOULD MUST normalize all string data to valid UTF-8 and receivers SHOULD
validate that received data conforms to UTF-8. The topic name validate that received data conforms to UTF-8. The topic name
SHOULD be ASCII. SHOULD be ASCII.
Stream Encoding Settings (``0x08``) Sender Protocol Settings (``0x08``)
-----------------------------------
This frame type advertises the sender's support for various protocol and
stream level features. The data advertised in this frame is used to influence
subsequent behavior of the current frame exchange channel.
The frame payload consists of a CBOR map. It may contain the following
bytestring keys:
contentencodings
(array of bytestring) A list of content encodings supported by the
sender, in order of most to least preferred.
Peers are allowed to encode stream data using any of the listed
encodings.
See the ``Content Encoding Profiles`` section for an enumeration
of supported content encodings.
If not defined, the value is assumed to be a list with the single value
``identity``, meaning only the no-op encoding is supported.
Senders MAY filter the set of advertised encodings against what it
knows the receiver supports (e.g. if the receiver advertised encodings
via the capabilities descriptor). However, doing so will prevent
servers from gaining an understanding of the aggregate capabilities
of clients. So clients are discouraged from doing so.
When this frame is not sent/received, the receiver assumes default values
for all keys.
If encountered, this frame type MUST be sent before any other frame type
in a channel.
The following flag values are defined for this frame type:
0x01
Data continuation. When set, an additional frame containing more protocol
settings immediately follows.
0x02
End of data. When set, the protocol settings data has been completely
sent.
The ``0x01`` flag is mutually exclusive with the ``0x02`` flag.
Stream Encoding Settings (``0x09``)
----------------------------------- -----------------------------------
This frame type holds information defining the content encoding This frame type holds information defining the content encoding
settings for a *stream*. settings for a *stream*.
This frame type is likely consumed by the protocol layer and is not This frame type is likely consumed by the protocol layer and is not
passed on to applications. passed on to applications.
This frame type MUST ONLY occur on frames having the *Beginning of Stream* This frame type MUST ONLY occur on frames having the *Beginning of Stream*
``Stream Flag`` set. ``Stream Flag`` set.
The payload of this frame defines what content encoding has (possibly) The payload of this frame defines what content encoding has (possibly)
been applied to the payloads of subsequent frames in this stream. been applied to the payloads of subsequent frames in this stream.
The payload begins with an 8-bit integer defining the length of the The payload consists of a series of CBOR values. The first value is a
encoding *profile*, followed by the string name of that profile, which bytestring denoting the content encoding profile of the data in this
must be an ASCII string. All bytes that follow can be used by that stream. Subsequent CBOR values supplement this simple value in a
profile for supplemental settings definitions. See the section below profile-specific manner. See the ``Content Encoding Profiles`` section
on defined encoding profiles. for more.
In the absence of this frame on a stream, it is assumed the stream is
using the ``identity`` content encoding.
The following flag values are defined for this frame type:
0x01
Data continuation. When set, an additional frame containing more encoding
settings immediately follows.
0x02
End of data. When set, the encoding settings data has been completely
sent.
The ``0x01`` flag is mutually exclusive with the ``0x02`` flag.
Stream States and Flags Stream States and Flags
======================= =======================
Streams can be in two states: *open* and *closed*. An *open* stream Streams can be in two states: *open* and *closed*. An *open* stream
is active and frames attached to that stream could arrive at any time. is active and frames attached to that stream could arrive at any time.
A *closed* stream is not active. If a frame attached to a *closed* A *closed* stream is not active. If a frame attached to a *closed*
stream arrives, that frame MUST have an appropriate stream flag stream arrives, that frame MUST have an appropriate stream flag
``closed``. Any content encoding context associated with this stream ``closed``. Any content encoding context associated with this stream
can be destroyed after processing the payload of this frame. can be destroyed after processing the payload of this frame.
0x04 0x04
Apply content encoding. When set, any content encoding settings Apply content encoding. When set, any content encoding settings
defined by the stream should be applied when attempting to read defined by the stream should be applied when attempting to read
the frame. When not set, the frame payload isn't encoded. the frame. When not set, the frame payload isn't encoded.
TODO consider making stream opening and closing communicated via
explicit frame types (e.g. a "stream state change" frame) rather than
flags on all frames. This would make stream state changes more explicit,
as they could only occur on specific frame types.
Streams Streams
======= =======
Streams - along with ``Request IDs`` - facilitate grouping of frames. Streams - along with ``Request IDs`` - facilitate grouping of frames.
But the purpose of each is quite different and the groupings they But the purpose of each is quite different and the groupings they
constitute are independent. constitute are independent.
A ``Request ID`` is essentially a tag. It tells you which logical A ``Request ID`` is essentially a tag. It tells you which logical
Content Encoding Profiles Content Encoding Profiles
========================= =========================
Streams can have named content encoding *profiles* associated with Streams can have named content encoding *profiles* associated with
them. A profile defines a shared understanding of content encoding them. A profile defines a shared understanding of content encoding
settings and behavior. settings and behavior.
The following profiles are defined: Profiles are described in the following sections.
identity
--------
The ``identity`` profile is a no-op encoding: the encoded bytes are
exactly the input bytes.
This profile MUST be supported by all peers.
In the absence of an identified profile, the ``identity`` profile is
assumed.
zstd-8mb
--------
Zstandard encoding (RFC 8478). Zstandard is a fast and effective lossless
compression format.
This profile allows decompressor window sizes of up to 8 MB.
zlib
----
zlib compressed data (RFC 1950). zlib is a widely-used and supported
lossless compression format.
TBD It isn't as fast as zstandard and it is recommended to use zstandard instead,
if possible.
Command Protocol Command Protocol
================ ================
A client can request that a remote run a command by sending it A client can request that a remote run a command by sending it
frames defining that command. This logical stream is composed of frames defining that command. This logical stream is composed of
1 or more ``Command Request`` frames and and 0 or more ``Command Data`` 1 or more ``Command Request`` frames and and 0 or more ``Command Data``
frames. frames.

mercurial/wireprotoframing.py

} }
FRAME_TYPE_COMMAND_REQUEST = 0x01 FRAME_TYPE_COMMAND_REQUEST = 0x01
FRAME_TYPE_COMMAND_DATA = 0x02 FRAME_TYPE_COMMAND_DATA = 0x02
FRAME_TYPE_COMMAND_RESPONSE = 0x03 FRAME_TYPE_COMMAND_RESPONSE = 0x03
FRAME_TYPE_ERROR_RESPONSE = 0x05 FRAME_TYPE_ERROR_RESPONSE = 0x05
FRAME_TYPE_TEXT_OUTPUT = 0x06 FRAME_TYPE_TEXT_OUTPUT = 0x06
FRAME_TYPE_PROGRESS = 0x07 FRAME_TYPE_PROGRESS = 0x07
FRAME_TYPE_STREAM_SETTINGS = 0x08 FRAME_TYPE_SENDER_PROTOCOL_SETTINGS = 0x08
FRAME_TYPE_STREAM_SETTINGS = 0x09
FRAME_TYPES = { FRAME_TYPES = {
b'command-request': FRAME_TYPE_COMMAND_REQUEST, b'command-request': FRAME_TYPE_COMMAND_REQUEST,
b'command-data': FRAME_TYPE_COMMAND_DATA, b'command-data': FRAME_TYPE_COMMAND_DATA,
b'command-response': FRAME_TYPE_COMMAND_RESPONSE, b'command-response': FRAME_TYPE_COMMAND_RESPONSE,
b'error-response': FRAME_TYPE_ERROR_RESPONSE, b'error-response': FRAME_TYPE_ERROR_RESPONSE,
b'text-output': FRAME_TYPE_TEXT_OUTPUT, b'text-output': FRAME_TYPE_TEXT_OUTPUT,
b'progress': FRAME_TYPE_PROGRESS, b'progress': FRAME_TYPE_PROGRESS,
b'sender-protocol-settings': FRAME_TYPE_SENDER_PROTOCOL_SETTINGS,
b'stream-settings': FRAME_TYPE_STREAM_SETTINGS, b'stream-settings': FRAME_TYPE_STREAM_SETTINGS,
} }
FLAG_COMMAND_REQUEST_NEW = 0x01 FLAG_COMMAND_REQUEST_NEW = 0x01
FLAG_COMMAND_REQUEST_CONTINUATION = 0x02 FLAG_COMMAND_REQUEST_CONTINUATION = 0x02
FLAG_COMMAND_REQUEST_MORE_FRAMES = 0x04 FLAG_COMMAND_REQUEST_MORE_FRAMES = 0x04
FLAG_COMMAND_REQUEST_EXPECT_DATA = 0x08 FLAG_COMMAND_REQUEST_EXPECT_DATA = 0x08
FLAG_COMMAND_RESPONSE_CONTINUATION = 0x01 FLAG_COMMAND_RESPONSE_CONTINUATION = 0x01
FLAG_COMMAND_RESPONSE_EOS = 0x02 FLAG_COMMAND_RESPONSE_EOS = 0x02
FLAGS_COMMAND_RESPONSE = { FLAGS_COMMAND_RESPONSE = {
b'continuation': FLAG_COMMAND_RESPONSE_CONTINUATION, b'continuation': FLAG_COMMAND_RESPONSE_CONTINUATION,
b'eos': FLAG_COMMAND_RESPONSE_EOS, b'eos': FLAG_COMMAND_RESPONSE_EOS,
} }
FLAG_SENDER_PROTOCOL_SETTINGS_CONTINUATION = 0x01
FLAG_SENDER_PROTOCOL_SETTINGS_EOS = 0x02
FLAGS_SENDER_PROTOCOL_SETTINGS = {
b'continuation': FLAG_SENDER_PROTOCOL_SETTINGS_CONTINUATION,
b'eos': FLAG_SENDER_PROTOCOL_SETTINGS_EOS,
}
FLAG_STREAM_ENCODING_SETTINGS_CONTINUATION = 0x01
FLAG_STREAM_ENCODING_SETTINGS_EOS = 0x02
FLAGS_STREAM_ENCODING_SETTINGS = {
b'continuation': FLAG_STREAM_ENCODING_SETTINGS_CONTINUATION,
b'eos': FLAG_STREAM_ENCODING_SETTINGS_EOS,
}
# Maps frame types to their available flags. # Maps frame types to their available flags.
FRAME_TYPE_FLAGS = { FRAME_TYPE_FLAGS = {
FRAME_TYPE_COMMAND_REQUEST: FLAGS_COMMAND_REQUEST, FRAME_TYPE_COMMAND_REQUEST: FLAGS_COMMAND_REQUEST,
FRAME_TYPE_COMMAND_DATA: FLAGS_COMMAND_DATA, FRAME_TYPE_COMMAND_DATA: FLAGS_COMMAND_DATA,
FRAME_TYPE_COMMAND_RESPONSE: FLAGS_COMMAND_RESPONSE, FRAME_TYPE_COMMAND_RESPONSE: FLAGS_COMMAND_RESPONSE,
FRAME_TYPE_ERROR_RESPONSE: {}, FRAME_TYPE_ERROR_RESPONSE: {},
FRAME_TYPE_TEXT_OUTPUT: {}, FRAME_TYPE_TEXT_OUTPUT: {},
FRAME_TYPE_PROGRESS: {}, FRAME_TYPE_PROGRESS: {},
FRAME_TYPE_STREAM_SETTINGS: {}, FRAME_TYPE_SENDER_PROTOCOL_SETTINGS: FLAGS_SENDER_PROTOCOL_SETTINGS,
FRAME_TYPE_STREAM_SETTINGS: FLAGS_STREAM_ENCODING_SETTINGS,
} }
ARGUMENT_RECORD_HEADER = struct.Struct(r'<HH') ARGUMENT_RECORD_HEADER = struct.Struct(r'<HH')
def humanflags(mapping, value): def humanflags(mapping, value):
"""Convert a numeric flags value to a human value, using a mapping table.""" """Convert a numeric flags value to a human value, using a mapping table."""
namemap = {v: k for k, v in mapping.iteritems()} namemap = {v: k for k, v in mapping.iteritems()}
flags = [] flags = []