( )⚙ D2902 wireproto: define frame to represent progress updates

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2902

wireproto: define frame to represent progress updates
ClosedPublic

Authored by indygreg on Mar 19 2018, 7:59 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGb0041036214e: wireproto: define frame to represent progress updates
Summary

Today, a long-running operation on a server may run without any sign
of progress on the client. This can lead to the conclusion that the
server has hung or the connection has dropped. In fact, connections
can and do time out due to inactivity. And a long-running server
operation can result in the connection dropping prematurely because
no data is being sent!

While we're inventing the new wire protocol, let's provide a mechanism
for communicating progress on potentially expensive server-side events.

We introduce a new frame type that conveys "progress" updates. This
frame type essentially holds the data required to formulate a
`ui.progress()` call.

We only define the frame right now. Implementing it will be a bit of
work since there is no analog to progress frames in the existing
wire protocol. We'll need to teach the ui object to write to the
wire protocol, etc.

The use of a CBOR map may seem wasteful, as this will encode key
names in every frame. This *is* wasteful. However, maps are
extensible. And the intent is to always use compression via
streams. Compression will make the overhead negligible since repeated
strings will be mostly eliminated over the wire.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Mar 19 2018, 7:59 PM
indygreg edited the summary of this revision. (Show Details)Mar 26 2018, 12:39 PM
indygreg updated this revision to Diff 7295.
indygreg edited the summary of this revision. (Show Details)Mar 26 2018, 2:13 PM
indygreg updated this revision to Diff 7311.
indygreg updated this revision to Diff 7438.Mar 30 2018, 4:32 PM
durin42 accepted this revision.Apr 3 2018, 11:36 AM
This revision is now accepted and ready to land.Apr 3 2018, 11:36 AM
Closed by commit rHGb0041036214e: wireproto: define frame to represent progress updates (authored by indygreg). · Explain WhyApr 3 2018, 12:50 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 7151

mercurial/help/internals/wireprotocol.txt

All data to be printed MUST be encoded into a single frame: this frame All data to be printed MUST be encoded into a single frame: this frame
does not support spanning data across multiple frames. does not support spanning data across multiple frames.
All textual data encoded in these frames is assumed to be line delimited. All textual data encoded in these frames is assumed to be line delimited.
The last atom in the frame SHOULD end with a newline (``\n``). If it The last atom in the frame SHOULD end with a newline (``\n``). If it
doesn't, clients MAY add a newline to facilitate immediate printing. doesn't, clients MAY add a newline to facilitate immediate printing.
Progress Update (``0x07``)
--------------------------
This frame holds the progress of an operation on the peer. Consumption
of these frames allows clients to display progress bars, estimated
completion times, etc.
Each frame defines the progress of a single operation on the peer. The
payload consists of:
* A 64-bit signed little-endian integer denoting the numeric position
of this topic.
* A 64-bit unsigned little-endian integer denoting the total/end
numeric position of this topic.
* An 8-bit unsigned integer denoting the string length of the topic
that is being updated.
* An 8-bit unsigned integer denoting the string length of the unit
label.
* A 16-bit unsigned little-endian integer denoting the string length
of the item currently being processed.
* 1 or more bytes containing the topic name, interpreted as UTF-8.
* 0 or more bytes containing the unit label, interpreted as UTF-8.
* 0 or more bytes containing the item name, interpreted as UTF-8.
Progress state is created when a frame is received referencing a
*topic* that isn't currently tracked. Progress tracking for that
*topic* is finished when a frame is received reporting the current
position of that topic as ``-1``.
Multiple *topics* may be active at any given time.
Rendering of progress information is not mandated or governed by this
specification: implementations MAY render progress information however
they see fit, including not at all.
The string data describing the topic SHOULD be static strings to
facilitate receivers localizing that string data. The emitter
MUST normalize all string data to valid UTF-8 and receivers SHOULD
validate that received data conforms to UTF-8.
Issuing Commands Issuing Commands
---------------- ----------------
A client can request that a remote run a command by sending it A client can request that a remote run a command by sending it
frames defining that command. This logical stream is composed of frames defining that command. This logical stream is composed of
1 ``Command Request`` frame, 0 or more ``Command Argument`` frames, 1 ``Command Request`` frame, 0 or more ``Command Argument`` frames,
and 0 or more ``Command Data`` frames. and 0 or more ``Command Data`` frames.

mercurial/wireprotoframing.py

DEFAULT_MAX_FRAME_SIZE = 32768 DEFAULT_MAX_FRAME_SIZE = 32768
FRAME_TYPE_COMMAND_NAME = 0x01 FRAME_TYPE_COMMAND_NAME = 0x01
FRAME_TYPE_COMMAND_ARGUMENT = 0x02 FRAME_TYPE_COMMAND_ARGUMENT = 0x02
FRAME_TYPE_COMMAND_DATA = 0x03 FRAME_TYPE_COMMAND_DATA = 0x03
FRAME_TYPE_BYTES_RESPONSE = 0x04 FRAME_TYPE_BYTES_RESPONSE = 0x04
FRAME_TYPE_ERROR_RESPONSE = 0x05 FRAME_TYPE_ERROR_RESPONSE = 0x05
FRAME_TYPE_TEXT_OUTPUT = 0x06 FRAME_TYPE_TEXT_OUTPUT = 0x06
FRAME_TYPE_PROGRESS = 0x07
FRAME_TYPES = { FRAME_TYPES = {
b'command-name': FRAME_TYPE_COMMAND_NAME, b'command-name': FRAME_TYPE_COMMAND_NAME,
b'command-argument': FRAME_TYPE_COMMAND_ARGUMENT, b'command-argument': FRAME_TYPE_COMMAND_ARGUMENT,
b'command-data': FRAME_TYPE_COMMAND_DATA, b'command-data': FRAME_TYPE_COMMAND_DATA,
b'bytes-response': FRAME_TYPE_BYTES_RESPONSE, b'bytes-response': FRAME_TYPE_BYTES_RESPONSE,
b'error-response': FRAME_TYPE_ERROR_RESPONSE, b'error-response': FRAME_TYPE_ERROR_RESPONSE,
b'text-output': FRAME_TYPE_TEXT_OUTPUT, b'text-output': FRAME_TYPE_TEXT_OUTPUT,
b'progress': FRAME_TYPE_PROGRESS,
} }
FLAG_COMMAND_NAME_EOS = 0x01 FLAG_COMMAND_NAME_EOS = 0x01
FLAG_COMMAND_NAME_HAVE_ARGS = 0x02 FLAG_COMMAND_NAME_HAVE_ARGS = 0x02
FLAG_COMMAND_NAME_HAVE_DATA = 0x04 FLAG_COMMAND_NAME_HAVE_DATA = 0x04
FLAGS_COMMAND = { FLAGS_COMMAND = {
b'eos': FLAG_COMMAND_NAME_EOS, b'eos': FLAG_COMMAND_NAME_EOS,
# Maps frame types to their available flags. # Maps frame types to their available flags.
FRAME_TYPE_FLAGS = { FRAME_TYPE_FLAGS = {
FRAME_TYPE_COMMAND_NAME: FLAGS_COMMAND, FRAME_TYPE_COMMAND_NAME: FLAGS_COMMAND,
FRAME_TYPE_COMMAND_ARGUMENT: FLAGS_COMMAND_ARGUMENT, FRAME_TYPE_COMMAND_ARGUMENT: FLAGS_COMMAND_ARGUMENT,
FRAME_TYPE_COMMAND_DATA: FLAGS_COMMAND_DATA, FRAME_TYPE_COMMAND_DATA: FLAGS_COMMAND_DATA,
FRAME_TYPE_BYTES_RESPONSE: FLAGS_BYTES_RESPONSE, FRAME_TYPE_BYTES_RESPONSE: FLAGS_BYTES_RESPONSE,
FRAME_TYPE_ERROR_RESPONSE: FLAGS_ERROR_RESPONSE, FRAME_TYPE_ERROR_RESPONSE: FLAGS_ERROR_RESPONSE,
FRAME_TYPE_TEXT_OUTPUT: {}, FRAME_TYPE_TEXT_OUTPUT: {},
FRAME_TYPE_PROGRESS: {},
} }
ARGUMENT_FRAME_HEADER = struct.Struct(r'<HH') ARGUMENT_FRAME_HEADER = struct.Struct(r'<HH')
@attr.s(slots=True) @attr.s(slots=True)
class frameheader(object): class frameheader(object):
"""Represents the data in a frame header.""" """Represents the data in a frame header."""