( )⚙ D2902 wireproto: define frame to represent progress updates

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2902

wireproto: define frame to represent progress updates
ClosedPublic

Authored by indygreg on Mar 19 2018, 7:59 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGb0041036214e: wireproto: define frame to represent progress updates
Summary

Today, a long-running operation on a server may run without any sign
of progress on the client. This can lead to the conclusion that the
server has hung or the connection has dropped. In fact, connections
can and do time out due to inactivity. And a long-running server
operation can result in the connection dropping prematurely because
no data is being sent!

While we're inventing the new wire protocol, let's provide a mechanism
for communicating progress on potentially expensive server-side events.

We introduce a new frame type that conveys "progress" updates. This
frame type essentially holds the data required to formulate a
`ui.progress()` call.

We only define the frame right now. Implementing it will be a bit of
work since there is no analog to progress frames in the existing
wire protocol. We'll need to teach the ui object to write to the
wire protocol, etc.

The use of a CBOR map may seem wasteful, as this will encode key
names in every frame. This *is* wasteful. However, maps are
extensible. And the intent is to always use compression via
streams. Compression will make the overhead negligible since repeated
strings will be mostly eliminated over the wire.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 19 2018, 7:59 PM
indygreg edited the summary of this revision. (Show Details)Mar 26 2018, 12:39 PM
indygreg updated this revision to Diff 7295.
indygreg edited the summary of this revision. (Show Details)Mar 26 2018, 2:13 PM
indygreg updated this revision to Diff 7311.
indygreg updated this revision to Diff 7438.Mar 30 2018, 4:32 PM
durin42 accepted this revision.Apr 3 2018, 11:36 AM
This revision is now accepted and ready to land.Apr 3 2018, 11:36 AM
Closed by commit rHGb0041036214e: wireproto: define frame to represent progress updates (authored by indygreg). · Explain WhyApr 3 2018, 12:50 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 7552

mercurial/help/internals/wireprotocol.txt

All data to be printed MUST be encoded into a single frame: this frame All data to be printed MUST be encoded into a single frame: this frame
does not support spanning data across multiple frames. does not support spanning data across multiple frames.
All textual data encoded in these frames is assumed to be line delimited. All textual data encoded in these frames is assumed to be line delimited.
The last atom in the frame SHOULD end with a newline (``\n``). If it The last atom in the frame SHOULD end with a newline (``\n``). If it
doesn't, clients MAY add a newline to facilitate immediate printing. doesn't, clients MAY add a newline to facilitate immediate printing.
Progress Update (``0x07``)
--------------------------
This frame holds the progress of an operation on the peer. Consumption
of these frames allows clients to display progress bars, estimated
completion times, etc.
Each frame defines the progress of a single operation on the peer. The
payload consists of a CBOR map with the following bytestring keys:
topic
Topic name (string)
pos
Current numeric position within the topic (integer)
total
Total/end numeric position of this topic (unsigned integer)
label (optional)
Unit label (string)
item (optional)
Item name (string)
Progress state is created when a frame is received referencing a
*topic* that isn't currently tracked. Progress tracking for that
*topic* is finished when a frame is received reporting the current
position of that topic as ``-1``.
Multiple *topics* may be active at any given time.
Rendering of progress information is not mandated or governed by this
specification: implementations MAY render progress information however
they see fit, including not at all.
The string data describing the topic SHOULD be static strings to
facilitate receivers localizing that string data. The emitter
MUST normalize all string data to valid UTF-8 and receivers SHOULD
validate that received data conforms to UTF-8. The topic name
SHOULD be ASCII.
Stream Encoding Settings (``0x08``) Stream Encoding Settings (``0x08``)
----------------------------------- -----------------------------------
This frame type holds information defining the content encoding This frame type holds information defining the content encoding
settings for a *stream*. settings for a *stream*.
This frame type is likely consumed by the protocol layer and is not This frame type is likely consumed by the protocol layer and is not
passed on to applications. passed on to applications.

mercurial/wireprotoframing.py

} }
FRAME_TYPE_COMMAND_NAME = 0x01 FRAME_TYPE_COMMAND_NAME = 0x01
FRAME_TYPE_COMMAND_ARGUMENT = 0x02 FRAME_TYPE_COMMAND_ARGUMENT = 0x02
FRAME_TYPE_COMMAND_DATA = 0x03 FRAME_TYPE_COMMAND_DATA = 0x03
FRAME_TYPE_BYTES_RESPONSE = 0x04 FRAME_TYPE_BYTES_RESPONSE = 0x04
FRAME_TYPE_ERROR_RESPONSE = 0x05 FRAME_TYPE_ERROR_RESPONSE = 0x05
FRAME_TYPE_TEXT_OUTPUT = 0x06 FRAME_TYPE_TEXT_OUTPUT = 0x06
FRAME_TYPE_PROGRESS = 0x07
FRAME_TYPE_STREAM_SETTINGS = 0x08 FRAME_TYPE_STREAM_SETTINGS = 0x08
FRAME_TYPES = { FRAME_TYPES = {
b'command-name': FRAME_TYPE_COMMAND_NAME, b'command-name': FRAME_TYPE_COMMAND_NAME,
b'command-argument': FRAME_TYPE_COMMAND_ARGUMENT, b'command-argument': FRAME_TYPE_COMMAND_ARGUMENT,
b'command-data': FRAME_TYPE_COMMAND_DATA, b'command-data': FRAME_TYPE_COMMAND_DATA,
b'bytes-response': FRAME_TYPE_BYTES_RESPONSE, b'bytes-response': FRAME_TYPE_BYTES_RESPONSE,
b'error-response': FRAME_TYPE_ERROR_RESPONSE, b'error-response': FRAME_TYPE_ERROR_RESPONSE,
b'text-output': FRAME_TYPE_TEXT_OUTPUT, b'text-output': FRAME_TYPE_TEXT_OUTPUT,
b'progress': FRAME_TYPE_PROGRESS,
b'stream-settings': FRAME_TYPE_STREAM_SETTINGS, b'stream-settings': FRAME_TYPE_STREAM_SETTINGS,
} }
FLAG_COMMAND_NAME_EOS = 0x01 FLAG_COMMAND_NAME_EOS = 0x01
FLAG_COMMAND_NAME_HAVE_ARGS = 0x02 FLAG_COMMAND_NAME_HAVE_ARGS = 0x02
FLAG_COMMAND_NAME_HAVE_DATA = 0x04 FLAG_COMMAND_NAME_HAVE_DATA = 0x04
FLAGS_COMMAND = { FLAGS_COMMAND = {
# Maps frame types to their available flags. # Maps frame types to their available flags.
FRAME_TYPE_FLAGS = { FRAME_TYPE_FLAGS = {
FRAME_TYPE_COMMAND_NAME: FLAGS_COMMAND, FRAME_TYPE_COMMAND_NAME: FLAGS_COMMAND,
FRAME_TYPE_COMMAND_ARGUMENT: FLAGS_COMMAND_ARGUMENT, FRAME_TYPE_COMMAND_ARGUMENT: FLAGS_COMMAND_ARGUMENT,
FRAME_TYPE_COMMAND_DATA: FLAGS_COMMAND_DATA, FRAME_TYPE_COMMAND_DATA: FLAGS_COMMAND_DATA,
FRAME_TYPE_BYTES_RESPONSE: FLAGS_BYTES_RESPONSE, FRAME_TYPE_BYTES_RESPONSE: FLAGS_BYTES_RESPONSE,
FRAME_TYPE_ERROR_RESPONSE: FLAGS_ERROR_RESPONSE, FRAME_TYPE_ERROR_RESPONSE: FLAGS_ERROR_RESPONSE,
FRAME_TYPE_TEXT_OUTPUT: {}, FRAME_TYPE_TEXT_OUTPUT: {},
FRAME_TYPE_PROGRESS: {},
FRAME_TYPE_STREAM_SETTINGS: {}, FRAME_TYPE_STREAM_SETTINGS: {},
} }
ARGUMENT_FRAME_HEADER = struct.Struct(r'<HH') ARGUMENT_FRAME_HEADER = struct.Struct(r'<HH')
@attr.s(slots=True) @attr.s(slots=True)
class frameheader(object): class frameheader(object):
"""Represents the data in a frame header.""" """Represents the data in a frame header."""