This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3535

wireprotov2: define and implement "filerevisionsslice" command
AbandonedPublic

Authored by indygreg on May 11 2018, 6:35 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

This command can be thought of as a batch version of "filerevision."
Clients specify a changeset and a pair of paths. For each path within
that range, file revision data is retrieved.

While the new wire protocol is very efficient at processing thousands
of command requests, batch commands are often intrinsically more
efficient. In this particular case, this batch command prevents the
client from having to transmit potentially hundreds of thousands or
even millions of file names and file nodes. Instead, it can encapsulate
that request in a fraction of the wire size.

On the server side, batch requests such as this command allow the
server to perform a multi-get fetch from storage. Compared to a
wire protocol command per file, this should be much easier to
optimize and be far more efficient on servers.

This command is theoretically compatible with narrow clones composed
of any set of files. We don't to employ a matcher to limit which files
are retrieved because clients can construct an arbitrary combination of
"filerevisionsslice" commands to request the relevant set. That being
said, we may want to implement a filtering mechanism on this command
someday. I'm inclined to add this as a follow-up.

On the Firefox repository, a "filerevisionsslice" request for the full
file range takes a server ~33s CPU time on my i7-6700K. That's without
any optimizations. Profiling indicates most of that time is spent in
revlog code opening revlogs and resolving fulltext data. I have
experimented with formalizing a more advanced file storage interface
that supports multi-get and hooking a Redis cache into that interface
and the results are *very* promising. But ~33s CPU time to serve
effectively a full revision checkout is a pretty good place to start!

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.May 11 2018, 6:35 PM
indygreg abandoned this revision.Aug 21 2018, 4:23 PM

Revision Contents
Changeset List

Diff 8646

mercurial/help/internals/wireprotocol.txt

allow retrieval of LFS blobs via this API or should the raw revision data allow retrieval of LFS blobs via this API or should the raw revision data
be the LFS data and we send a flag indicating it is to be interpreted as be the LFS data and we send a flag indicating it is to be interpreted as
LFS. LFS.
The file revision data is the raw data as tracked by Mercurial. In The file revision data is the raw data as tracked by Mercurial. In
addition to the raw file data, it may contain a header defining metadata addition to the raw file data, it may contain a header defining metadata
such as copy information. such as copy information.
filerevisionsslice
------------------
Obtain fulltext file revision data for a set of files on a specific revision.
This command allows the caller to obtain the fulltext file revision data
for 1 or more file paths, specified as a range between two paths.
The command accepts the following arguments:
node
(bytestring) Changeset revision whose file data to obtain.
startpath
(bytestring) The tracked path at which file iteration should begin.
endpath
(bytestring) The tracked path at which file iteration should end.
This command effectively opens the manifest for the requested start
revision, *seeks* to ``startpath`` and then starts iterating paths until
it reaches ``endpath``. For each path encountered - including ``startpath``
and ``endpath`` themselves - fulltext file revision data is obtained.
The response bytestream starts with a CBOR map that defines the payload
to follow. This map has the following bytestring keys:
totalitems (optional)
(unsigned integer) Total number of files whose data will be returned.
totalrevisionsize (optional)
(unsigned integer) Total size of all file revisions data, in bytes.
Following the CBOR map header are a series of segments. Each segments consists
of a CBOR map followed by a bytestring, possibly of indefinite length,
containing the revision data. The CBOR map has the following bytestring keys:
path
(bytestring) Tracked file path.
node
(bytestring) File revision (what should appear in manifest).
censored (optional)
(boolean) Indicates whether the revision data has been censored. If true,
``size`` should be 0. An empty bytestring will follow this CBOR map.
renamed (optional)
(array) If present, will be an array of 2 bytestrings defining the
copy/rename metadata for this revision. The first element is the path
the file was copied from. The second is its node revision.
size
(unsigned integer) The size of the revision's fulltext data. The size is
not the count of bytes that follow this CBOR map, as the revision data
is CBOR encoded.
Revision data may contain a header with metadata. So a consumer cannot stream
the revision data to a file without first processing the stream looking
for this header.
heads heads
----- -----
Obtain DAG heads in the repository. Obtain DAG heads in the repository.
The command accepts the following arguments: The command accepts the following arguments:
publiconly (optional) publiconly (optional)

mercurial/wireprotov2server.py

# force this on the client. So split into an indefinite length # force this on the client. So split into an indefinite length
# bytestring and send those chunks. # bytestring and send those chunks.
for chunk in cborutil.streamencodeindefinitebytestring(fulltext): for chunk in cborutil.streamencodeindefinitebytestring(fulltext):
yield chunk yield chunk
return wireprototypes.v2streamingresponse(senddata(), return wireprototypes.v2streamingresponse(senddata(),
compressible=True) compressible=True)
@wireprotocommand('filerevisionsslice',
args={
'node': b'0123456789abcdef...',
'startpath': b'a/foo',
'endpath': b'b/bar',
},
permission='pull')
def filerevisionsslice(repo, proto, node, startpath, endpath):
# TODO consider advertising a limit for maximum slice size to limit work
# per invocation.
try:
ctx = repo[node]
except error.RepoLookupError:
return wireprototypes.v2errorresponse(
_('unknown revision'))
if startpath > endpath:
startpath, endpath = endpath, startpath
m = ctx.manifest()
# Find relevant files.
# TODO linear scans of manifests are bad. Manifest needs to grow an API
# that allows iteration starting from a key.
relevantpaths = []
for path, fnode in m.iteritems():
if path < startpath:
continue
relevantpaths.append((path, fnode))
if path >= endpath:
break
def senddata():
chunks = cborutil.streamencodemap({
'totalitems': len(relevantpaths),
})
for chunk in chunks:
yield chunk
for path, fnode in relevantpaths:
res = {
'path': path,
'node': fnode,
}
fl = repo.file(path)
rev = fl.rev(fnode)
if fl.iscensored(rev):
res['censored'] = True
res['size'] = 0
for chunk in cborutil.streamencodemap(res):
yield chunk
# Always yield a bytestring for consistency.
for chunk in cborutil.streamencodebytestring(b''):
yield chunk
continue
# TODO support not having to buffer entire fulltext. But the
# file storage APIs don't support this yet, so...
fulltext = fl.revision(fnode)
res['size'] = len(fulltext)
renamed = fl.renamed(fnode)
if renamed:
res['renamed'] = renamed
for chunk in cborutil.streamencodemap(res):
yield chunk
# While our implementation must buffer in the fulltext, we shouldn't
# force this on the client. So split into an indefinite length
# bytestring and send those chunks.
for chunk in cborutil.streamencodeindefinitebytestring(fulltext):
yield chunk
return wireprototypes.v2streamingresponse(senddata(),
compressible=True)
@wireprotocommand('heads', @wireprotocommand('heads',
args={ args={
'publiconly': False, 'publiconly': False,
}, },
permission='pull') permission='pull')
def headsv2(repo, proto, publiconly=False): def headsv2(repo, proto, publiconly=False):
if publiconly: if publiconly:
repo = repo.filtered('immutable') repo = repo.filtered('immutable')

tests/test-http-protocol.t

s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-cbor\r\n s> Content-Type: application/mercurial-cbor\r\n
s> Content-Length: *\r\n (glob) s> Content-Length: *\r\n (glob)
s> \r\n s> \r\n
s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
sending heads command sending heads command
s> POST /api/exp-http-v2-0001/ro/heads HTTP/1.1\r\n s> POST /api/exp-http-v2-0001/ro/heads HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-exp-framing-0005\r\n s> accept: application/mercurial-exp-framing-0005\r\n
s> content-type: application/mercurial-exp-framing-0005\r\n s> content-type: application/mercurial-exp-framing-0005\r\n
s> content-length: 20\r\n s> content-length: 20\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n s> user-agent: Mercurial debugwireproto\r\n

tests/test-wireproto-command-capabilities.t

s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-cbor\r\n s> Content-Type: application/mercurial-cbor\r\n
s> Content-Length: *\r\n (glob) s> Content-Length: *\r\n (glob)
s> \r\n s> \r\n
s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
cbor> {b'apibase': b'api/', b'apis': {b'exp-http-v2-0001': {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}}, b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'} cbor> {b'apibase': b'api/', b'apis': {b'exp-http-v2-0001': {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'filerevisionsslice': {b'args': {b'endpath': b'b/bar', b'node': b'0123456789abcdef...', b'startpath': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}}, b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'}
capabilities command returns expected info capabilities command returns expected info
$ sendhttpv2peerhandshake << EOF $ sendhttpv2peerhandshake << EOF
> command capabilities > command capabilities
> EOF > EOF
creating http peer for wire protocol version 2 creating http peer for wire protocol version 2
s> GET /?cmd=capabilities HTTP/1.1\r\n s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> vary: X-HgProto-1,X-HgUpgrade-1\r\n s> vary: X-HgProto-1,X-HgUpgrade-1\r\n
s> x-hgproto-1: cbor\r\n s> x-hgproto-1: cbor\r\n
s> x-hgupgrade-1: exp-http-v2-0001\r\n s> x-hgupgrade-1: exp-http-v2-0001\r\n
s> accept: application/mercurial-0.1\r\n s> accept: application/mercurial-0.1\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n s> user-agent: Mercurial debugwireproto\r\n
s> \r\n s> \r\n
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-cbor\r\n s> Content-Type: application/mercurial-cbor\r\n
s> Content-Length: *\r\n (glob) s> Content-Length: *\r\n (glob)
s> \r\n s> \r\n
s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash s> \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
sending capabilities command sending capabilities command
s> POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n s> POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n s> Accept-Encoding: identity\r\n
s> *\r\n (glob) s> *\r\n (glob)
s> content-type: application/mercurial-exp-framing-0005\r\n s> content-type: application/mercurial-exp-framing-0005\r\n
s> content-length: 27\r\n s> content-length: 27\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob) s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n s> user-agent: Mercurial debugwireproto\r\n
s> \r\n s> \r\n
s> \x13\x00\x00\x01\x00\x01\x01\x11\xa1DnameLcapabilities s> \x13\x00\x00\x01\x00\x01\x01\x11\xa1DnameLcapabilities
s> makefile('rb', None) s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-exp-framing-0005\r\n s> Content-Type: application/mercurial-exp-framing-0005\r\n
s> Transfer-Encoding: chunked\r\n s> Transfer-Encoding: chunked\r\n
s> \r\n s> \r\n
s> 2c1\r\n s> 324\r\n
s> \xb9\x02\x00\x01\x00\x02\x012 s> \x1c\x03\x00\x01\x00\x02\x012
s> \xa1FstatusBok\xa4Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005 s> \xa1FstatusBok\xa4Hcommands\xabEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullKfilehistory\xa2Dargs\xa3DpathLpath/to/fileMfilenodebases\x81HdeadbeefMfilenodeheads\x81HdeadbeefKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLfilerevision\xa2Dargs\xa2DpathEa/fooHfilenodeS0123456789abcdef...Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullRfilerevisionsslice\xa2Dargs\xa3DnodeS0123456789abcdef...GendpathEb/barIstartpathEa/fooKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005
s> \r\n s> \r\n
received frame(size=697; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos) received frame(size=796; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
s> 0\r\n s> 0\r\n
s> \r\n s> \r\n
response: [{b'status': b'ok'}, {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}] response: [{b'status': b'ok'}, {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'filehistory': {b'args': {b'filenodebases': [b'deadbeef'], b'filenodeheads': [b'deadbeef'], b'path': b'path/to/file'}, b'permissions': [b'pull']}, b'filerevision': {b'args': {b'filenode': b'0123456789abcdef...', b'path': b'a/foo'}, b'permissions': [b'pull']}, b'filerevisionsslice': {b'args': {b'endpath': b'b/bar', b'node': b'0123456789abcdef...', b'startpath': b'a/foo'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}]
$ cat error.log $ cat error.log

tests/test-wireproto-command-filerevisionsslice.t

  • This file was added.
$ . $TESTDIR/wireprotohelpers.sh
$ hg init server
$ enablehttpv2 server
$ cd server
$ echo a0 > a
$ echo b0 > b
$ mkdir dir0
$ echo c0 > dir0/c
$ echo d0 > dir0/d
$ echo e0 > dir0/e
$ echo z0 > z
$ hg -q commit -A -m 'commit 0'
$ echo a1 > a
$ echo b1 > b
$ hg commit -m 'commit 1'
$ echo ca > dir0/ca
$ echo dd > dir0/dd
$ hg -q commit -A -m 'commit 2'
$ hg mv a a-moved
$ hg commit -m 'commit 3'
$ hg log -T '{rev}:{node} {desc}\n'
3:29c3cfed01c8dca4e2ed052998a0a4783a364265 commit 3
2:4a8fbd921812449fb3cbaa4f8ef90082bee356d1 commit 2
1:a168c837ab2d7935d50000845178380528427f01 commit 1
0:fcf140e9e1ac1191ac879211b8918a79dfbb9f0d commit 0
$ hg serve -p $HGPORT -d --pid-file hg.pid -E error.log
$ cat hg.pid > $DAEMON_PIDS
Request for unknown node fails
$ sendhttpv2peer << EOF
> command filerevisionsslice
> node aaaaaaaaaaaaaaaaaaaa
> startpath a
> endpath z
> EOF
creating http peer for wire protocol version 2
sending filerevisionsslice command
s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-exp-framing-0005\r\n
s> content-type: application/mercurial-exp-framing-0005\r\n
s> content-length: 87\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> O\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeTaaaaaaaaaaaaaaaaaaaaGendpathAzIstartpathAaDnameRfilerevisionsslice
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-exp-framing-0005\r\n
s> Transfer-Encoding: chunked\r\n
s> \r\n
s> 36\r\n
s> .\x00\x00\x01\x00\x02\x012
s> \xa2Eerror\xa1GmessagePunknown revisionFstatusEerror
s> \r\n
received frame(size=46; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
s> 0\r\n
s> \r\n
response: [{b'error': {b'message': b'unknown revision'}, b'status': b'error'}]
Fetching a single file works
$ sendhttpv2peer << EOF
> command filerevisionsslice
> node \xfc\xf1\x40\xe9\xe1\xac\x11\x91\xac\x87\x92\x11\xb8\x91\x8a\x79\xdf\xbb\x9f\x0d
> startpath z
> endpath z
> EOF
creating http peer for wire protocol version 2
sending filerevisionsslice command
s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-exp-framing-0005\r\n
s> content-type: application/mercurial-exp-framing-0005\r\n
s> content-length: 87\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> O\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeT\xfc\xf1@\xe9\xe1\xac\x11\x91\xac\x87\x92\x11\xb8\x91\x8ay\xdf\xbb\x9f\rGendpathAzIstartpathAzDnameRfilerevisionsslice
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-exp-framing-0005\r\n
s> Transfer-Encoding: chunked\r\n
s> \r\n
s> 13\r\n
s> \x0b\x00\x00\x01\x00\x02\x011
s> \xa1FstatusBok
s> \r\n
received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
s> 43\r\n
s> ;\x00\x00\x01\x00\x02\x000
s> \xa1Jtotalitems\x01\xa3DnodeT)\xc0;\xbeA\x0e\xf7\x8e\xb2\xe2\x05\x90\xc1R\x88\xde\xb6\x9f)CDpathAzDsize\x03_Cz0\n
s> \xff
s> \r\n
received frame(size=59; request=1; stream=2; streamflags=; type=command-response; flags=)
s> 8\r\n
s> \x00\x00\x00\x01\x00\x02\x002
s> \r\n
s> 0\r\n
s> \r\n
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
response: [{b'status': b'ok'}, {b'totalitems': 1}, {b'node': b')\xc0;\xbeA\x0e\xf7\x8e\xb2\xe2\x05\x90\xc1R\x88\xde\xb6\x9f)C', b'path': b'z', b'size': 3}, bytearray['z0\n']]
Fetching a range of files works
$ sendhttpv2peer << EOF
> command filerevisionsslice
> node \xa1\x68\xc8\x37\xab\x2d\x79\x35\xd5\x00\x00\x84\x51\x78\x38\x05\x28\x42\x7f\x01
> startpath b
> endpath dir0/d
> EOF
creating http peer for wire protocol version 2
sending filerevisionsslice command
s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-exp-framing-0005\r\n
s> content-type: application/mercurial-exp-framing-0005\r\n
s> content-length: 92\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> T\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeT\xa1h\xc87\xab-y5\xd5\x00\x00\x84Qx8\x05(B\x7f\x01GendpathFdir0/dIstartpathAbDnameRfilerevisionsslice
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-exp-framing-0005\r\n
s> Transfer-Encoding: chunked\r\n
s> \r\n
s> 13\r\n
s> \x0b\x00\x00\x01\x00\x02\x011
s> \xa1FstatusBok
s> \r\n
received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
s> a9\r\n
s> \xa1\x00\x00\x01\x00\x02\x000
s> \xa1Jtotalitems\x03\xa3DnodeT\xb1zk\xd3g=\x9a\xb8\xce\xd5\x81\xa2\t\xf6/=\xa5\xccExDpathAbDsize\x03_Cb1\n
s> \xff\xa3DnodeT\x91DE4j\x0c\xa0b\x9b\xd4|\xeb]\xfe\x07\xe4\xd4\xcf%\x01DpathFdir0/cDsize\x03_Cc0\n
s> \xff\xa3DnodeTS\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4&DpathFdir0/dDsize\x03_Cd0\n
s> \xff
s> \r\n
received frame(size=161; request=1; stream=2; streamflags=; type=command-response; flags=)
s> 8\r\n
s> \x00\x00\x00\x01\x00\x02\x002
s> \r\n
s> 0\r\n
s> \r\n
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
response: [{b'status': b'ok'}, {b'totalitems': 3}, {b'node': b'\xb1zk\xd3g=\x9a\xb8\xce\xd5\x81\xa2\t\xf6/=\xa5\xccEx', b'path': b'b', b'size': 3}, bytearray['b1\n'], {b'node': b'\x91DE4j\x0c\xa0b\x9b\xd4|\xeb]\xfe\x07\xe4\xd4\xcf%\x01', b'path': b'dir0/c', b'size': 3}, bytearray['c0\n'], {b'node': b'S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4&', b'path': b'dir0/d', b'size': 3}, bytearray['d0\n']]
Copy metadata is included when appropriate
$ sendhttpv2peer << EOF
> command filerevisionsslice
> node \x29\xc3\xcf\xed\x01\xc8\xdc\xa4\xe2\xed\x05\x29\x98\xa0\xa4\x78\x3a\x36\x42\x65
> startpath a-moved
> endpath a-moved
> EOF
creating http peer for wire protocol version 2
sending filerevisionsslice command
s> POST /api/exp-http-v2-0001/ro/filerevisionsslice HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-exp-framing-0005\r\n
s> content-type: application/mercurial-exp-framing-0005\r\n
s> content-length: 99\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: Mercurial debugwireproto\r\n
s> \r\n
s> [\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa3DnodeT)\xc3\xcf\xed\x01\xc8\xdc\xa4\xe2\xed\x05)\x98\xa0\xa4x:6BeGendpathGa-movedIstartpathGa-movedDnameRfilerevisionsslice
s> makefile('rb', None)
s> HTTP/1.1 200 OK\r\n
s> Server: testing stub value\r\n
s> Date: $HTTP_DATE$\r\n
s> Content-Type: application/mercurial-exp-framing-0005\r\n
s> Transfer-Encoding: chunked\r\n
s> \r\n
s> 13\r\n
s> \x0b\x00\x00\x01\x00\x02\x011
s> \xa1FstatusBok
s> \r\n
received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
s> a9\r\n
s> \xa1\x00\x00\x01\x00\x02\x000
s> \xa1Jtotalitems\x01\xa4DnodeT4\x7f\xcdS\x0b\xe0J\x85\xe8EoM\xc1\x0b"\x16G#/\x84DpathGa-movedGrenamed\x82AaT\x9a8\x12)\x97\xb3\xac\x97\xbe*\x9a\xa2\xe5V\x83\x83A\xfd\xf2\xccDsize\x18A_XA\x01\n
s> copy: a\n
s> copyrev: 9a38122997b3ac97be2a9aa2e556838341fdf2cc\n
s> \x01\n
s> a1\n
s> \xff
s> \r\n
received frame(size=161; request=1; stream=2; streamflags=; type=command-response; flags=)
s> 8\r\n
s> \x00\x00\x00\x01\x00\x02\x002
s> \r\n
s> 0\r\n
s> \r\n
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
response: [{b'status': b'ok'}, {b'totalitems': 1}, {b'node': b'4\x7f\xcdS\x0b\xe0J\x85\xe8EoM\xc1\x0b"\x16G#/\x84', b'path': b'a-moved', b'renamed': [b'a', b'\x9a8\x12)\x97\xb3\xac\x97\xbe*\x9a\xa2\xe5V\x83\x83A\xfd\xf2\xcc'], b'size': 65}, bytearray['\x01\ncopy: a\ncopyrev: 9a38122997b3ac97be2a9aa2e556838341fdf2cc\n\x01\na1\n']]
$ cat error.log