This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5169

exchangev2: support fetching shallow files history
ClosedPublic

Authored by indygreg on Oct 19 2018, 1:01 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG229d23cdb203: exchangev2: support fetching shallow files history
Summary

This commit teaches the exchangev2 client code to handle fetching shallow
files data.

Only shallow fetching of files data is supported: shallow fetching of
changeset and manifest data is explicitly not yet supported.

Previously, we would fetch file revisions for changesets that were received
by the current pull operation. In the new model, we calculate the set of
"relevant" changesets given the pull depth and only fetch files data for
those changesets.

We also teach the "filesdata" command invocation to vary parameters as needed.

The implementation here is far from complete or optimal. Subsequent pulls will
end up re-fetching a lot of files data. But the application of this data should
mostly be a no-op on the client, so it isn't a big deal.

Depending on the order file revisions are fetched in, revisions could get
inserted with the wrong revision number relationships. I think the best way
to deal with this is to remove revision numbers from storage and to either
dynamically derive them (by reconstructing a DAG from nodes/parents) or remove
revision numbers from the file storage interface completely.

A missing API that we'll likely want to write pretty soon is "ensure files
for revision(s) are present." We can kind of cajole exchangev2.pull() to do
this. But it isn't very efficient. For example, in simple cases like
widening the store to obtain data for a single revision, it is probably
more efficient to walk the manifest and find exactly which file revisions
are missing and to make explicit requests for just their data. In more
advanced cases, asking the server for all files data may be more efficient,
even though it requires sending data the client already has. There is tons
of room for future experimentation here. And TBH I'm not sure what the
final state will be.

Anyway, this commit gets us pretty close to being able to have shallow
and narrow checkouts with exchangev2/sqlite storage. Close enough that a
minimal extension should be able to provide fill in the gaps until the code
in core stabilizes and there is a user-facing way to trigger the
narrow/shallow bits from hg clone without also implying using of the
narrow extension...

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Oct 19 2018, 1:01 PM
Closed by commit rHG229d23cdb203: exchangev2: support fetching shallow files history (authored by indygreg). · Explain WhyOct 22 2018, 11:49 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 12275

mercurial/exchangev2.py

) )
from . import ( from . import (
bookmarks, bookmarks,
error, error,
mdiff, mdiff,
narrowspec, narrowspec,
phases, phases,
pycompat, pycompat,
repository,
setdiscovery, setdiscovery,
) )
def pull(pullop): def pull(pullop):
"""Pull using wire protocol version 2.""" """Pull using wire protocol version 2."""
repo = pullop.repo repo = pullop.repo
remote = pullop.remote remote = pullop.remote
# Write bookmark updates. # Write bookmark updates.
bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'], bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'],
remote.url(), pullop.gettransaction, remote.url(), pullop.gettransaction,
explicit=pullop.explicitbookmarks) explicit=pullop.explicitbookmarks)
manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes']) manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes'])
# We don't properly support shallow changeset and manifest yet. So we apply
# depth limiting locally.
if pullop.depth:
relevantcsetnodes = set()
clnode = repo.changelog.node
for rev in repo.revs(b'ancestors(%ln, %d)',
pullheads, pullop.depth - 1):
relevantcsetnodes.add(clnode(rev))
csetrelevantfilter = lambda n: n in relevantcsetnodes
else:
csetrelevantfilter = lambda n: True
# If obtaining the raw store files, we need to scan the full repo to # If obtaining the raw store files, we need to scan the full repo to
# derive all the changesets, manifests, and linkrevs. # derive all the changesets, manifests, and linkrevs.
if usingrawchangelogandmanifest: if usingrawchangelogandmanifest:
csetsforfiles = [] csetsforfiles = []
mnodesforfiles = [] mnodesforfiles = []
manifestlinkrevs = {} manifestlinkrevs = {}
for rev in repo: for rev in repo:
ctx = repo[rev] ctx = repo[rev]
node = ctx.node()
if not csetrelevantfilter(node):
continue
mnode = ctx.manifestnode() mnode = ctx.manifestnode()
csetsforfiles.append(ctx.node()) csetsforfiles.append(node)
mnodesforfiles.append(mnode) mnodesforfiles.append(mnode)
manifestlinkrevs[mnode] = rev manifestlinkrevs[mnode] = rev
else: else:
csetsforfiles = csetres['added'] csetsforfiles = [n for n in csetres['added'] if csetrelevantfilter(n)]
mnodesforfiles = manres['added'] mnodesforfiles = manres['added']
manifestlinkrevs = manres['linkrevs'] manifestlinkrevs = manres['linkrevs']
# Find all file nodes referenced by added manifests and fetch those # Find all file nodes referenced by added manifests and fetch those
# revisions. # revisions.
fnodes = _derivefilesfrommanifests(repo, narrowmatcher, mnodesforfiles) fnodes = _derivefilesfrommanifests(repo, narrowmatcher, mnodesforfiles)
_fetchfilesfromcsets(repo, tr, remote, pathfilter, fnodes, csetsforfiles, _fetchfilesfromcsets(repo, tr, remote, pathfilter, fnodes, csetsforfiles,
manifestlinkrevs) manifestlinkrevs, shallow=bool(pullop.depth))
def _checkuserawstorefiledata(pullop): def _checkuserawstorefiledata(pullop):
"""Check whether we should use rawstorefiledata command to retrieve data.""" """Check whether we should use rawstorefiledata command to retrieve data."""
repo = pullop.repo repo = pullop.repo
remote = pullop.remote remote = pullop.remote
# Command to obtain raw store data isn't available. # Command to obtain raw store data isn't available.
store = repo.file(path) store = repo.file(path)
store.addgroup( store.addgroup(
iterrevisions(objs, progress), iterrevisions(objs, progress),
locallinkrevs[path].__getitem__, locallinkrevs[path].__getitem__,
weakref.proxy(tr)) weakref.proxy(tr))
def _fetchfilesfromcsets(repo, tr, remote, pathfilter, fnodes, csets, def _fetchfilesfromcsets(repo, tr, remote, pathfilter, fnodes, csets,
manlinkrevs): manlinkrevs, shallow=False):
"""Fetch file data from explicit changeset revisions.""" """Fetch file data from explicit changeset revisions."""
def iterrevisions(objs, remaining, progress): def iterrevisions(objs, remaining, progress):
while remaining: while remaining:
filerevision = next(objs) filerevision = next(objs)
node = filerevision[b'node'] node = filerevision[b'node']
extrafields = {} extrafields = {}
for field, size in filerevision.get(b'fieldsfollowing', []): for field, size in filerevision.get(b'fieldsfollowing', []):
extrafields[field] = next(objs) extrafields[field] = next(objs)
if b'delta' in extrafields: if b'delta' in extrafields:
basenode = filerevision[b'deltabasenode'] basenode = filerevision[b'deltabasenode']
delta = extrafields[b'delta'] delta = extrafields[b'delta']
elif b'revision' in extrafields: elif b'revision' in extrafields:
basenode = nullid basenode = nullid
revision = extrafields[b'revision'] revision = extrafields[b'revision']
delta = mdiff.trivialdiffheader(len(revision)) + revision delta = mdiff.trivialdiffheader(len(revision)) + revision
else: else:
continue continue
if b'linknode' in filerevision:
linknode = filerevision[b'linknode']
else:
linknode = node
yield ( yield (
node, node,
filerevision[b'parents'][0], filerevision[b'parents'][0],
filerevision[b'parents'][1], filerevision[b'parents'][1],
node, linknode,
basenode, basenode,
delta, delta,
# Flags not yet supported. # Flags not yet supported.
0, 0,
) )
progress.increment() progress.increment()
remaining -= 1 remaining -= 1
progress = repo.ui.makeprogress( progress = repo.ui.makeprogress(
_('files'), unit=_('chunks'), _('files'), unit=_('chunks'),
total=sum(len(v) for v in fnodes.itervalues())) total=sum(len(v) for v in fnodes.itervalues()))
commandmeta = remote.apidescriptor[b'commands'][b'filesdata'] commandmeta = remote.apidescriptor[b'commands'][b'filesdata']
batchsize = commandmeta.get(b'recommendedbatchsize', 50000) batchsize = commandmeta.get(b'recommendedbatchsize', 50000)
shallowfiles = repository.REPO_FEATURE_SHALLOW_FILE_STORAGE in repo.features
fields = {b'parents', b'revision'}
clrev = repo.changelog.rev
# There are no guarantees that we'll have ancestor revisions if
# a) this repo has shallow file storage b) shallow data fetching is enabled.
# Force remote to not delta against possibly unknown revisions when these
# conditions hold.
haveparents = not (shallowfiles or shallow)
# Similarly, we may not have calculated linkrevs for all incoming file
# revisions. Ask the remote to do work for us in this case.
if not haveparents:
fields.add(b'linknode')
for i in pycompat.xrange(0, len(csets), batchsize): for i in pycompat.xrange(0, len(csets), batchsize):
batch = [x for x in csets[i:i + batchsize]] batch = [x for x in csets[i:i + batchsize]]
if not batch: if not batch:
continue continue
with remote.commandexecutor() as e: with remote.commandexecutor() as e:
args = { args = {
b'revisions': [{ b'revisions': [{
b'type': b'changesetexplicit', b'type': b'changesetexplicit',
b'nodes': batch, b'nodes': batch,
}], }],
b'fields': {b'parents', b'revision'}, b'fields': fields,
b'haveparents': True, b'haveparents': haveparents,
} }
if pathfilter: if pathfilter:
args[b'pathfilter'] = pathfilter args[b'pathfilter'] = pathfilter
objs = e.callcommand(b'filesdata', args).result() objs = e.callcommand(b'filesdata', args).result()
# First object is an overall header. # First object is an overall header.
overall = next(objs) overall = next(objs)
# We have overall['totalpaths'] segments. # We have overall['totalpaths'] segments.
for i in pycompat.xrange(overall[b'totalpaths']): for i in pycompat.xrange(overall[b'totalpaths']):
header = next(objs) header = next(objs)
path = header[b'path'] path = header[b'path']
store = repo.file(path) store = repo.file(path)
linkrevs = { linkrevs = {
fnode: manlinkrevs[mnode] fnode: manlinkrevs[mnode]
for fnode, mnode in fnodes[path].iteritems()} for fnode, mnode in fnodes[path].iteritems()}
def getlinkrev(node):
if node in linkrevs:
return linkrevs[node]
else:
return clrev(node)
store.addgroup(iterrevisions(objs, header[b'totalitems'], store.addgroup(iterrevisions(objs, header[b'totalitems'],
progress), progress),
linkrevs.__getitem__, getlinkrev,
weakref.proxy(tr)) weakref.proxy(tr),
maybemissingparents=shallow)

tests/pullext.py

# pullext.py - Simple extension to test pulling # pullext.py - Simple extension to test pulling
# #
# Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com> # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
from mercurial.i18n import _ from mercurial.i18n import _
from mercurial import ( from mercurial import (
commands, commands,
error,
extensions, extensions,
localrepo, localrepo,
repository, repository,
) )
def clonecommand(orig, ui, repo, *args, **kwargs): def clonecommand(orig, ui, repo, *args, **kwargs):
if kwargs.get(r'include') or kwargs.get(r'exclude'): if kwargs.get(r'include') or kwargs.get(r'exclude'):
kwargs[r'narrow'] = True kwargs[r'narrow'] = True
if kwargs.get(r'depth'):
try:
kwargs[r'depth'] = int(kwargs[r'depth'])
except ValueError:
raise error.Abort(_('--depth must be an integer'))
return orig(ui, repo, *args, **kwargs) return orig(ui, repo, *args, **kwargs)
def featuresetup(ui, features): def featuresetup(ui, features):
features.add(repository.NARROW_REQUIREMENT) features.add(repository.NARROW_REQUIREMENT)
def extsetup(ui): def extsetup(ui):
entry = extensions.wrapcommand(commands.table, 'clone', clonecommand) entry = extensions.wrapcommand(commands.table, 'clone', clonecommand)
hasinclude = any(x[1] == 'include' for x in entry[1]) hasinclude = any(x[1] == 'include' for x in entry[1])
hasdepth = any(x[1] == 'depth' for x in entry[1])
if not hasinclude: if not hasinclude:
entry[1].append(('', 'include', [], entry[1].append(('', 'include', [],
_('pattern of file/directory to clone'))) _('pattern of file/directory to clone')))
entry[1].append(('', 'exclude', [], entry[1].append(('', 'exclude', [],
_('pattern of file/directory to not clone'))) _('pattern of file/directory to not clone')))
if not hasdepth:
entry[1].append(('', 'depth', '',
_('ancestry depth of changesets to fetch')))
localrepo.featuresetupfuncs.add(featuresetup) localrepo.featuresetupfuncs.add(featuresetup)

tests/test-wireproto-exchangev2-shallow.t

  • This file was added.
#require sqlite
Tests for wire protocol version 2 exchange.
Tests in this file should be folded into existing tests once protocol
v2 has enough features that it can be enabled via #testcase in existing
tests.
$ . $TESTDIR/wireprotohelpers.sh
$ enablehttpv2client
$ cat >> $HGRCPATH << EOF
> [extensions]
> sqlitestore =
> pullext = $TESTDIR/pullext.py
> [storage]
> new-repo-backend=sqlite
> EOF
Configure a server
$ hg init server-basic
$ enablehttpv2 server-basic
$ cd server-basic
$ mkdir dir0 dir1
$ echo a0 > a
$ echo b0 > b
$ hg -q commit -A -m 'commit 0'
$ echo c0 > dir0/c
$ echo d0 > dir0/d
$ hg -q commit -A -m 'commit 1'
$ echo e0 > dir1/e
$ echo f0 > dir1/f
$ hg -q commit -A -m 'commit 2'
$ echo c1 > dir0/c
$ echo e1 > dir1/e
$ hg commit -m 'commit 3'
$ echo c2 > dir0/c
$ echo e2 > dir1/e
$ echo f1 > dir1/f
$ hg commit -m 'commit 4'
$ echo a1 > a
$ echo b1 > b
$ hg commit -m 'commit 5'
$ hg log -G -T '{node} {desc}'
@ 93a8bd067ed2840d9aa810ad598168383a3a2c3a commit 5
|
o dc666cf9ecf3d94e6b830f30e5f1272e2a9164d9 commit 4
|
o 97765fc3cd624fd1fa0176932c21ffd16adf432e commit 3
|
o 47fe012ab237a8c7fc0c78f9f26d5866eef3f825 commit 2
|
o b709380892b193c1091d3a817f706052e346821b commit 1
|
o 3390ef850073fbc2f0dfff2244342c8e9229013a commit 0
$ hg serve -p $HGPORT -d --pid-file hg.pid -E error.log
$ cat hg.pid > $DAEMON_PIDS
$ cd ..
Shallow clone pulls down latest revision of every file
$ hg --debug clone --depth 1 http://localhost:$HGPORT client-shallow-1
using http://localhost:$HGPORT/
sending capabilities command
query 1; heads
sending 2 commands
sending command heads: {}
sending command known: {
'nodes': []
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command changesetdata: {
'fields': set([
'bookmarks',
'parents',
'phase',
'revision'
]),
'revisions': [
{
'heads': [
'\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
],
'roots': [],
'type': 'changesetdagrange'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1170; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
add changeset 3390ef850073
add changeset b709380892b1
add changeset 47fe012ab237
add changeset 97765fc3cd62
add changeset dc666cf9ecf3
add changeset 93a8bd067ed2
checking for updated bookmarks
sending 1 commands
sending command manifestdata: {
'fields': set([
'parents',
'revision'
]),
'haveparents': True,
'nodes': [
'\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
'|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
'\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
'113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3',
'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby',
'\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83'
],
'tree': ''
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1515; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command filesdata: {
'fields': set([
'linknode',
'parents',
'revision'
]),
'haveparents': False,
'revisions': [
{
'nodes': [
'\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
],
'type': 'changesetexplicit'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
updating the branch cache
new changesets 3390ef850073:93a8bd067ed2
updating to branch default
resolving manifests
branchmerge: False, force: False, partial: False
ancestor: 000000000000, local: 000000000000+, remote: 93a8bd067ed2
a: remote created -> g
getting a
b: remote created -> g
getting b
dir0/c: remote created -> g
getting dir0/c
dir0/d: remote created -> g
getting dir0/d
dir1/e: remote created -> g
getting dir1/e
dir1/f: remote created -> g
getting dir1/f
6 files updated, 0 files merged, 0 files removed, 0 files unresolved
(sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
$ sqlite3 -line client-shallow-1/.hg/store/db.sqlite << EOF
> SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata ORDER BY id ASC;
> EOF
id = 1
path = a
revnum = 0
node = \x9a8\x12)\x97\xb3\xac\x97\xbe*\x9a\xa2\xe5V\x83\x83A\xfd\xf2\xcc (esc)
p1rev = -1
p2rev = -1
linkrev = 5
flags = 2
id = 2
path = b
revnum = 0
node = \xb1zk\xd3g=\x9a\xb8\xce\xd5\x81\xa2 \xf6/=\xa5\xccEx (esc)
p1rev = -1
p2rev = -1
linkrev = 5
flags = 2
id = 3
path = dir0/c
revnum = 0
node = I\x1d\xa1\xbb\x89\xeax\xc0\xc0\xa2s[\x16\xce}\x93\x1d\xc8\xe2\r (esc)
p1rev = -1
p2rev = -1
linkrev = 4
flags = 2
id = 4
path = dir0/d
revnum = 0
node = S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4& (esc)
p1rev = -1
p2rev = -1
linkrev = 1
flags = 0
id = 5
path = dir1/e
revnum = 0
node = ]\xf3\xac\xd8\xd0\xc7\xfaP\x98\xd0'\x9a\x044\xc3\x02\x9e+x\xe1 (esc)
p1rev = -1
p2rev = -1
linkrev = 4
flags = 2
id = 6
path = dir1/f
revnum = 0
node = (\xc7v\xae\x08\xd0\xd5^\xb4\x06H\xb4\x01\xb9\x0f\xf5DH4\x8e (esc)
p1rev = -1
p2rev = -1
linkrev = 4
flags = 2
Test a shallow clone with only some files
$ hg --debug clone --depth 1 --include dir0/ http://localhost:$HGPORT client-shallow-narrow-1
using http://localhost:$HGPORT/
sending capabilities command
query 1; heads
sending 2 commands
sending command heads: {}
sending command known: {
'nodes': []
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command changesetdata: {
'fields': set([
'bookmarks',
'parents',
'phase',
'revision'
]),
'revisions': [
{
'heads': [
'\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
],
'roots': [],
'type': 'changesetdagrange'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1170; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
add changeset 3390ef850073
add changeset b709380892b1
add changeset 47fe012ab237
add changeset 97765fc3cd62
add changeset dc666cf9ecf3
add changeset 93a8bd067ed2
checking for updated bookmarks
sending 1 commands
sending command manifestdata: {
'fields': set([
'parents',
'revision'
]),
'haveparents': True,
'nodes': [
'\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
'|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
'\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
'113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3',
'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby',
'\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83'
],
'tree': ''
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1515; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command filesdata: {
'fields': set([
'linknode',
'parents',
'revision'
]),
'haveparents': False,
'pathfilter': {
'include': [
'path:dir0'
]
},
'revisions': [
{
'nodes': [
'\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
],
'type': 'changesetexplicit'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=355; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
updating the branch cache
new changesets 3390ef850073:93a8bd067ed2
updating to branch default
resolving manifests
branchmerge: False, force: False, partial: False
ancestor: 000000000000, local: 000000000000+, remote: 93a8bd067ed2
dir0/c: remote created -> g
getting dir0/c
dir0/d: remote created -> g
getting dir0/d
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
(sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
$ sqlite3 -line client-shallow-narrow-1/.hg/store/db.sqlite << EOF
> SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata ORDER BY id ASC;
> EOF
id = 1
path = dir0/c
revnum = 0
node = I\x1d\xa1\xbb\x89\xeax\xc0\xc0\xa2s[\x16\xce}\x93\x1d\xc8\xe2\r (esc)
p1rev = -1
p2rev = -1
linkrev = 4
flags = 2
id = 2
path = dir0/d
revnum = 0
node = S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4& (esc)
p1rev = -1
p2rev = -1
linkrev = 1
flags = 0
Cloning an old revision with depth=1 works
$ hg --debug clone --depth 1 -r 97765fc3cd624fd1fa0176932c21ffd16adf432e http://localhost:$HGPORT client-shallow-2
using http://localhost:$HGPORT/
sending capabilities command
sending 1 commands
sending command lookup: {
'key': '97765fc3cd624fd1fa0176932c21ffd16adf432e'
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=21; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
query 1; heads
sending 2 commands
sending command heads: {}
sending command known: {
'nodes': []
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command changesetdata: {
'fields': set([
'bookmarks',
'parents',
'phase',
'revision'
]),
'revisions': [
{
'heads': [
'\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
],
'roots': [],
'type': 'changesetdagrange'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=783; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
add changeset 3390ef850073
add changeset b709380892b1
add changeset 47fe012ab237
add changeset 97765fc3cd62
checking for updated bookmarks
sending 1 commands
sending command manifestdata: {
'fields': set([
'parents',
'revision'
]),
'haveparents': True,
'nodes': [
'\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
'|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
'\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
'113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3'
],
'tree': ''
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=967; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command filesdata: {
'fields': set([
'linknode',
'parents',
'revision'
]),
'haveparents': False,
'revisions': [
{
'nodes': [
'\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
],
'type': 'changesetexplicit'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
updating the branch cache
new changesets 3390ef850073:97765fc3cd62
updating to branch default
resolving manifests
branchmerge: False, force: False, partial: False
ancestor: 000000000000, local: 000000000000+, remote: 97765fc3cd62
a: remote created -> g
getting a
b: remote created -> g
getting b
dir0/c: remote created -> g
getting dir0/c
dir0/d: remote created -> g
getting dir0/d
dir1/e: remote created -> g
getting dir1/e
dir1/f: remote created -> g
getting dir1/f
6 files updated, 0 files merged, 0 files removed, 0 files unresolved
(sent 6 HTTP requests and * bytes; received * bytes in responses) (glob)
Incremental pull of shallow clone fetches new changesets
$ hg --cwd client-shallow-2 --debug pull http://localhost:$HGPORT
pulling from http://localhost:$HGPORT/
using http://localhost:$HGPORT/
sending capabilities command
query 1; heads
sending 2 commands
sending command heads: {}
sending command known: {
'nodes': [
'\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=2; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
searching for changes
all local heads known remotely
sending 1 commands
sending command changesetdata: {
'fields': set([
'bookmarks',
'parents',
'phase',
'revision'
]),
'revisions': [
{
'heads': [
'\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
],
'roots': [
'\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
],
'type': 'changesetdagrange'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=400; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
add changeset dc666cf9ecf3
add changeset 93a8bd067ed2
checking for updated bookmarks
sending 1 commands
sending command manifestdata: {
'fields': set([
'parents',
'revision'
]),
'haveparents': True,
'nodes': [
'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby',
'\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83'
],
'tree': ''
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=561; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command filesdata: {
'fields': set([
'linknode',
'parents',
'revision'
]),
'haveparents': False,
'revisions': [
{
'nodes': [
'\xdcfl\xf9\xec\xf3\xd9Nk\x83\x0f0\xe5\xf1\'.*\x91d\xd9',
'\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:'
],
'type': 'changesetexplicit'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1373; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
updating the branch cache
new changesets dc666cf9ecf3:93a8bd067ed2
(run 'hg update' to get a working copy)
(sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
$ hg --cwd client-shallow-2 up tip
merging dir0/c
merging dir1/e
3 files updated, 2 files merged, 0 files removed, 0 files unresolved

tests/test-wireproto-exchangev2.t

#if reporevlogstore #if reporevlogstore
$ find client-stream-2/.hg/store -type f -name '*.i' | sort $ find client-stream-2/.hg/store -type f -name '*.i' | sort
client-stream-2/.hg/store/00changelog.i client-stream-2/.hg/store/00changelog.i
client-stream-2/.hg/store/00manifest.i client-stream-2/.hg/store/00manifest.i
client-stream-2/.hg/store/data/dir0/c.i client-stream-2/.hg/store/data/dir0/c.i
client-stream-2/.hg/store/data/dir0/d.i client-stream-2/.hg/store/data/dir0/d.i
#endif #endif
Shallow clone doesn't work with revlogs
$ hg --debug --config extensions.pullext=$TESTDIR/pullext.py clone --depth 1 -U http://localhost:$HGPORT client-shallow-revlogs
using http://localhost:$HGPORT/
sending capabilities command
query 1; heads
sending 2 commands
sending command heads: {}
sending command known: {
'nodes': []
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command changesetdata: {
'fields': set([
'bookmarks',
'parents',
'phase',
'revision'
]),
'revisions': [
{
'heads': [
'\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
],
'roots': [],
'type': 'changesetdagrange'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=783; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
add changeset 3390ef850073
add changeset b709380892b1
add changeset 47fe012ab237
add changeset 97765fc3cd62
checking for updated bookmarks
sending 1 commands
sending command manifestdata: {
'fields': set([
'parents',
'revision'
]),
'haveparents': True,
'nodes': [
'\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A',
'|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8',
'\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb',
'113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3'
],
'tree': ''
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=967; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
sending 1 commands
sending command filesdata: {
'fields': set([
'linknode',
'parents',
'revision'
]),
'haveparents': False,
'revisions': [
{
'nodes': [
'\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.'
],
'type': 'changesetexplicit'
}
]
}
received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos)
received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
transaction abort!
rollback completed
(sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
abort: revlog storage does not support missing parents write mode
[255]