This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4767

exchangev2: recognize narrow patterns when pulling
AbandonedPublic

Authored by indygreg on Sep 26 2018, 5:44 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

pulloperation instances were recently taught to record file
include and exclude patterns to facilitate narrow file transfer.
Teaching the exchangev2 code to transfer a subset of files is
as simple as constructing a narrow matcher from these patterns and
filtering all seen file paths through it.

In addition to testing presence of revlogs, we also test for presence
of copy/rename metadata when the source revision is outside the
narrow spec. This verifes that our narrow filelog class is working
properly.

Keep in mind that this change only influences file data: we're
still fetching all changeset and manifest data. So, there's still
a ton of "partial clone" to implement in exchangev2.

On a personal note, I derive gratification that this feature requires
very few lines of new code to implement. And notably, none of those
lines influence the parameters sent to the server: the client just
requests less repository data than it did before.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Sep 26 2018, 5:44 PM
indygreg planned changes to this revision.Oct 3 2018, 2:11 PM

I'll be rebasing this locally as part of future feature work. I plan to resubmit again later.

indygreg abandoned this revision.Oct 17 2018, 4:11 AM

I'll resubmit this as part of another series.

Revision Contents
Changeset List

PathPackages
Mmercurial/exchangev2.py (16 lines)
Mtests/test-fastcheckout.t (73 lines)

Diff 11438

mercurial/exchangev2.py

from .node import ( from .node import (
nullid, nullid,
short, short,
) )
from . import ( from . import (
bookmarks, bookmarks,
error, error,
mdiff, mdiff,
narrowspec,
phases, phases,
pycompat, pycompat,
setdiscovery, setdiscovery,
) )
def pull(pullop): def pull(pullop):
"""Pull using wire protocol version 2.""" """Pull using wire protocol version 2."""
repo = pullop.repo repo = pullop.repo
remote = pullop.remote remote = pullop.remote
tr = pullop.trmanager.transaction() tr = pullop.trmanager.transaction()
# We don't use the repo's narrow matcher here because the patterns passed
# to exchange.pull() could be different.
narrowmatcher = narrowspec.match(repo.root,
# Empty maps to nevermatcher. So always
# set includes if missing.
pullop.includepats or {'path:.'},
pullop.excludepats)
# Figure out what needs to be fetched. # Figure out what needs to be fetched.
common, fetch, remoteheads = _pullchangesetdiscovery( common, fetch, remoteheads = _pullchangesetdiscovery(
repo, remote, pullop.heads, abortwhenunrelated=pullop.force) repo, remote, pullop.heads, abortwhenunrelated=pullop.force)
# And fetch the data. # And fetch the data.
pullheads = pullop.heads or remoteheads pullheads = pullop.heads or remoteheads
csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads) csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads)
bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'], bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'],
remote.url(), pullop.gettransaction, remote.url(), pullop.gettransaction,
explicit=pullop.explicitbookmarks) explicit=pullop.explicitbookmarks)
manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes']) manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes'])
# Find all file nodes referenced by added manifests and fetch those # Find all file nodes referenced by added manifests and fetch those
# revisions. # revisions.
fnodes = _derivefilesfrommanifests(repo, manres['added']) fnodes = _derivefilesfrommanifests(repo, narrowmatcher, manres['added'])
_fetchfiles(repo, tr, remote, fnodes, manres['linkrevs']) _fetchfiles(repo, tr, remote, fnodes, manres['linkrevs'])
def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True): def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
"""Determine which changesets need to be pulled.""" """Determine which changesets need to be pulled."""
if heads: if heads:
knownnode = repo.changelog.hasnode knownnode = repo.changelog.hasnode
if all(knownnode(head) for head in heads): if all(knownnode(head) for head in heads):
progress.complete() progress.complete()
return { return {
'added': added, 'added': added,
'linkrevs': linkrevs, 'linkrevs': linkrevs,
} }
def _derivefilesfrommanifests(repo, manifestnodes): def _derivefilesfrommanifests(repo, matcher, manifestnodes):
"""Determine what file nodes are relevant given a set of manifest nodes. """Determine what file nodes are relevant given a set of manifest nodes.
Returns a dict mapping file paths to dicts of file node to first manifest Returns a dict mapping file paths to dicts of file node to first manifest
node. node.
""" """
ml = repo.manifestlog ml = repo.manifestlog
fnodes = collections.defaultdict(dict) fnodes = collections.defaultdict(dict)
for manifestnode in manifestnodes: for manifestnode in manifestnodes:
m = ml.get(b'', manifestnode) m = ml.get(b'', manifestnode)
# TODO this will pull in unwanted nodes because it takes the storage # TODO this will pull in unwanted nodes because it takes the storage
# delta into consideration. What we really want is something that takes # delta into consideration. What we really want is something that takes
# the delta between the manifest's parents. And ideally we would # the delta between the manifest's parents. And ideally we would
# ignore file nodes that are known locally. For now, ignore both # ignore file nodes that are known locally. For now, ignore both
# these limitations. This will result in incremental fetches requesting # these limitations. This will result in incremental fetches requesting
# data we already have. So this is far from ideal. # data we already have. So this is far from ideal.
md = m.readfast() md = m.readfast()
for path, fnode in md.items(): for path, fnode in md.items():
if matcher(path):
fnodes[path].setdefault(fnode, manifestnode) fnodes[path].setdefault(fnode, manifestnode)
return fnodes return fnodes
def _fetchfiles(repo, tr, remote, fnodes, linkrevs): def _fetchfiles(repo, tr, remote, fnodes, linkrevs):
def iterrevisions(objs, progress): def iterrevisions(objs, progress):
for filerevision in objs: for filerevision in objs:
node = filerevision[b'node'] node = filerevision[b'node']

tests/test-fastcheckout.t

$ . $TESTDIR/wireprotohelpers.sh $ . $TESTDIR/wireprotohelpers.sh
$ cat >> $HGRCPATH << EOF $ cat >> $HGRCPATH << EOF
> [extensions] > [extensions]
> fastcheckout = > fastcheckout =
> [diff]
> git = true
> EOF > EOF
$ hg init server $ hg init server
$ cd server $ cd server
$ echo 0 > foo $ echo 0 > foo
$ mkdir -p dir0/child0 dir0/child1 dir1 $ mkdir -p dir0/child0 dir0/child1 dir1
$ echo 0 > dir0/file0.txt $ echo 0 > dir0/file0.txt
$ hg commit -m 'commit 1' $ hg commit -m 'commit 1'
$ echo 2 > dir1/py3.py $ echo 2 > dir1/py3.py
$ hg commit -m 'commit 2' $ hg commit -m 'commit 2'
$ echo 0 > dir0/file3.txt $ echo 0 > dir0/file3.txt
$ hg commit -A -m 'commit 3' $ hg commit -A -m 'commit 3'
adding dir0/file3.txt adding dir0/file3.txt
$ hg cp dir1/file1.txt dir0/file1-copy.txt
$ hg commit -m 'commit 4'
$ hg log -G -T '{rev}:{node} {desc}' $ hg log -G -T '{rev}:{node} {desc}'
@ 3:27c5539872f8d2cfd5b95cab1fcaaee9b9bf9459 commit 3 @ 4:1816adf6a180913ad5db2a795fd09e02bd01657b commit 4
|
o 3:27c5539872f8d2cfd5b95cab1fcaaee9b9bf9459 commit 3
| |
o 2:15c769176aab806212712ee9df382ebc7b63ade1 commit 2 o 2:15c769176aab806212712ee9df382ebc7b63ade1 commit 2
| |
o 1:805a128077322d4be238a5d74719670ef29969af commit 1 o 1:805a128077322d4be238a5d74719670ef29969af commit 1
| |
o 0:6a99379240831e67873d9bf61a687f20b08e6b95 commit 0 o 0:6a99379240831e67873d9bf61a687f20b08e6b95 commit 0
$ find narrow-dest0/.hg/store -type f | sort $ find narrow-dest0/.hg/store -type f | sort
narrow-dest0/.hg/store/00changelog.i narrow-dest0/.hg/store/00changelog.i
narrow-dest0/.hg/store/00manifest.i narrow-dest0/.hg/store/00manifest.i
narrow-dest0/.hg/store/data/dir0/child0/c.c.i narrow-dest0/.hg/store/data/dir0/child0/c.c.i
narrow-dest0/.hg/store/data/dir0/child0/py0.py.i narrow-dest0/.hg/store/data/dir0/child0/py0.py.i
narrow-dest0/.hg/store/data/dir0/child1/py1.py.i narrow-dest0/.hg/store/data/dir0/child1/py1.py.i
narrow-dest0/.hg/store/data/dir0/child1/py2.py.i narrow-dest0/.hg/store/data/dir0/child1/py2.py.i
narrow-dest0/.hg/store/data/dir0/file0.txt.i narrow-dest0/.hg/store/data/dir0/file0.txt.i
narrow-dest0/.hg/store/data/dir1/file1.txt.i
narrow-dest0/.hg/store/data/dir1/file2.txt.i
narrow-dest0/.hg/store/data/dir1/py3.py.i
narrow-dest0/.hg/store/data/foo.i
narrow-dest0/.hg/store/fncache narrow-dest0/.hg/store/fncache
narrow-dest0/.hg/store/narrowspec narrow-dest0/.hg/store/narrowspec
narrow-dest0/.hg/store/phaseroots narrow-dest0/.hg/store/phaseroots
narrow-dest0/.hg/store/undo narrow-dest0/.hg/store/undo
narrow-dest0/.hg/store/undo.backupfiles narrow-dest0/.hg/store/undo.backupfiles
narrow-dest0/.hg/store/undo.phaseroots narrow-dest0/.hg/store/undo.phaseroots
#endif #endif
Incremental pull fetches new files in narrow spec
$ hg fastcheckout http://localhost:$HGPORT 27c5539872 narrow-dest0
fetching data from http://localhost:$HGPORT
searching for changes
new changesets 27c5539872f8
updating to 27c5539872f8
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
#if reporevlogstore
$ find narrow-dest0/.hg/store -type f | sort
narrow-dest0/.hg/store/00changelog.i
narrow-dest0/.hg/store/00manifest.i
narrow-dest0/.hg/store/data/dir0/child0/c.c.i
narrow-dest0/.hg/store/data/dir0/child0/py0.py.i
narrow-dest0/.hg/store/data/dir0/child1/py1.py.i
narrow-dest0/.hg/store/data/dir0/child1/py2.py.i
narrow-dest0/.hg/store/data/dir0/file0.txt.i
narrow-dest0/.hg/store/data/dir0/file3.txt.i
narrow-dest0/.hg/store/fncache
narrow-dest0/.hg/store/narrowspec
narrow-dest0/.hg/store/phaseroots
narrow-dest0/.hg/store/undo
narrow-dest0/.hg/store/undo.backup.fncache
narrow-dest0/.hg/store/undo.backup.phaseroots
narrow-dest0/.hg/store/undo.backupfiles
narrow-dest0/.hg/store/undo.phaseroots
#endif
Rename metadata for file outside of narrow spec is silently dropped
$ hg fastcheckout http://localhost:$HGPORT 1816adf6a18 narrow-dest0
fetching data from http://localhost:$HGPORT
searching for changes
new changesets 1816adf6a180
updating to 1816adf6a180
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg --cwd narrow-dest0 diff -c .
diff --git a/dir0/file1-copy.txt b/dir0/file1-copy.txt
new file mode 100644
--- /dev/null
+++ b/dir0/file1-copy.txt
@@ -0,0 +1,1 @@
+0
Rename metadata for file inside narrow spec is reported
$ hg fastcheckout http://localhost:$HGPORT 1816adf6a18 rename-included
fetching data from http://localhost:$HGPORT
new changesets 6a9937924083:1816adf6a180
updating to 1816adf6a180
11 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg --cwd rename-included diff -c .
diff --git a/dir1/file1.txt b/dir0/file1-copy.txt
copy from dir1/file1.txt
copy to dir0/file1-copy.txt
Should not have any server-side errors Should not have any server-side errors
$ cat error.log $ cat error.log