This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1857

pull: re-run discovery and pullbundle2 if server didn't send all heads
AbandonedPublic

Authored by joerg.sonnenberger on Jan 12 2018, 5:08 PM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

joerg.sonnenberger added a comment.Jan 12 2018, 5:27 PM

This is a proof-of-concept. There is an interaction with obsoletion that I am still trying to understand (and fix).

joerg.sonnenberger added a comment.Jan 13 2018, 6:12 AM

With the last update, the obs issues are resolved.

indygreg requested changes to this revision.Jan 14 2018, 5:04 PM
indygreg added a subscriber: indygreg.

I'm generally in favor of this functionality. It enables some interesting server features (such as pullbundles).

This review needs to get hooked up to the Phabricator "stack" as the pullbundles patch(es). Using hg phabsend A::B should do that.

I'd like to see a test around a server not sending changegroup data. I /think/ the existing code will abort the while True loop in this case. But I don't fully understand when various attributes on pullop are updated. I'd also feel more comfortable about things if there were an explicit check in the loop that the tip of changelog increased if revisions were requested. I'm worried about getting into an infinite loop due to a misbehaving server. I /think/ I'd like to see the establishment of a contract that if revision data is requested, the server MUST respond with revision data.

This revision now requires changes to proceed.Jan 14 2018, 5:04 PM
joerg.sonnenberger abandoned this revision.Jan 15 2018, 2:26 PM

Merging into the parent review.

Revision Contents
Changeset List

PathPackages
Mmercurial/exchange.py (19 lines)
Mmercurial/wireproto.py (3 lines)
Mtests/test-pull-r.t (37 lines)

Diff 4824

mercurial/exchange.py

raise error.Abort(msg) raise error.Abort(msg)
pullop.trmanager = transactionmanager(repo, 'pull', remote.url()) pullop.trmanager = transactionmanager(repo, 'pull', remote.url())
with repo.wlock(), repo.lock(), pullop.trmanager: with repo.wlock(), repo.lock(), pullop.trmanager:
# This should ideally be in _pullbundle2(). However, it needs to run # This should ideally be in _pullbundle2(). However, it needs to run
# before discovery to avoid extra work. # before discovery to avoid extra work.
_maybeapplyclonebundle(pullop) _maybeapplyclonebundle(pullop)
streamclone.maybeperformlegacystreamclone(pullop) streamclone.maybeperformlegacystreamclone(pullop)
while True:
_pulldiscovery(pullop) _pulldiscovery(pullop)
if pullop.canusebundle2: if not pullop.canusebundle2:
break
_pullbundle2(pullop) _pullbundle2(pullop)
# The server may send a partial reply, i.e. when inlining
# pre-computed bundles. In that case, re-run the discovery
# phase and bundle again. There are two indicators that the
# process is finished:
# - no changes have been received (cgresult)
# - all remote heads are known locally
# The head check must use the unfiltered view as obsoletion
# markers can hide heads.
if not pullop.cgresult:
break
unficl = repo.unfiltered().changelog
if all(unficl.hasnode(n) for n in pullop.rheads):
break
_pullchangeset(pullop) _pullchangeset(pullop)
_pullphase(pullop) _pullphase(pullop)
_pullbookmarks(pullop) _pullbookmarks(pullop)
_pullobsolete(pullop) _pullobsolete(pullop)
# storing remotenames # storing remotenames
if repo.ui.configbool('experimental', 'remotenames'): if repo.ui.configbool('experimental', 'remotenames'):
logexchange.pullremotenames(repo, remote) logexchange.pullremotenames(repo, remote)

mercurial/wireproto.py

hint=bundle2requiredhint) hint=bundle2requiredhint)
try: try:
clheads = set(repo.changelog.heads()) clheads = set(repo.changelog.heads())
heads = set(opts.get('heads', set())) heads = set(opts.get('heads', set()))
common = set(opts.get('common', set())) common = set(opts.get('common', set()))
common.discard(nullid) common.discard(nullid)
if repo.ui.configbool('server', 'pullbundle') and if repo.ui.configbool('server', 'pullbundle'):
self.capable('partial-pull'):
# Check if a pre-built bundle covers this request. # Check if a pre-built bundle covers this request.
bundle = find_pullbundle(repo, opts, clheads, heads, common) bundle = find_pullbundle(repo, opts, clheads, heads, common)
if bundle: if bundle:
return streamres(gen=util.filechunkiter(bundle), return streamres(gen=util.filechunkiter(bundle),
prefer_uncompressed=True) prefer_uncompressed=True)
if repo.ui.configbool('server', 'disablefullbundle'): if repo.ui.configbool('server', 'disablefullbundle'):
# Check to see if this is a full clone. # Check to see if this is a full clone.

tests/test-pull-r.t

new changesets effea6de0384 new changesets effea6de0384
(run 'hg heads' to see heads, 'hg merge' to merge) (run 'hg heads' to see heads, 'hg merge' to merge)
$ cd .. $ cd ..
$ killdaemons.py $ killdaemons.py
$ grep 'sending pullbundle ' repo-server.txt $ grep 'sending pullbundle ' repo-server.txt
sending pullbundle "0.hg" sending pullbundle "0.hg"
sending pullbundle "1.hg" sending pullbundle "1.hg"
sending pullbundle "2.hg" sending pullbundle "2.hg"
Test pullbundle functionality for incremental pulls
$ cd repo
$ hg serve --debug -p $HGPORT2 --pid-file=../repo.pid > ../repo-server.txt 2>&1 &
$ while ! grep listening ../repo-server.txt > /dev/null; do sleep 1; done
$ cat ../repo.pid >> $DAEMON_PIDS
$ cd ..
$ hg clone http://localhost:$HGPORT2/ repo.pullbundle2
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
new changesets bbd179dfa0a7:66e3ba28d0d7
updating to branch default
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ killdaemons.py
$ grep 'sending pullbundle ' repo-server.txt
sending pullbundle "0.hg"
sending pullbundle "2.hg"
sending pullbundle "1.hg"