( )⚙ D4480 exchangev2: start to implement pull with wire protocol v2

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4480

exchangev2: start to implement pull with wire protocol v2
ClosedPublic

Authored by indygreg on Sep 5 2018, 12:20 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGa86d21e70b2b: exchangev2: start to implement pull with wire protocol v2
Summary

Wire protocol version 2 will take a substantially different
approach to exchange than version 1 (at least as far as pulling
is concerned).

This commit establishes a new exchangev2 module for holding
code related to exchange using wire protocol v2. I could have
added things to the existing exchange module. But it is already
quite big. And doing things inline isn't in question because
the existing code is already littered with conditional code
for various states of support for the existing wire protocol
as it evolved over 10+ years. A new module gives us a chance
to make a clean break.

This approach does mean we'll end up writing some duplicate
code. And there's a significant chance we'll miss functionality
as code is ported. The plan is to eventually add #testcase's
to existing tests so the new wire protocol is tested side-by-side
with the existing one. This will hopefully tease out any
features that weren't ported properly. But before we get there,
we need to build up support for the new exchange methods.

Our journey towards implementing a new exchange begins with pulling.
And pulling begins with discovery.

The discovery code added to exchangev2 is heavily drawn from
the following functions:

  • exchange._pulldiscoverychangegroup
  • discovery.findcommonincoming

For now, we build on top of existing discovery mechanisms. The
new wire protocol should be capable of doing things more efficiently.
But I'd rather defer on this problem.

To foster the transition, we invent a fake capability on the HTTPv2
peer and have the main pull code in exchange.py call into exchangev2
when the new wire protocol is being used.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Sep 5 2018, 12:20 PM
durin42 added a subscriber: durin42.Sep 12 2018, 10:42 AM
durin42 added inline comments.
mercurial/exchange.py
1512

What's the deal with this function-level import? It doesn't appear to be avoiding a cycle...

indygreg planned changes to this revision.Sep 12 2018, 12:56 PM
indygreg added inline comments.
mercurial/exchange.py
1512

At one time the series introduced a cycle. But the current series doesn't do that. So I suppose this can be rewritten to a file-level import for now.

indygreg updated this revision to Diff 10961.Sep 12 2018, 1:11 PM
Closed by commit rHGa86d21e70b2b: exchangev2: start to implement pull with wire protocol v2 (authored by indygreg). · Explain WhySep 14 2018, 11:15 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 11058

mercurial/exchange.py

attr, attr,
) )
from . import ( from . import (
bookmarks as bookmod, bookmarks as bookmod,
bundle2, bundle2,
changegroup, changegroup,
discovery, discovery,
error, error,
exchangev2,
lock as lockmod, lock as lockmod,
logexchange, logexchange,
narrowspec, narrowspec,
obsolete, obsolete,
phases, phases,
pushkey, pushkey,
pycompat, pycompat,
repository, repository,
if missing: if missing:
msg = _("required features are not" msg = _("required features are not"
" supported in the destination:" " supported in the destination:"
" %s") % (', '.join(sorted(missing))) " %s") % (', '.join(sorted(missing)))
raise error.Abort(msg) raise error.Abort(msg)
pullop.trmanager = transactionmanager(repo, 'pull', remote.url()) pullop.trmanager = transactionmanager(repo, 'pull', remote.url())
with repo.wlock(), repo.lock(), pullop.trmanager: with repo.wlock(), repo.lock(), pullop.trmanager:
# Use the modern wire protocol, if available.
if remote.capable('exchangev2'):
exchangev2.pull(pullop)
durin42Unsubmitted
Not Done

What's the deal with this function-level import? It doesn't appear to be avoiding a cycle...

durin42: What's the deal with this function-level import? It doesn't appear to be avoiding a cycle...
indygregAuthorUnsubmitted
Not Done

At one time the series introduced a cycle. But the current series doesn't do that. So I suppose this can be rewritten to a file-level import for now.

indygreg: At one time the series introduced a cycle. But the current series doesn't do that. So I suppose…
else:
# This should ideally be in _pullbundle2(). However, it needs to run # This should ideally be in _pullbundle2(). However, it needs to run
# before discovery to avoid extra work. # before discovery to avoid extra work.
_maybeapplyclonebundle(pullop) _maybeapplyclonebundle(pullop)
streamclone.maybeperformlegacystreamclone(pullop) streamclone.maybeperformlegacystreamclone(pullop)
_pulldiscovery(pullop) _pulldiscovery(pullop)
if pullop.canusebundle2: if pullop.canusebundle2:
_fullpullbundle2(repo, pullop) _fullpullbundle2(repo, pullop)
_pullchangeset(pullop) _pullchangeset(pullop)
_pullphase(pullop) _pullphase(pullop)
_pullbookmarks(pullop) _pullbookmarks(pullop)
_pullobsolete(pullop) _pullobsolete(pullop)
# storing remotenames # storing remotenames
if repo.ui.configbool('experimental', 'remotenames'): if repo.ui.configbool('experimental', 'remotenames'):
logexchange.pullremotenames(repo, remote) logexchange.pullremotenames(repo, remote)
return pullop return pullop
# list of steps to perform discovery before pull # list of steps to perform discovery before pull

mercurial/exchangev2.py

  • This file was added.
# exchangev2.py - repository exchange for wire protocol version 2
#
# Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import
from .node import (
nullid,
)
from . import (
setdiscovery,
)
def pull(pullop):
"""Pull using wire protocol version 2."""
repo = pullop.repo
remote = pullop.remote
# Figure out what needs to be fetched.
common, fetch, remoteheads = _pullchangesetdiscovery(
repo, remote, pullop.heads, abortwhenunrelated=pullop.force)
def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
"""Determine which changesets need to be pulled."""
if heads:
knownnode = repo.changelog.hasnode
if all(knownnode(head) for head in heads):
return heads, False, heads
# TODO wire protocol version 2 is capable of more efficient discovery
# than setdiscovery. Consider implementing something better.
common, fetch, remoteheads = setdiscovery.findcommonheads(
repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated)
common = set(common)
remoteheads = set(remoteheads)
# If a remote head is filtered locally, put it back in the common set.
# See the comment in exchange._pulldiscoverychangegroup() for more.
if fetch and remoteheads:
nodemap = repo.unfiltered().changelog.nodemap
common |= {head for head in remoteheads if head in nodemap}
if set(remoteheads).issubset(common):
fetch = []
common.discard(nullid)
return common, fetch, remoteheads

mercurial/httppeer.py

# advertised from the "capabilities" wire protocol command. However, # advertised from the "capabilities" wire protocol command. However,
# version 2 of that command works differently. # version 2 of that command works differently.
# Maps to commands that are available. # Maps to commands that are available.
if name in ('branchmap', 'getbundle', 'known', 'lookup', 'pushkey'): if name in ('branchmap', 'getbundle', 'known', 'lookup', 'pushkey'):
return True return True
# Other concepts. # Other concepts.
if name in ('bundle2',): # TODO remove exchangev2 once we have a command implemented.
if name in ('bundle2', 'exchangev2'):
return True return True
# Alias command-* to presence of command of that name. # Alias command-* to presence of command of that name.
if name.startswith('command-'): if name.startswith('command-'):
return name[len('command-'):] in self._descriptor['commands'] return name[len('command-'):] in self._descriptor['commands']
return False return False

tests/test-wireproto-exchangev2.t

  • This file was added.
Tests for wire protocol version 2 exchange.
Tests in this file should be folded into existing tests once protocol
v2 has enough features that it can be enabled via #testcase in existing
tests.
$ . $TESTDIR/wireprotohelpers.sh
$ enablehttpv2client
$ hg init server-simple
$ enablehttpv2 server-simple
$ cd server-simple
$ cat >> .hg/hgrc << EOF
> [phases]
> publish = false
> EOF
$ echo a0 > a
$ echo b0 > b
$ hg -q commit -A -m 'commit 0'
$ echo a1 > a
$ hg commit -m 'commit 1'
$ hg phase --public -r .
$ echo a2 > a
$ hg commit -m 'commit 2'
$ hg -q up -r 0
$ echo b1 > b
$ hg -q commit -m 'head 2 commit 1'
$ echo b2 > b
$ hg -q commit -m 'head 2 commit 2'
$ hg serve -p $HGPORT -d --pid-file hg.pid -E error.log
$ cat hg.pid > $DAEMON_PIDS
$ cd ..
Test basic clone
$ hg --debug clone -U http://localhost:$HGPORT client-simple
using http://localhost:$HGPORT/
sending capabilities command
query 1; heads
sending 2 commands
sending command heads: {}
sending command known: {
'nodes': []
}
received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
received frame(size=43; request=1; stream=2; streamflags=; type=command-response; flags=continuation)
received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
received frame(size=11; request=3; stream=2; streamflags=; type=command-response; flags=continuation)
received frame(size=1; request=3; stream=2; streamflags=; type=command-response; flags=continuation)
received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos)

tests/wireprotohelpers.sh

enablehttpv2() { enablehttpv2() {
cat >> $1/.hg/hgrc << EOF cat >> $1/.hg/hgrc << EOF
[experimental] [experimental]
web.apiserver = true web.apiserver = true
web.api.http-v2 = true web.api.http-v2 = true
EOF EOF
} }
enablehttpv2client() {
cat >> $HGRCPATH << EOF
[experimental]
httppeer.advertise-v2 = true
EOF
}