This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9801

discovery: add a devel.discovery.exchange-heads
ClosedPublic

Authored by marmoute on Jan 15 2021, 7:26 PM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG6ee9bd69ff7f: discovery: add a devel.discovery.exchange-heads
Summary

Currently all discovery start with testing local and remote heads. For analysis
purpose we make it possible to disable that initial "handshake" and start
discovery with the whole repository as undecided.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Jan 15 2021, 7:26 PM
pulkit added a subscriber: pulkit.Jan 16 2021, 4:04 AM
pulkit added inline comments.
mercurial/configitems.py
705

hm, s/will not start but with/will not start with?

mercurial/setdiscovery.py
447

Any reason to not drop False?

marmoute updated this revision to Diff 24976.Jan 16 2021, 8:04 AM
pulkit accepted this revision.Jan 17 2021, 11:42 AM
This revision is now accepted and ready to land.Jan 17 2021, 11:42 AM
marmoute added a commit: rHG6ee9bd69ff7f: discovery: add a devel.discovery.exchange-heads.Jan 17 2021, 1:44 PM
Closed by commit rHG6ee9bd69ff7f: discovery: add a devel.discovery.exchange-heads (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 25080

mercurial/configitems.py

b'debug.repo-filters', b'debug.repo-filters',
default=False, default=False,
) )
coreconfigitem( coreconfigitem(
b'devel', b'devel',
b'debug.peer-request', b'debug.peer-request',
default=False, default=False,
) )
# If discovery.exchange-heads is False, the discovery will not start with
pulkitUnsubmitted
Not Done

hm, s/will not start but with/will not start with?

pulkit: hm, `s/will not start but with/will not start with`?
# remote head fetching and local head querying.
coreconfigitem(
b'devel',
b'discovery.exchange-heads',
default=True,
)
# If discovery.grow-sample is False, the sample size used in set discovery will # If discovery.grow-sample is False, the sample size used in set discovery will
# not be increased through the process # not be increased through the process
coreconfigitem( coreconfigitem(
b'devel', b'devel',
b'discovery.grow-sample', b'discovery.grow-sample',
default=True, default=True,
) )
# discovery.grow-sample.rate control the rate at which the sample grow # discovery.grow-sample.rate control the rate at which the sample grow

mercurial/setdiscovery.py

clnode = cl.node clnode = cl.node
clrev = cl.rev clrev = cl.rev
if ancestorsof is not None: if ancestorsof is not None:
ownheads = [clrev(n) for n in ancestorsof] ownheads = [clrev(n) for n in ancestorsof]
else: else:
ownheads = [rev for rev in cl.headrevs() if rev != nullrev] ownheads = [rev for rev in cl.headrevs() if rev != nullrev]
initial_head_exchange = ui.configbool(b'devel', b'discovery.exchange-heads')
# We also ask remote about all the local heads. That set can be arbitrarily # We also ask remote about all the local heads. That set can be arbitrarily
# large, so we used to limit it size to `initialsamplesize`. We no longer # large, so we used to limit it size to `initialsamplesize`. We no longer
# do as it proved counter productive. The skipped heads could lead to a # do as it proved counter productive. The skipped heads could lead to a
# large "undecided" set, slower to be clarified than if we asked the # large "undecided" set, slower to be clarified than if we asked the
# question for all heads right away. # question for all heads right away.
# #
# We are already fetching all server heads using the `heads` commands, # We are already fetching all server heads using the `heads` commands,
# sending a equivalent number of heads the other way should not have a # sending a equivalent number of heads the other way should not have a
# of local heads. # of local heads.
# #
# The only case where we can be more efficient using `known` request on # The only case where we can be more efficient using `known` request on
# ancestors are case were all the "missing" local heads are based on a # ancestors are case were all the "missing" local heads are based on a
# few changeset, also "missing". This means we would have a "complex" # few changeset, also "missing". This means we would have a "complex"
# graph (with many heads) attached to, but very independant to a the # graph (with many heads) attached to, but very independant to a the
# "simple" graph on the server. This is a fairly usual case and have # "simple" graph on the server. This is a fairly usual case and have
# not been met in the wild so far. # not been met in the wild so far.
if initial_head_exchange:
if remote.limitedarguments: if remote.limitedarguments:
sample = _limitsample(ownheads, initialsamplesize) sample = _limitsample(ownheads, initialsamplesize)
# indices between sample and externalized version must match # indices between sample and externalized version must match
sample = list(sample) sample = list(sample)
else: else:
sample = ownheads sample = ownheads
ui.debug(b"query 1; heads\n") ui.debug(b"query 1; heads\n")
roundtrips += 1 roundtrips += 1
with remote.commandexecutor() as e: with remote.commandexecutor() as e:
fheads = e.callcommand(b'heads', {}) fheads = e.callcommand(b'heads', {})
fknown = e.callcommand( fknown = e.callcommand(
b'known', b'known',
{ {
b'nodes': [clnode(r) for r in sample], b'nodes': [clnode(r) for r in sample],
}, },
) )
srvheadhashes, yesno = fheads.result(), fknown.result() srvheadhashes, yesno = fheads.result(), fknown.result()
if audit is not None: if audit is not None:
audit[b'total-roundtrips'] = 1 audit[b'total-roundtrips'] = 1
if cl.tip() == nullid: if cl.tip() == nullid:
if srvheadhashes != [nullid]: if srvheadhashes != [nullid]:
return [nullid], True, srvheadhashes return [nullid], True, srvheadhashes
return [nullid], False, [] return [nullid], False, []
else:
# we still need the remote head for the function return
with remote.commandexecutor() as e:
fheads = e.callcommand(b'heads', {})
srvheadhashes = fheads.result()
# start actual discovery (we note this before the next "if" for # start actual discovery (we note this before the next "if" for
# compatibility reasons) # compatibility reasons)
ui.status(_(b"searching for changes\n")) ui.status(_(b"searching for changes\n"))
knownsrvheads = [] # revnos of remote heads that are known locally knownsrvheads = [] # revnos of remote heads that are known locally
for node in srvheadhashes: for node in srvheadhashes:
if node == nullid: if node == nullid:
continue continue
try: try:
knownsrvheads.append(clrev(node)) knownsrvheads.append(clrev(node))
# Catches unknown and filtered nodes. # Catches unknown and filtered nodes.
except error.LookupError: except error.LookupError:
continue continue
if initial_head_exchange:
# early exit if we know all the specified remote heads already # early exit if we know all the specified remote heads already
if len(knownsrvheads) == len(srvheadhashes): if len(knownsrvheads) == len(srvheadhashes):
ui.debug(b"all remote heads known locally\n") ui.debug(b"all remote heads known locally\n")
return srvheadhashes, False, srvheadhashes return srvheadhashes, False, srvheadhashes
if len(sample) == len(ownheads) and all(yesno): if len(sample) == len(ownheads) and all(yesno):
ui.note(_(b"all local changesets known remotely\n")) ui.note(_(b"all local changesets known remotely\n"))
ownheadhashes = [clnode(r) for r in ownheads] ownheadhashes = [clnode(r) for r in ownheads]
return ownheadhashes, True, srvheadhashes return ownheadhashes, True, srvheadhashes
# full blown discovery # full blown discovery
# if the server has a limit to its arguments size, we can't grow the sample. # if the server has a limit to its arguments size, we can't grow the sample.
hard_limit_sample = remote.limitedarguments hard_limit_sample = remote.limitedarguments
grow_sample = local.ui.configbool(b'devel', b'discovery.grow-sample') grow_sample = local.ui.configbool(b'devel', b'discovery.grow-sample')
hard_limit_sample = hard_limit_sample and grow_sample hard_limit_sample = hard_limit_sample and grow_sample
randomize = ui.configbool(b'devel', b'discovery.randomize') randomize = ui.configbool(b'devel', b'discovery.randomize')
disco = partialdiscovery( disco = partialdiscovery(
local, ownheads, hard_limit_sample, randomize=randomize local, ownheads, hard_limit_sample, randomize=randomize
) )
if initial_head_exchange:
# treat remote heads (and maybe own heads) as a first implicit sample # treat remote heads (and maybe own heads) as a first implicit sample
# response # response
disco.addcommons(knownsrvheads) disco.addcommons(knownsrvheads)
disco.addinfo(zip(sample, yesno)) disco.addinfo(zip(sample, yesno))
full = False full = not initial_head_exchange
pulkitUnsubmitted
Not Done

Any reason to not drop False?

pulkit: Any reason to not drop `False`?
progress = ui.makeprogress(_(b'searching'), unit=_(b'queries')) progress = ui.makeprogress(_(b'searching'), unit=_(b'queries'))
while not disco.iscomplete(): while not disco.iscomplete():
if full or disco.hasinfo(): if full or disco.hasinfo():
if full: if full:
ui.note(_(b"sampling from both directions\n")) ui.note(_(b"sampling from both directions\n"))
else: else:
ui.debug(b"taking initial sample\n") ui.debug(b"taking initial sample\n")

tests/test-setdiscovery.t

roots: 260 roots: 260
first undecided set: 1340 first undecided set: 1340
heads: 260 heads: 260
roots: 1 roots: 1
common: 300 common: 300
missing: 1040 missing: 1040
common heads: 3ee37d65064a common heads: 3ee37d65064a
$ hg -R a debugdiscovery b --debug --config devel.discovery.randomize=false --config devel.discovery.grow-sample.rate=1.01 $ hg -R a debugdiscovery b --debug --config devel.discovery.exchange-heads=false --config devel.discovery.randomize=false --config devel.discovery.grow-sample.rate=1.01
comparing with b comparing with b
query 1; heads
searching for changes searching for changes
taking quick initial sample
query 2; still undecided: 1080, sample size is: 100
sampling from both directions sampling from both directions
query 3; still undecided: 980, sample size is: 200 query 1; still undecided: 1340, sample size is: 200
sampling from both directions
query 2; still undecided: 795, sample size is: 202
sampling from both directions sampling from both directions
query 4; still undecided: 497, sample size is: 202 query 3; still undecided: 525, sample size is: 204
sampling from both directions sampling from both directions
query 5; still undecided: 294, sample size is: 204 query 4; still undecided: 252, sample size is: 206
sampling from both directions sampling from both directions
query 6; still undecided: 90, sample size is: 90 query 5; still undecided: 44, sample size is: 44
6 total queries in *s (glob) 5 total queries in *s (glob)
elapsed time: * seconds (glob) elapsed time: * seconds (glob)
round-trips: 6 round-trips: 5
heads summary: heads summary:
total common heads: 1 total common heads: 1
also local heads: 0 also local heads: 0
also remote heads: 0 also remote heads: 0
both: 0 both: 0
local heads: 260 local heads: 260
common: 0 common: 0
missing: 260 missing: 260