This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9945

debugdiscovery: add flags to run discovery on subsets of the local repo
ClosedPublic

Authored by marmoute on Feb 2 2021, 12:43 AM.

Download Raw Diff

Details

Reviewers

pulkit

Group Reviewers

hg-reviewers

Commits

rHG4f5e9a77ff7a: debugdiscovery: add flags to run discovery on subsets of the local repo

Summary

Generating new repository using strip of local clone is very expensive for large
repositories. And such large repository are the most likely to requires
debugging around discovery. So we add a simple way to run discovery using
provided sets of heads.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Feb 2 2021, 12:43 AM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptFeb 2 2021, 12:43 AM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

marmoute updated this revision to Diff 25455.Feb 2 2021, 9:33 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

pulkit accepted this revision.Feb 10 2021, 7:20 AM

This revision is now accepted and ready to land.Feb 10 2021, 7:20 AM

marmoute added a commit: rHG4f5e9a77ff7a: debugdiscovery: add flags to run discovery on subsets of the local repo.Feb 10 2021, 8:06 AM

Closed by commit rHG4f5e9a77ff7a: debugdiscovery: add flags to run discovery on subsets of the local repo (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/debugcommands.py (62 lines)
M			tests/test-completion.t (2 lines)
M			tests/test-setdiscovery.t (136 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	25453		Feb 2 2021, 12:43 AM	★	★
Diff 2	25455		Feb 2 2021, 9:33 AM	★	★
Diff 3	25462		Feb 2 2021, 12:01 PM	★	★
Diff 4	25521	rHG4f5e9a77ff7ab343d030f27ab92e334e8d9882d7	Jan 29 2021, 9:23 AM	★	★

Diff 25521

mercurial/debugcommands.py

	obsutil,			obsutil,
	pathutil,			pathutil,
	phases,			phases,
	policy,			policy,
	pvec,			pvec,
	pycompat,			pycompat,
	registrar,			registrar,
	repair,			repair,
				repoview,
	revlog,			revlog,
	revset,			revset,
	revsetlang,			revsetlang,
	scmutil,			scmutil,
	setdiscovery,			setdiscovery,
	simplemerge,			simplemerge,
	sshpeer,			sshpeer,
	sslutil,			sslutil,
	(			(
	b'',			b'',
	b'nonheads',			b'nonheads',
	None,			None,
	_(b'use old-style discovery with non-heads included'),			_(b'use old-style discovery with non-heads included'),
	),			),
	(b'', b'rev', [], b'restrict discovery to this set of revs'),			(b'', b'rev', [], b'restrict discovery to this set of revs'),
	(b'', b'seed', b'12323', b'specify the random seed use for discovery'),			(b'', b'seed', b'12323', b'specify the random seed use for discovery'),
				(
				b'',
				b'local-as-revs',
				"",
				'treat local has having these revisions only',
				),
				(
				b'',
				b'remote-as-revs',
				"",
				'use local as remote, with only these these revisions',
				),
	]			]
	+ cmdutil.remoteopts,			+ cmdutil.remoteopts,
	_(b'[--rev REV] [OTHER]'),			_(b'[--rev REV] [OTHER]'),
	)			)
	def debugdiscovery(ui, repo, remoteurl=b"default", **opts):			def debugdiscovery(ui, repo, remoteurl=b"default", **opts):
	"""runs the changeset discovery protocol in isolation"""			"""runs the changeset discovery protocol in isolation

				The local peer can be "replaced" by a subset of the local repository by
				using the `--local-as-revs` flag. Int he same way, usual `remote` peer can
				be "replaced" by a subset of the local repository using the
				`--local-as-revs` flag. This is useful to efficiently debug pathological
				discovery situation.
				"""
	opts = pycompat.byteskwargs(opts)			opts = pycompat.byteskwargs(opts)
				unfi = repo.unfiltered()

				# setup potential extra filtering
				local_revs = opts[b"local_as_revs"]
				remote_revs = opts[b"remote_as_revs"]

				# make sure tests are repeatable
				random.seed(int(opts[b'seed']))

				if not remote_revs:

	remoteurl, branches = hg.parseurl(ui.expandpath(remoteurl))			remoteurl, branches = hg.parseurl(ui.expandpath(remoteurl))
	remote = hg.peer(repo, opts, remoteurl)			remote = hg.peer(repo, opts, remoteurl)
	ui.status(_(b'comparing with %s\n') % util.hidepassword(remoteurl))			ui.status(_(b'comparing with %s\n') % util.hidepassword(remoteurl))
				else:
				branches = (None, [])
				remote_filtered_revs = scmutil.revrange(
				unfi, [b"not (::(%s))" % remote_revs]
				)
				remote_filtered_revs = frozenset(remote_filtered_revs)

	# make sure tests are repeatable			def remote_func(x):
	random.seed(int(opts[b'seed']))			return remote_filtered_revs

				repoview.filtertable[b'debug-discovery-remote-filter'] = remote_func

				remote = repo.peer()
				remote._repo = remote._repo.filtered(b'debug-discovery-remote-filter')

				if local_revs:
				local_filtered_revs = scmutil.revrange(
				unfi, [b"not (::(%s))" % local_revs]
				)
				local_filtered_revs = frozenset(local_filtered_revs)

				def local_func(x):
				return local_filtered_revs

				repoview.filtertable[b'debug-discovery-local-filter'] = local_func
				repo = repo.filtered(b'debug-discovery-local-filter')

	data = {}			data = {}
	if opts.get(b'old'):			if opts.get(b'old'):

	def doit(pushedrevs, remoteheads, remote=remote):			def doit(pushedrevs, remoteheads, remote=remote):
	if not util.safehasattr(remote, b'branches'):			if not util.safehasattr(remote, b'branches'):
	# enable in-client legacy support			# enable in-client legacy support
	remote = localrepo.locallegacypeer(remote.local())			remote = localrepo.locallegacypeer(remote.local())

tests/test-completion.t

	debugcommands:			debugcommands:
	debugcomplete: options			debugcomplete: options
	debugcreatestreamclonebundle:			debugcreatestreamclonebundle:
	debugdag: tags, branches, dots, spaces			debugdag: tags, branches, dots, spaces
	debugdata: changelog, manifest, dir			debugdata: changelog, manifest, dir
	debugdate: extended			debugdate: extended
	debugdeltachain: changelog, manifest, dir, template			debugdeltachain: changelog, manifest, dir, template
	debugdirstate: nodates, dates, datesort			debugdirstate: nodates, dates, datesort
	debugdiscovery: old, nonheads, rev, seed, ssh, remotecmd, insecure			debugdiscovery: old, nonheads, rev, seed, local-as-revs, remote-as-revs, ssh, remotecmd, insecure
	debugdownload: output			debugdownload: output
	debugextensions: template			debugextensions: template
	debugfileset: rev, all-files, show-matcher, show-stage			debugfileset: rev, all-files, show-matcher, show-stage
	debugformat: template			debugformat: template
	debugfsinfo:			debugfsinfo:
	debuggetbundle: head, common, type			debuggetbundle: head, common, type
	debugignore:			debugignore:
	debugindex: changelog, manifest, dir, template			debugindex: changelog, manifest, dir, template

tests/test-setdiscovery.t

	heads: 1			heads: 1
	roots: 1			roots: 1
	first undecided set: 1			first undecided set: 1
	heads: 1			heads: 1
	roots: 1			roots: 1
	common: 0			common: 0
	missing: 1			missing: 1
	common heads: 66f7d451a68b			common heads: 66f7d451a68b

				$ cd ..


				Test debuging discovery using different subset of the same repository
				=====================================================================

				remote is a local subset
				------------------------

				remote will be last 25 heads of the local graph

				$ cd $TESTTMP/manyheads
				$ hg -R a debugdiscovery \
				> --debug \
				> --remote-as-revs 'last(heads(all()), 25)' \
				> --config devel.discovery.randomize=false
				query 1; heads
				searching for changes
				all remote heads known locally
				elapsed time: * seconds (glob)
				round-trips: 1
				heads summary:
				total common heads: 25
				also local heads: 25
				also remote heads: 25
				both: 25
				local heads: 260
				common: 25
				missing: 235
				remote heads: 25
				common: 25
				unknown: 0
				local changesets: 1340
				common: 400
				heads: 25
				roots: 1
				missing: 940
				heads: 235
				roots: 235
				first undecided set: 940
				heads: 235
				roots: 235
				common: 0
				missing: 940
				common heads: 0dfd965d91c6 0fe09b60448d 14a17233ce9d 175c0a3072cf 1c51e2c80832 1e51600e0698 24eb5f9bdbab 25ce09526613 36bd00abde57 426989fdefa0 596d87362679 5dd1039ea5c0 5ef24f022278 5f230dc19419 80b39998accb 88f40688ffb5 9e37ddf8c632 abf4d55b075e b2ce801fddfe b368b6ac3ce3 c959bf2e869c c9fba6ba4e2e d783207cf649 d9a51e256f21 e3717a4e3753

				local is a local subset
				------------------------

				remote will be last 25 heads of the local graph

				$ cd $TESTTMP/manyheads
				$ hg -R a debugdiscovery b \
				> --debug \
				> --local-as-revs 'first(heads(all()), 25)' \
				> --config devel.discovery.randomize=false
				comparing with b
				query 1; heads
				searching for changes
				taking quick initial sample
				query 2; still undecided: 375, sample size is: 81
				sampling from both directions
				query 3; still undecided: 3, sample size is: 3
				3 total queries *s (glob)
				elapsed time: * seconds (glob)
				round-trips: 3
				heads summary:
				total common heads: 1
				also local heads: 0
				also remote heads: 0
				both: 0
				local heads: 25
				common: 0
				missing: 25
				remote heads: 1
				common: 0
				unknown: 1
				local changesets: 400
				common: 300
				heads: 1
				roots: 1
				missing: 100
				heads: 25
				roots: 25
				first undecided set: 400
				heads: 25
				roots: 1
				common: 300
				missing: 100
				common heads: 3ee37d65064a

				both local and remove are subset
				------------------------

				remote will be last 25 heads of the local graph

				$ cd $TESTTMP/manyheads
				$ hg -R a debugdiscovery \
				> --debug \
				> --local-as-revs 'first(heads(all()), 25)' \
				> --remote-as-revs 'last(heads(all()), 25)' \
				> --config devel.discovery.randomize=false
				query 1; heads
				searching for changes
				taking quick initial sample
				query 2; still undecided: 375, sample size is: 81
				sampling from both directions
				query 3; still undecided: 3, sample size is: 3
				3 total queries in *s (glob)
				elapsed time: * seconds (glob)
				round-trips: 3
				heads summary:
				total common heads: 1
				also local heads: 0
				also remote heads: 0
				both: 0
				local heads: 25
				common: 0
				missing: 25
				remote heads: 25
				common: 0
				unknown: 25
				local changesets: 400
				common: 300
				heads: 1
				roots: 1
				missing: 100
				heads: 25
				roots: 25
				first undecided set: 400
				heads: 25
				roots: 1
				common: 300
				missing: 100
				common heads: 3ee37d65064a