This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9503

copies: avoid materializing a full directory map during copy tracing
ClosedPublic

Authored by spectral on Dec 2 2020, 2:43 PM.

Download Raw Diff

Details

Reviewers

pulkit

Group Reviewers

hg-reviewers

Commits

rHGb9588ff9b66a: copies: avoid materializing a full directory map during copy tracing

Summary

Materializing a full copy of every directory in a treemanifest repo can be quite
expensive, even with a narrow matcher. For flat manifest repos, this should be
equivalent - it will still materialize (and cache) a dict of all of the dirs
inside of the manifest object, we just don't get a copy of it.

In a repo I have here, this brings the time for a simple rebase from 11.197s to
4.609s.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

spectral created this revision.Dec 2 2020, 2:43 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptDec 2 2020, 2:43 PM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

spectral updated this revision to Diff 23973.Dec 2 2020, 2:49 PM

pulkit accepted this revision.Dec 3 2020, 3:13 AM

This revision is now accepted and ready to land.Dec 3 2020, 3:13 AM

spectral added a commit: rHGb9588ff9b66a: copies: avoid materializing a full directory map during copy tracing.Dec 3 2020, 4:37 AM

Closed by commit rHGb9588ff9b66a: copies: avoid materializing a full directory map during copy tracing (authored by spectral). · Explain Why

This revision was automatically updated to reflect the committed changes.

baymax added a parent revision: D9509: rebase: clear merge state when aborting in-memory merge on dirty working copy.Dec 3 2020, 6:15 AM

baymax added a child revision: D9504: tests: use the right python when running dummyssh for narrow.

baymax updated this revision to Diff 23996.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/copies.py (5 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	23967		Dec 2 2020, 2:43 PM	★	★
Diff 2	23973		Dec 2 2020, 2:49 PM	★	★
Diff 3	23988	rHGb9588ff9b66a7a32a3822b6a6de0a4363ed435f2	Dec 2 2020, 2:05 PM	★	★
Diff 4	23996		Dec 3 2020, 6:15 AM	★	★

Status	Author	Revision
Closed	spectral	D9511 statprof: separate functions and "line", assume 4 digit line numbers
Closed	spectral	D9510 statprof: fix off-by-one-line error in output
Closed	marmoute	D9513 phab-refresh: do not error out when the stack is empty
Closed	marmoute	D9507 run-tests: allow some slack about 'waiting on lock' message
Closed	marmoute	D9506 pull: flush stdin after the `pull from` message
Closed	marmoute	D9505 tests: explicitly skip the lock warning in some remotefilelog tests
Closed	marmoute	D9504 tests: use the right python when running dummyssh for narrow
Closed	spectral	D9503 copies: avoid materializing a full directory map during copy tracing
Closed	martinvonz	D9509 rebase: clear merge state when aborting in-memory merge on dirty working copy
Closed	martinvonz	D9508 tests: show that in-memory rebase leaves state when working copy is dirty

Diff 23988

mercurial/copies.py


	ctx: the context for one of the sides			ctx: the context for one of the sides
	copy: files copied on the same side (as ctx)			copy: files copied on the same side (as ctx)
	fullcopy: files copied on the same side (as ctx), including those that			fullcopy: files copied on the same side (as ctx), including those that
	merge.manifestmerge() won't care about			merge.manifestmerge() won't care about
	addedfiles: added files on the other side (compared to ctx)			addedfiles: added files on the other side (compared to ctx)
	"""			"""
	# generate a directory move map			# generate a directory move map
	d = ctx.dirs()
	invalid = set()			invalid = set()
	dirmove = {}			dirmove = {}

	# examine each file copy for a potential directory move, which is			# examine each file copy for a potential directory move, which is
	# when all the files in a directory are moved to a new directory			# when all the files in a directory are moved to a new directory
	for dst, src in pycompat.iteritems(fullcopy):			for dst, src in pycompat.iteritems(fullcopy):
	dsrc, ddst = pathutil.dirname(src), pathutil.dirname(dst)			dsrc, ddst = pathutil.dirname(src), pathutil.dirname(dst)
	if dsrc in invalid:			if dsrc in invalid:
	# already seen to be uninteresting			# already seen to be uninteresting
	continue			continue
	elif dsrc in d and ddst in d:			elif ctx.hasdir(dsrc) and ctx.hasdir(ddst):
	# directory wasn't entirely moved locally			# directory wasn't entirely moved locally
	invalid.add(dsrc)			invalid.add(dsrc)
	elif dsrc in dirmove and dirmove[dsrc] != ddst:			elif dsrc in dirmove and dirmove[dsrc] != ddst:
	# files from the same directory moved to two different places			# files from the same directory moved to two different places
	invalid.add(dsrc)			invalid.add(dsrc)
	else:			else:
	# looks good so far			# looks good so far
	dirmove[dsrc] = ddst			dirmove[dsrc] = ddst

	for i in invalid:			for i in invalid:
	if i in dirmove:			if i in dirmove:
	del dirmove[i]			del dirmove[i]
	del d, invalid			del invalid

	if not dirmove:			if not dirmove:
	return {}, {}			return {}, {}

	dirmove = {k + b"/": v + b"/" for k, v in pycompat.iteritems(dirmove)}			dirmove = {k + b"/": v + b"/" for k, v in pycompat.iteritems(dirmove)}

	for d in dirmove:			for d in dirmove:
	repo.ui.debug(			repo.ui.debug(