This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5991

copies: do copy tracing based on ctx.p[12]copies() if configured
ClosedPublic

Authored by martinvonz on Feb 20 2019, 7:27 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG49ad315b39ee: copies: do copy tracing based on ctx.p[12]copies() if configured
rHG2d562af6014b: copies: do copy tracing based on ctx.p[12]copies() if configured
Summary

This adds an option to do copy tracing in a changeset-optimized
way. If the metadata is stored in filelogs, this is obviously going to
be suboptimal. The point is that it provides a way of transitioning to
changeset-stored metadata.

Some of the tests behave a little differently, but they all seem
resonable to me.

The config option may very well be renamed later when it's clearer
what options we want and how they will behave.

When the test suite is run with --extra-config-opt to use the new copy
tracing, all tests pass, besides test-copies.t (which fails in the
same way as you can see in this patch).

hg debugpathcopies 4.0 4.8 reports 82 copies. With this option
enabled, the only difference is this:

-mercurial/pure/bdiff.py -> mercurial/cffi/bdiff.py
+setup_bdiff_cffi.py -> mercurial/cffi/bdiff.py

I believe that happened because it was renamed in different ways on
different sides of a merge and the new algorithm arbitrarily prefers
copies that happened on p1. The runtime is about 0.85 seconds with the
old copy tracing and 5.7 seconds with the new copy tracing. That's
kind of slow, but actually better than I had expected.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

martinvonz created this revision.Feb 20 2019, 7:27 PM
Closed by commit rHG2d562af6014b: copies: do copy tracing based on ctx.p[12]copies() if configured (authored by martinvonz). · Explain WhyFeb 20 2019, 9:35 PM
This revision was automatically updated to reflect the committed changes.
martinvonz added a comment.Mar 20 2019, 2:36 AM

When the test suite is run with --extra-config-opt to use the new copy
tracing, all tests pass, besides test-copies.t (which fails in the
same way as you can see in this patch).

For the record, I don't know what I did wrong that time, because that seems to be a lie. These tests fail:

test-status.t:
The 'Other "bug" highlight' changed and is not buggy with the new copy tracing enabled

test-mv-cp-st-diff.t:
Different debug output from the new algorithm

test-commit-amend.t:
A rename inside a merge commit is now reported. The source file existed on both sides of the merge, so I suppose the old algorithm recorded it as being from p2.

Revision Contents
Changeset List

PathPackages
Mmercurial/configitems.py (3 lines)
Mmercurial/copies.py (74 lines)
Mtests/test-copies.t (13 lines)

Diff 14171

mercurial/configitems.py

default='on', default='on',
) )
coreconfigitem('experimental', 'copytrace.movecandidateslimit', coreconfigitem('experimental', 'copytrace.movecandidateslimit',
default=100, default=100,
) )
coreconfigitem('experimental', 'copytrace.sourcecommitlimit', coreconfigitem('experimental', 'copytrace.sourcecommitlimit',
default=100, default=100,
) )
coreconfigitem('experimental', 'copies.read-from',
default="filelog-only",
)
coreconfigitem('experimental', 'crecordtest', coreconfigitem('experimental', 'crecordtest',
default=None, default=None,
) )
coreconfigitem('experimental', 'directaccess', coreconfigitem('experimental', 'directaccess',
default=False, default=False,
) )
coreconfigitem('experimental', 'directaccess.revnums', coreconfigitem('experimental', 'directaccess.revnums',
default=False, default=False,

mercurial/copies.py

mb = b.manifest() mb = b.manifest()
return mb.filesnotin(ma, match=match) return mb.filesnotin(ma, match=match)
def _committedforwardcopies(a, b, match): def _committedforwardcopies(a, b, match):
"""Like _forwardcopies(), but b.rev() cannot be None (working copy)""" """Like _forwardcopies(), but b.rev() cannot be None (working copy)"""
# files might have to be traced back to the fctx parent of the last # files might have to be traced back to the fctx parent of the last
# one-side-only changeset, but not further back than that # one-side-only changeset, but not further back than that
repo = a._repo repo = a._repo
if repo.ui.config('experimental', 'copies.read-from') == 'compatibility':
return _changesetforwardcopies(a, b, match)
debug = repo.ui.debugflag and repo.ui.configbool('devel', 'debug.copies') debug = repo.ui.debugflag and repo.ui.configbool('devel', 'debug.copies')
dbg = repo.ui.debug dbg = repo.ui.debug
if debug: if debug:
dbg('debug.copies: looking into rename from %s to %s\n' dbg('debug.copies: looking into rename from %s to %s\n'
% (a, b)) % (a, b))
limit = _findlimit(repo, a, b) limit = _findlimit(repo, a, b)
if debug: if debug:
dbg('debug.copies: search limit: %d\n' % limit) dbg('debug.copies: search limit: %d\n' % limit)
if debug: if debug:
dbg('debug.copies: rename of: %s\n' % ofctx._path) dbg('debug.copies: rename of: %s\n' % ofctx._path)
cm[f] = ofctx.path() cm[f] = ofctx.path()
if debug: if debug:
dbg('debug.copies: time: %f seconds\n' dbg('debug.copies: time: %f seconds\n'
% (util.timer() - start)) % (util.timer() - start))
return cm return cm
def _changesetforwardcopies(a, b, match):
if a.rev() == node.nullrev:
return {}
repo = a.repo()
children = {}
cl = repo.changelog
missingrevs = cl.findmissingrevs(common=[a.rev()], heads=[b.rev()])
for r in missingrevs:
for p in cl.parentrevs(r):
if p == node.nullrev:
continue
if p not in children:
children[p] = [r]
else:
children[p].append(r)
roots = set(children) - set(missingrevs)
# 'work' contains 3-tuples of a (revision number, parent number, copies).
# The parent number is only used for knowing which parent the copies dict
# came from.
work = [(r, 1, {}) for r in roots]
heapq.heapify(work)
while work:
r, i1, copies1 = heapq.heappop(work)
if work and work[0][0] == r:
# We are tracing copies from both parents
r, i2, copies2 = heapq.heappop(work)
copies = {}
ctx = repo[r]
p1man, p2man = ctx.p1().manifest(), ctx.p2().manifest()
allcopies = set(copies1) | set(copies2)
# TODO: perhaps this filtering should be done as long as ctx
# is merge, whether or not we're tracing from both parent.
for dst in allcopies:
if not match(dst):
continue
if dst not in copies2:
# Copied on p1 side: mark as copy from p1 side if it didn't
# already exist on p2 side
if dst not in p2man:
copies[dst] = copies1[dst]
elif dst not in copies1:
# Copied on p2 side: mark as copy from p2 side if it didn't
# already exist on p1 side
if dst not in p1man:
copies[dst] = copies2[dst]
else:
# Copied on both sides: mark as copy from p1 side
copies[dst] = copies1[dst]
else:
copies = copies1
if r == b.rev():
return copies
for c in children[r]:
childctx = repo[c]
if r == childctx.p1().rev():
parent = 1
childcopies = childctx.p1copies()
else:
assert r == childctx.p2().rev()
parent = 2
childcopies = childctx.p2copies()
if not match.always():
childcopies = {dst: src for dst, src in childcopies.items()
if match(dst)}
childcopies = _chain(a, childctx, copies, childcopies)
heapq.heappush(work, (c, parent, childcopies))
assert False
def _forwardcopies(a, b, match=None): def _forwardcopies(a, b, match=None):
"""find {dst@b: src@a} copy mapping where a is an ancestor of b""" """find {dst@b: src@a} copy mapping where a is an ancestor of b"""
match = a.repo().narrowmatch(match) match = a.repo().narrowmatch(match)
# check for working copy # check for working copy
if b.rev() is None: if b.rev() is None:
if a == b.p1(): if a == b.p1():
# short-circuit to avoid issues with merge states # short-circuit to avoid issues with merge states

tests/test-copies.t

#testcases filelog compatibility
$ cat >> $HGRCPATH << EOF $ cat >> $HGRCPATH << EOF
> [alias] > [alias]
> l = log -G -T '{rev} {desc}\n{files}\n' > l = log -G -T '{rev} {desc}\n{files}\n'
> EOF > EOF
#if compatibility
$ cat >> $HGRCPATH << EOF
> [experimental]
> copies.read-from = compatibility
> EOF
#endif
$ REPONUM=0 $ REPONUM=0
$ newrepo() { $ newrepo() {
> cd $TESTTMP > cd $TESTTMP
> REPONUM=`expr $REPONUM + 1` > REPONUM=`expr $REPONUM + 1`
> hg init repo-$REPONUM > hg init repo-$REPONUM
> cd repo-$REPONUM > cd repo-$REPONUM
> } > }
x x
$ hg debugp1copies -r 2 $ hg debugp1copies -r 2
x -> z x -> z
$ hg debugp2copies -r 2 $ hg debugp2copies -r 2
It's a little weird that it shows up on both sides It's a little weird that it shows up on both sides
$ hg debugpathcopies 1 2 $ hg debugpathcopies 1 2
x -> z x -> z
$ hg debugpathcopies 0 2 $ hg debugpathcopies 0 2
x -> z x -> z (filelog !)
Copy file that exists on both sides of the merge, different content Copy file that exists on both sides of the merge, different content
$ newrepo $ newrepo
$ echo branch1 > x $ echo branch1 > x
$ hg ci -Aqm 'add x on branch 1' $ hg ci -Aqm 'add x on branch 1'
$ hg co -q null $ hg co -q null
$ echo branch2 > x $ echo branch2 > x
$ hg ci -Aqm 'add x on branch 2' $ hg ci -Aqm 'add x on branch 2'
| | y z | | y z
o | 1 rename x to z o | 1 rename x to z
|/ x z |/ x z
o 0 add x and y o 0 add x and y
x y x y
$ hg debugpathcopies 1 4 $ hg debugpathcopies 1 4
$ hg debugpathcopies 2 4 $ hg debugpathcopies 2 4
$ hg debugpathcopies 0 4 $ hg debugpathcopies 0 4
x -> z x -> z (filelog !)
y -> z (compatibility !)
$ hg debugpathcopies 1 5 $ hg debugpathcopies 1 5
$ hg debugpathcopies 2 5 $ hg debugpathcopies 2 5
$ hg debugpathcopies 0 5 $ hg debugpathcopies 0 5
x -> z x -> z