This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8982

merge: disable `m2-vs-ma` diff optimization in case of flat manifests
Changes PlannedPublic

Authored by pulkit on Sep 5 2020, 3:08 AM.

Details

Reviewers
marmoute
Group Reviewers
hg-reviewers
Summary

4d504e541d3d1a139f8ebbf8c405291140fd1853 introduced an optimization where in
some cases we first diff ancestor and second parent, get the list of files
changed between them and then only diff for these files between first parent and
second parent.

However as the commit message mentions, this optimization is only helpful when
rebasing very old commits and using treemanifest. On flat manifests, this
optimization rather adds a bit of overhead.

This makes optimization not much useful on flat manifests and can be disabled.

Now, while working on solving criss-cross merge issues where we want to create
new filenode for some files in initial merges, this optimization was causing
issues. Since it's filters file for which diff needs to be performed, the files
we are concerned with gets filters out and we don't get to store anything
related to those files.

Hence, want to disabling the optimization for flat manifests.

The tests changes are result of fact that we diff more files now in m1-vs-m2
comparison.

Small note: all these criss cross and new filenode thing I am talking about will
be behind a config flag. So nothing invasive yet :)

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

pulkit created this revision.Sep 5 2020, 3:08 AM
marmoute accepted this revision.Sep 11 2020, 5:05 AM
marmoute added a subscriber: marmoute.

Looks good. Out of curiosity, did you do any performance measurement of this?

pulkit planned changes to this revision.Sep 30 2020, 8:52 AM

I will get back to this later. This is no longer needed for the merge work I am doing.

Revision Contents
Changeset List

Diff 22668

mercurial/merge.py

# care for the dirty state of the working directory. # care for the dirty state of the working directory.
if any(wctx.sub(s).dirty() for s in wctx.substate): if any(wctx.sub(s).dirty() for s in wctx.substate):
m1[b'.hgsubstate'] = modifiednodeid m1[b'.hgsubstate'] = modifiednodeid
# Don't use m2-vs-ma optimization if: # Don't use m2-vs-ma optimization if:
# - ma is the same as m1 or m2, which we're just going to diff again later # - ma is the same as m1 or m2, which we're just going to diff again later
# - The caller specifically asks for a full diff, which is useful during bid # - The caller specifically asks for a full diff, which is useful during bid
# merge. # merge.
if pa not in ([wctx, p2] + wctx.parents()) and not forcefulldiff: # - flat manifests are use (reasoning below)
#
# The optimization is more effective when treemanifests are in play and
# this saves the time by avoiding loading almost the entire tree.
# Meanwhile this optimization also adds a bit of overhead for flat
# manifests especially when rebasing very recent commits because of
# additional diff call
# (refer commit message of 4d504e541d3d1a139f8ebbf8c405291140fd1853)
#
# Rebasing recent changesets is more common case than rebasing a cset
# which is 10k commits old
# Hence let's disable this optimization for flat manifests
if (
pa not in ([wctx, p2] + wctx.parents())
and not forcefulldiff
and scmutil.istreemanifest(repo)
):
# Identify which files are relevant to the merge, so we can limit the # Identify which files are relevant to the merge, so we can limit the
# total m1-vs-m2 diff to just those files. This has significant # total m1-vs-m2 diff to just those files. This has significant
# performance benefits in large repositories. # performance benefits in large repositories.
relevantfiles = set(ma.diff(m2).keys()) relevantfiles = set(ma.diff(m2).keys())
# For copied and moved files, we need to add the source file too. # For copied and moved files, we need to add the source file too.
for copykey, copyvalue in pycompat.iteritems(branch_copies1.copy): for copykey, copyvalue in pycompat.iteritems(branch_copies1.copy):
if copyvalue in relevantfiles: if copyvalue in relevantfiles:

tests/test-graft.t

on local side: on local side:
src: 'c' -> dst: 'b' src: 'c' -> dst: 'b'
checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: True, partial: False branchmerge: True, force: True, partial: False
ancestor: 4c60f11aa304, local: 6b9e5368ca4e+, remote: 97f8bfe72746 ancestor: 4c60f11aa304, local: 6b9e5368ca4e+, remote: 97f8bfe72746
e: remote is newer -> g e: remote is newer -> g
getting e getting e
b: remote unchanged -> k
c: local not present, remote unchanged -> ka
committing files: committing files:
e e
committing manifest committing manifest
committing changelog committing changelog
updating the branch cache updating the branch cache
$ HGEDITOR=cat hg graft 4 3 --log --debug $ HGEDITOR=cat hg graft 4 3 --log --debug
scanning for duplicate grafts scanning for duplicate grafts
grafting 4:9c233e8e184d "4" grafting 4:9c233e8e184d "4"
all copies found (* = to merge, ! = divergent, % = renamed and deleted): all copies found (* = to merge, ! = divergent, % = renamed and deleted):
on local side: on local side:
src: 'c' -> dst: 'b' src: 'c' -> dst: 'b'
checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: True, partial: False branchmerge: True, force: True, partial: False
ancestor: 4c60f11aa304, local: 1905859650ec+, remote: 9c233e8e184d ancestor: 4c60f11aa304, local: 1905859650ec+, remote: 9c233e8e184d
d: remote is newer -> g d: remote is newer -> g
getting d getting d
b: remote unchanged -> k
c: local not present, remote unchanged -> ka
preserving e for resolve of e preserving e for resolve of e
e: versions differ -> m (premerge) e: versions differ -> m (premerge)
picked tool ':merge' for e (binary False symlink False changedelete False) picked tool ':merge' for e (binary False symlink False changedelete False)
merging e merging e
my e@1905859650ec+ other e@9c233e8e184d ancestor e@4c60f11aa304 my e@1905859650ec+ other e@9c233e8e184d ancestor e@4c60f11aa304
e: versions differ -> m (merge) e: versions differ -> m (merge)
picked tool ':merge' for e (binary False symlink False changedelete False) picked tool ':merge' for e (binary False symlink False changedelete False)
my e@1905859650ec+ other e@9c233e8e184d ancestor e@4c60f11aa304 my e@1905859650ec+ other e@9c233e8e184d ancestor e@4c60f11aa304
src: 'a' -> dst: 'b' * src: 'a' -> dst: 'b' *
on remote side: on remote side:
src: 'a' -> dst: 'b' * src: 'a' -> dst: 'b' *
checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: True, partial: False branchmerge: True, force: True, partial: False
ancestor: b592ea63bb0c, local: 7e61b508e709+, remote: 7a4785234d87 ancestor: b592ea63bb0c, local: 7e61b508e709+, remote: 7a4785234d87
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
g: local not present, remote unchanged -> ka
nothing to commit, clearing merge state nothing to commit, clearing merge state
note: graft of 13:7a4785234d87 created no changes to commit note: graft of 13:7a4785234d87 created no changes to commit
$ hg log -r 'destination(13)' $ hg log -r 'destination(13)'
All copies of a cset All copies of a cset
$ hg log -r 'origin(13) or destination(origin(13))' $ hg log -r 'origin(13) or destination(origin(13))'
changeset: 2:5c095ad7e90f changeset: 2:5c095ad7e90f
user: test user: test
date: Thu Jan 01 00:00:00 1970 +0000 date: Thu Jan 01 00:00:00 1970 +0000

tests/test-issue1802.t

755 * a 755 * a
Simulate a Windows merge: Simulate a Windows merge:
$ hg --config extensions.n=$TESTTMP/noexec.py merge --debug $ hg --config extensions.n=$TESTTMP/noexec.py merge --debug
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: a03b0deabf2b, local: d6fa54f68ae1+, remote: 2d8bcf2dda39 ancestor: a03b0deabf2b, local: d6fa54f68ae1+, remote: 2d8bcf2dda39
b: ancestor missing, remote missing -> kn
a: update permissions -> e a: update permissions -> e
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit) (branch merge, don't forget to commit)
Simulate a Windows commit: Simulate a Windows commit:
$ hg --config extensions.n=$TESTTMP/noexec.py commit -m 'win: merge' $ hg --config extensions.n=$TESTTMP/noexec.py commit -m 'win: merge'
$ hg manifest -v $ hg manifest -v
755 * a 755 * a
644 b 644 b
$ cd .. $ cd ..

tests/test-issue522.t

$ hg ci -qAm 'add bar' $ hg ci -qAm 'add bar'
$ hg merge --debug $ hg merge --debug
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: bbd179dfa0a7, local: 71766447bdbb+, remote: 4d9e78aaceee ancestor: bbd179dfa0a7, local: 71766447bdbb+, remote: 4d9e78aaceee
foo: remote is newer -> g foo: remote is newer -> g
getting foo getting foo
bar: ancestor missing, remote missing -> kn
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit) (branch merge, don't forget to commit)
$ hg debugstate | grep foo $ hg debugstate | grep foo
m 0 -2 unset foo m 0 -2 unset foo
$ hg st -A foo $ hg st -A foo
M foo M foo

tests/test-issue672.t

checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: 81f4b099af3d, local: c64f439569a9+, remote: c12dcd37c90a ancestor: 81f4b099af3d, local: c64f439569a9+, remote: c12dcd37c90a
1: other deleted -> r 1: other deleted -> r
removing 1 removing 1
1a: remote created -> g 1a: remote created -> g
getting 1a getting 1a
2: remote unchanged -> k
1 files updated, 0 files merged, 1 files removed, 0 files unresolved 1 files updated, 0 files merged, 1 files removed, 0 files unresolved
(branch merge, don't forget to commit) (branch merge, don't forget to commit)
$ hg ci -m merge1 # 3 $ hg ci -m merge1 # 3
$ hg co -C 2 $ hg co -C 2
1 files updated, 0 files merged, 1 files removed, 0 files unresolved 1 files updated, 0 files merged, 1 files removed, 0 files unresolved

tests/test-rebase-conflicts.t

getting f1.txt getting f1.txt
merge against 9:e31216eec445 merge against 9:e31216eec445
detach base 8:8e4e2c1a07ae detach base 8:8e4e2c1a07ae
resolving manifests resolving manifests
branchmerge: True, force: True, partial: False branchmerge: True, force: True, partial: False
ancestor: 8e4e2c1a07ae, local: 4bc80088dc6b+, remote: e31216eec445 ancestor: 8e4e2c1a07ae, local: 4bc80088dc6b+, remote: e31216eec445
f1.txt: remote is newer -> g f1.txt: remote is newer -> g
getting f1.txt getting f1.txt
f2.txt: local not present, remote unchanged -> ka
committing files: committing files:
f1.txt f1.txt
committing manifest committing manifest
committing changelog committing changelog
updating the branch cache updating the branch cache
rebased as 19c888675e13 rebased as 19c888675e13
rebase status stored rebase status stored
rebasing 10:2f2496ddf49d "merge" (tip) rebasing 10:2f2496ddf49d "merge" (tip)
future parents are 11 and 7 future parents are 11 and 7
already in destination already in destination
merge against 10:2f2496ddf49d merge against 10:2f2496ddf49d
detach base 9:e31216eec445 detach base 9:e31216eec445
resolving manifests resolving manifests
branchmerge: True, force: True, partial: False branchmerge: True, force: True, partial: False
ancestor: e31216eec445, local: 19c888675e13+, remote: 2f2496ddf49d ancestor: e31216eec445, local: 19c888675e13+, remote: 2f2496ddf49d
f1.txt: remote is newer -> g f1.txt: remote is newer -> g
getting f1.txt getting f1.txt
f2.txt: local not present, remote unchanged -> ka
committing files: committing files:
f1.txt f1.txt
committing manifest committing manifest
committing changelog committing changelog
updating the branch cache updating the branch cache
rebased as 2a7f09cac94c rebased as 2a7f09cac94c
rebase status stored rebase status stored
rebase merging completed rebase merging completed

tests/test-rename-dir-merge.t

src: 'a/b' -> dst: 'b/b' src: 'a/b' -> dst: 'b/b'
checking for directory renames checking for directory renames
discovered dir src: 'a/' -> dst: 'b/' discovered dir src: 'a/' -> dst: 'b/'
pending file src: 'a/c' -> dst: 'b/c' pending file src: 'a/c' -> dst: 'b/c'
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: f9b20c0d4c51, local: 397f8b00a740+, remote: ce36d17b18fb ancestor: f9b20c0d4c51, local: 397f8b00a740+, remote: ce36d17b18fb
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
b/a: file is rename dest and rename source did not change -> k
b/b: file is rename dest and rename source did not change -> k
a/a: local not present, remote unchanged -> ka
a/b: local not present, remote unchanged -> ka
b/c: local directory rename - get from a/c -> dg b/c: local directory rename - get from a/c -> dg
getting a/c to b/c getting a/c to b/c
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit) (branch merge, don't forget to commit)
$ echo a/* b/* $ echo a/* b/*
a/d b/a b/b b/c a/d b/a b/b b/c
$ hg st -C $ hg st -C

tests/test-rename-merge1.t

resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: af1939970a1c, local: 044f8520aeeb+, remote: 85c198ef2f6c ancestor: af1939970a1c, local: 044f8520aeeb+, remote: 85c198ef2f6c
note: possible conflict - a2 was renamed multiple times to: note: possible conflict - a2 was renamed multiple times to:
b2 b2
c2 c2
b2: remote created -> g b2: remote created -> g
getting b2 getting b2
c2: ancestor missing, remote missing -> kn
preserving a for resolve of b preserving a for resolve of b
removing a removing a
b: remote moved from a -> m (premerge) b: remote moved from a -> m (premerge)
picked tool ':merge' for b (binary False symlink False changedelete False) picked tool ':merge' for b (binary False symlink False changedelete False)
merging a and b to b merging a and b to b
my b@044f8520aeeb+ other b@85c198ef2f6c ancestor a@af1939970a1c my b@044f8520aeeb+ other b@85c198ef2f6c ancestor a@af1939970a1c
premerge successful premerge successful
1 files updated, 1 files merged, 0 files removed, 0 files unresolved 1 files updated, 1 files merged, 0 files removed, 0 files unresolved

tests/test-rename-merge2.t

b b
all copies found (* = to merge, ! = divergent, % = renamed and deleted): all copies found (* = to merge, ! = divergent, % = renamed and deleted):
on remote side: on remote side:
src: 'a' -> dst: 'b' * src: 'a' -> dst: 'b' *
checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: 924404dff337, local: e300d1c794ec+, remote: 4ce40f5aca24 ancestor: 924404dff337, local: e300d1c794ec+, remote: 4ce40f5aca24
a: remote unchanged -> k
preserving a for resolve of b preserving a for resolve of b
preserving rev for resolve of rev
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
preserving rev for resolve of rev
b: remote copied from a -> m (premerge) b: remote copied from a -> m (premerge)
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
picked tool '* ../merge' for b (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for b (binary False symlink False changedelete False) (glob)
merging a and b to b merging a and b to b
my b@e300d1c794ec+ other b@4ce40f5aca24 ancestor a@924404dff337 my b@e300d1c794ec+ other b@4ce40f5aca24 ancestor a@924404dff337
premerge successful premerge successful
rev: versions differ -> m (premerge) rev: versions differ -> m (premerge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)
b b
all copies found (* = to merge, ! = divergent, % = renamed and deleted): all copies found (* = to merge, ! = divergent, % = renamed and deleted):
on local side: on local side:
src: 'a' -> dst: 'b' src: 'a' -> dst: 'b'
checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: 924404dff337, local: 86a2aa42fc76+, remote: 97c705ade336 ancestor: 924404dff337, local: 86a2aa42fc76+, remote: 97c705ade336
preserving rev for resolve of rev b: ancestor missing, remote missing -> kn
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
preserving rev for resolve of rev
rev: versions differ -> m (premerge) rev: versions differ -> m (premerge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)
merging rev merging rev
my rev@86a2aa42fc76+ other rev@97c705ade336 ancestor rev@924404dff337 my rev@86a2aa42fc76+ other rev@97c705ade336 ancestor rev@924404dff337
rev: versions differ -> m (merge) rev: versions differ -> m (merge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)
my rev@86a2aa42fc76+ other rev@97c705ade336 ancestor rev@924404dff337 my rev@86a2aa42fc76+ other rev@97c705ade336 ancestor rev@924404dff337
launching merge tool: * ../merge *$TESTTMP/t/t/rev* * * (glob) launching merge tool: * ../merge *$TESTTMP/t/t/rev* * * (glob)
b b
all copies found (* = to merge, ! = divergent, % = renamed and deleted): all copies found (* = to merge, ! = divergent, % = renamed and deleted):
on local side: on local side:
src: 'a' -> dst: 'b' src: 'a' -> dst: 'b'
checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: 924404dff337, local: 02963e448370+, remote: 97c705ade336 ancestor: 924404dff337, local: 02963e448370+, remote: 97c705ade336
preserving rev for resolve of rev a: local not present, remote unchanged -> ka
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
b: ancestor missing, remote missing -> kn
preserving rev for resolve of rev
rev: versions differ -> m (premerge) rev: versions differ -> m (premerge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)
merging rev merging rev
my rev@02963e448370+ other rev@97c705ade336 ancestor rev@924404dff337 my rev@02963e448370+ other rev@97c705ade336 ancestor rev@924404dff337
rev: versions differ -> m (merge) rev: versions differ -> m (merge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)
my rev@02963e448370+ other rev@97c705ade336 ancestor rev@924404dff337 my rev@02963e448370+ other rev@97c705ade336 ancestor rev@924404dff337
launching merge tool: * ../merge *$TESTTMP/t/t/rev* * * (glob) launching merge tool: * ../merge *$TESTTMP/t/t/rev* * * (glob)
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: 924404dff337, local: 02963e448370+, remote: fe905ef2c33e ancestor: 924404dff337, local: 02963e448370+, remote: fe905ef2c33e
note: possible conflict - a was renamed multiple times to: note: possible conflict - a was renamed multiple times to:
b b
c c
c: remote created -> g c: remote created -> g
getting c getting c
b: ancestor missing, remote missing -> kn
preserving rev for resolve of rev preserving rev for resolve of rev
rev: versions differ -> m (premerge) rev: versions differ -> m (premerge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)
merging rev merging rev
my rev@02963e448370+ other rev@fe905ef2c33e ancestor rev@924404dff337 my rev@02963e448370+ other rev@fe905ef2c33e ancestor rev@924404dff337
rev: versions differ -> m (merge) rev: versions differ -> m (merge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)
my rev@02963e448370+ other rev@fe905ef2c33e ancestor rev@924404dff337 my rev@02963e448370+ other rev@fe905ef2c33e ancestor rev@924404dff337
-------------- --------------
all copies found (* = to merge, ! = divergent, % = renamed and deleted): all copies found (* = to merge, ! = divergent, % = renamed and deleted):
on remote side: on remote side:
src: 'a' -> dst: 'b' * src: 'a' -> dst: 'b' *
checking for directory renames checking for directory renames
resolving manifests resolving manifests
branchmerge: True, force: False, partial: False branchmerge: True, force: False, partial: False
ancestor: 924404dff337, local: 0b76e65c8289+, remote: 4ce40f5aca24 ancestor: 924404dff337, local: 0b76e65c8289+, remote: 4ce40f5aca24
a: remote unchanged -> k
preserving b for resolve of b preserving b for resolve of b
preserving rev for resolve of rev
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
preserving rev for resolve of rev
b: both renamed from a -> m (premerge) b: both renamed from a -> m (premerge)
starting 4 threads for background file closing (?) starting 4 threads for background file closing (?)
picked tool '* ../merge' for b (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for b (binary False symlink False changedelete False) (glob)
merging b merging b
my b@0b76e65c8289+ other b@4ce40f5aca24 ancestor a@924404dff337 my b@0b76e65c8289+ other b@4ce40f5aca24 ancestor a@924404dff337
premerge successful premerge successful
rev: versions differ -> m (premerge) rev: versions differ -> m (premerge)
picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob) picked tool '* ../merge' for rev (binary False symlink False changedelete False) (glob)