This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5196

changegroup: restore default node ordering (issue6001)
ClosedPublic

Authored by lothiraldan on Oct 30 2018, 12:28 PM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Commits
rHG634b45317459: changegroup: restore default node ordering (issue6001)
Summary

Changeset db5501d9 changed the default node ordering from "storage" to
"linearize".

While the new API is more explicit and cleaner, the "linearize" order is
problematic on certain repositories like netbeans where it makes bundling
slower the more nodes we bundle.

Pushing and pulling 100 changesets was ~20% slower and pushing and pulling
1000 changesets was ~600% slower.

A very quick analysis of profile traces showed that the pull operation was
taking more time creating the delta.

Putting back the old default order seems to be the safe option. With more time
during the next cycle, we can understand better the impact of sorting with the
DAG order by default, the source of the regression and how to mitigate it.

/!\ We are still waiting for the full performance impact but with this patch,
bundling and pulling locally (not on the performance workstation) 1000
changesets on the netbeans repository is as fast as before the regression.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lothiraldan created this revision.Oct 30 2018, 12:28 PM
indygreg accepted this revision.Oct 31 2018, 3:06 PM
indygreg added a subscriber: indygreg.

db5501d93bcf6ada3426a45e901094fd877e370f changed a test. I'm surprised this patch doesn't include that revert as well. I'll verify the test as part of landing and amend as necessary.

This revision is now accepted and ready to land.Oct 31 2018, 3:06 PM
Closed by commit rHG634b45317459: changegroup: restore default node ordering (issue6001) (authored by lothiraldan). · Explain WhyOct 31 2018, 3:09 PM
This revision was automatically updated to reflect the committed changes.
lothiraldan added a comment.Nov 1 2018, 7:38 AM

With the fix, we are back to pre-regression performance on the test suite.

Can we update the last paragraph of the changeset description to replace it with the previous phrase?

There is two tests that are actually slower between d9b3cc3d (the parent of the regression) and the fix, which are:

+         206±2ms         310±40ms     1.51  basic_commands.PushPullTimeSuite.time_push('mozilla-central-2018-08-01', 'local', 'same', 'default')
+         175±1ms         226±30ms     1.30  basic_commands.PushPullTimeSuite.time_push('mozilla-central-2018-08-01', 'local', 'same', None)

There are push operations between the same repository, no-op push. As they are no-op push, I think there is no bundle generation so the fix shouldn't impact the performance. We may have another slowdown.

Revision Contents
Changeset List

Diff 12354

mercurial/utils/storageutil.py

See ``ifiledata.emitrevisions()`` interface documentation. See ``ifiledata.emitrevisions()`` interface documentation.
""" """
fnode = store.node fnode = store.node
frev = store.rev frev = store.rev
if nodesorder == 'nodes': if nodesorder == 'nodes':
revs = [frev(n) for n in nodes] revs = [frev(n) for n in nodes]
elif nodesorder == 'storage': elif nodesorder == 'linear':
revs = sorted(frev(n) for n in nodes)
else:
revs = set(frev(n) for n in nodes) revs = set(frev(n) for n in nodes)
revs = dagop.linearize(revs, store.parentrevs) revs = dagop.linearize(revs, store.parentrevs)
else: # storage and default
revs = sorted(frev(n) for n in nodes)
prevrev = None prevrev = None
if deltaprevious or assumehaveparentrevisions: if deltaprevious or assumehaveparentrevisions:
prevrev = store.parentrevs(revs[0])[0] prevrev = store.parentrevs(revs[0])[0]
# Set of revs available to delta against. # Set of revs available to delta against.
available = set() available = set()

tests/test-generaldelta.t

adding manifests adding manifests
adding file changes adding file changes
added 55 changesets with 53 changes to 53 files (+2 heads) added 55 changesets with 53 changes to 53 files (+2 heads)
new changesets 61246295ee1e:c930ac4a5b32 new changesets 61246295ee1e:c930ac4a5b32
updating to branch default updating to branch default
14 files updated, 0 files merged, 0 files removed, 0 files unresolved 14 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg -R relax-chain debugdeltachain -m $ hg -R relax-chain debugdeltachain -m
rev chain# chainlen prev delta size rawsize chainsize ratio lindist extradist extraratio rev chain# chainlen prev delta size rawsize chainsize ratio lindist extradist extraratio
0 1 1 -1 base 47 46 47 1.02174 47 0 0.00000 0 1 1 -1 base 46 45 46 1.02222 46 0 0.00000
1 1 2 0 p1 58 92 105 1.14130 105 0 0.00000 1 1 2 0 p1 57 90 103 1.14444 103 0 0.00000
2 1 3 1 p1 58 138 163 1.18116 163 0 0.00000 2 1 3 1 p1 57 135 160 1.18519 160 0 0.00000
3 1 4 2 p1 58 184 221 1.20109 221 0 0.00000 3 1 4 2 p1 57 180 217 1.20556 217 0 0.00000
4 1 5 3 p1 58 230 279 1.21304 279 0 0.00000 4 1 5 3 p1 57 225 274 1.21778 274 0 0.00000
5 1 6 4 p1 58 276 337 1.22101 337 0 0.00000 5 1 6 4 p1 57 270 331 1.22593 331 0 0.00000
6 1 7 5 p1 58 322 395 1.22671 395 0 0.00000 6 2 1 -1 base 46 45 46 1.02222 46 0 0.00000
7 1 8 6 p1 58 368 453 1.23098 453 0 0.00000 7 2 2 6 p1 57 90 103 1.14444 103 0 0.00000
8 1 9 7 p1 58 414 511 1.23430 511 0 0.00000 8 2 3 7 p1 57 135 160 1.18519 160 0 0.00000
9 1 10 8 p1 58 460 569 1.23696 569 0 0.00000 9 2 4 8 p1 57 180 217 1.20556 217 0 0.00000
10 1 11 9 p1 58 506 627 1.23913 627 0 0.00000 10 2 5 9 p1 58 226 275 1.21681 275 0 0.00000
11 1 12 10 p1 58 552 685 1.24094 685 0 0.00000 11 2 6 10 p1 58 272 333 1.22426 333 0 0.00000
12 1 13 11 p1 58 598 743 1.24247 743 0 0.00000 12 2 7 11 p1 58 318 391 1.22956 391 0 0.00000
13 1 14 12 p1 58 644 801 1.24379 801 0 0.00000 13 2 8 12 p1 58 364 449 1.23352 449 0 0.00000
14 1 15 13 p1 58 690 859 1.24493 859 0 0.00000 14 2 9 13 p1 58 410 507 1.23659 507 0 0.00000
15 1 16 14 p1 58 736 917 1.24592 917 0 0.00000 15 2 10 14 p1 58 456 565 1.23904 565 0 0.00000
16 1 17 15 p1 58 782 975 1.24680 975 0 0.00000 16 2 11 15 p1 58 502 623 1.24104 623 0 0.00000
17 1 18 16 p1 58 828 1033 1.24758 1033 0 0.00000 17 2 12 16 p1 58 548 681 1.24270 681 0 0.00000
18 1 19 17 p1 58 874 1091 1.24828 1091 0 0.00000 18 3 1 -1 base 47 46 47 1.02174 47 0 0.00000
19 1 20 18 p1 58 920 1149 1.24891 1149 0 0.00000 19 3 2 18 p1 58 92 105 1.14130 105 0 0.00000
20 1 21 19 p1 58 966 1207 1.24948 1207 0 0.00000 20 3 3 19 p1 58 138 163 1.18116 163 0 0.00000
21 1 22 20 p1 58 1012 1265 1.25000 1265 0 0.00000 21 3 4 20 p1 58 184 221 1.20109 221 0 0.00000
22 1 23 21 p1 58 1058 1323 1.25047 1323 0 0.00000 22 3 5 21 p1 58 230 279 1.21304 279 0 0.00000
23 1 24 22 p1 58 1104 1381 1.25091 1381 0 0.00000 23 3 6 22 p1 58 276 337 1.22101 337 0 0.00000
24 1 25 23 p1 58 1150 1439 1.25130 1439 0 0.00000 24 3 7 23 p1 58 322 395 1.22671 395 0 0.00000
25 1 26 24 p1 58 1196 1497 1.25167 1497 0 0.00000 25 3 8 24 p1 58 368 453 1.23098 453 0 0.00000
26 1 27 25 p1 58 1242 1555 1.25201 1555 0 0.00000 26 3 9 25 p1 58 414 511 1.23430 511 0 0.00000
27 1 28 26 p1 58 1288 1613 1.25233 1613 0 0.00000 27 3 10 26 p1 58 460 569 1.23696 569 0 0.00000
28 1 29 27 p1 58 1334 1671 1.25262 1671 0 0.00000 28 3 11 27 p1 58 506 627 1.23913 627 0 0.00000
29 1 30 28 p1 58 1380 1729 1.25290 1729 0 0.00000 29 3 12 28 p1 58 552 685 1.24094 685 0 0.00000
30 1 31 29 p1 58 1426 1787 1.25316 1787 0 0.00000 30 3 13 29 p1 58 598 743 1.24247 743 0 0.00000
31 2 1 -1 base 46 45 46 1.02222 46 0 0.00000 31 3 14 30 p1 58 644 801 1.24379 801 0 0.00000
32 2 2 31 p1 57 90 103 1.14444 103 0 0.00000 32 3 15 31 p1 58 690 859 1.24493 859 0 0.00000
33 2 3 32 p1 57 135 160 1.18519 160 0 0.00000 33 3 16 32 p1 58 736 917 1.24592 917 0 0.00000
34 2 4 33 p1 57 180 217 1.20556 217 0 0.00000 34 3 17 33 p1 58 782 975 1.24680 975 0 0.00000
35 2 5 34 p1 57 225 274 1.21778 274 0 0.00000 35 3 18 34 p1 58 828 1033 1.24758 1033 0 0.00000
36 2 6 35 p1 57 270 331 1.22593 331 0 0.00000 36 3 19 35 p1 58 874 1091 1.24828 1091 0 0.00000
37 2 7 36 p1 58 316 389 1.23101 389 0 0.00000 37 3 20 36 p1 58 920 1149 1.24891 1149 0 0.00000
38 2 8 37 p1 58 362 447 1.23481 447 0 0.00000 38 3 21 37 p1 58 966 1207 1.24948 1207 0 0.00000
39 3 1 -1 base 46 45 46 1.02222 46 0 0.00000 39 3 22 38 p1 58 1012 1265 1.25000 1265 0 0.00000
40 3 2 39 p1 57 90 103 1.14444 103 0 0.00000 40 3 23 39 p1 58 1058 1323 1.25047 1323 0 0.00000
41 3 3 40 p1 57 135 160 1.18519 160 0 0.00000 41 3 24 40 p1 58 1104 1381 1.25091 1381 0 0.00000
42 3 4 41 p1 57 180 217 1.20556 217 0 0.00000 42 3 25 41 p1 58 1150 1439 1.25130 1439 0 0.00000
43 3 5 42 p1 58 226 275 1.21681 275 0 0.00000 43 3 26 42 p1 58 1196 1497 1.25167 1497 0 0.00000
44 3 6 43 p1 58 272 333 1.22426 333 0 0.00000 44 3 27 43 p1 58 1242 1555 1.25201 1555 0 0.00000
45 3 7 44 p1 58 318 391 1.22956 391 0 0.00000 45 3 28 44 p1 58 1288 1613 1.25233 1613 0 0.00000
46 3 8 45 p1 58 364 449 1.23352 449 0 0.00000 46 3 29 45 p1 58 1334 1671 1.25262 1671 0 0.00000
47 3 9 46 p1 58 410 507 1.23659 507 0 0.00000 47 3 30 46 p1 58 1380 1729 1.25290 1729 0 0.00000
48 3 10 47 p1 58 456 565 1.23904 565 0 0.00000 48 3 31 47 p1 58 1426 1787 1.25316 1787 0 0.00000
49 3 11 48 p1 58 502 623 1.24104 623 0 0.00000 49 4 1 -1 base 197 316 197 0.62342 197 0 0.00000
50 3 12 49 p1 58 548 681 1.24270 681 0 0.00000 50 4 2 49 p1 58 362 255 0.70442 255 0 0.00000
51 3 13 50 p1 58 594 739 1.24411 739 0 0.00000 51 2 13 17 p1 58 594 739 1.24411 2781 2042 2.76319
52 3 14 51 p1 58 640 797 1.24531 797 0 0.00000 52 5 1 -1 base 369 640 369 0.57656 369 0 0.00000
53 4 1 -1 base 0 0 0 0.00000 0 0 0.00000 53 6 1 -1 base 0 0 0 0.00000 0 0 0.00000
54 5 1 -1 base 369 640 369 0.57656 369 0 0.00000 54 7 1 -1 base 369 640 369 0.57656 369 0 0.00000
$ hg clone --pull source-repo --config experimental.maxdeltachainspan=0 noconst-chain --config format.generaldelta=yes $ hg clone --pull source-repo --config experimental.maxdeltachainspan=0 noconst-chain --config format.generaldelta=yes
requesting all changes requesting all changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 55 changesets with 53 changes to 53 files (+2 heads) added 55 changesets with 53 changes to 53 files (+2 heads)
new changesets 61246295ee1e:c930ac4a5b32 new changesets 61246295ee1e:c930ac4a5b32
updating to branch default updating to branch default
14 files updated, 0 files merged, 0 files removed, 0 files unresolved 14 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg -R noconst-chain debugdeltachain -m $ hg -R noconst-chain debugdeltachain -m
rev chain# chainlen prev delta size rawsize chainsize ratio lindist extradist extraratio rev chain# chainlen prev delta size rawsize chainsize ratio lindist extradist extraratio
0 1 1 -1 base 47 46 47 1.02174 47 0 0.00000 0 1 1 -1 base 46 45 46 1.02222 46 0 0.00000
1 1 2 0 p1 58 92 105 1.14130 105 0 0.00000 1 1 2 0 p1 57 90 103 1.14444 103 0 0.00000
2 1 3 1 p1 58 138 163 1.18116 163 0 0.00000 2 1 3 1 p1 57 135 160 1.18519 160 0 0.00000
3 1 4 2 p1 58 184 221 1.20109 221 0 0.00000 3 1 4 2 p1 57 180 217 1.20556 217 0 0.00000
4 1 5 3 p1 58 230 279 1.21304 279 0 0.00000 4 1 5 3 p1 57 225 274 1.21778 274 0 0.00000
5 1 6 4 p1 58 276 337 1.22101 337 0 0.00000 5 1 6 4 p1 57 270 331 1.22593 331 0 0.00000
6 1 7 5 p1 58 322 395 1.22671 395 0 0.00000 6 2 1 -1 base 46 45 46 1.02222 46 0 0.00000
7 1 8 6 p1 58 368 453 1.23098 453 0 0.00000 7 2 2 6 p1 57 90 103 1.14444 103 0 0.00000
8 1 9 7 p1 58 414 511 1.23430 511 0 0.00000 8 2 3 7 p1 57 135 160 1.18519 160 0 0.00000
9 1 10 8 p1 58 460 569 1.23696 569 0 0.00000 9 2 4 8 p1 57 180 217 1.20556 217 0 0.00000
10 1 11 9 p1 58 506 627 1.23913 627 0 0.00000 10 2 5 9 p1 58 226 275 1.21681 275 0 0.00000
11 1 12 10 p1 58 552 685 1.24094 685 0 0.00000 11 2 6 10 p1 58 272 333 1.22426 333 0 0.00000
12 1 13 11 p1 58 598 743 1.24247 743 0 0.00000 12 2 7 11 p1 58 318 391 1.22956 391 0 0.00000
13 1 14 12 p1 58 644 801 1.24379 801 0 0.00000 13 2 8 12 p1 58 364 449 1.23352 449 0 0.00000
14 1 15 13 p1 58 690 859 1.24493 859 0 0.00000 14 2 9 13 p1 58 410 507 1.23659 507 0 0.00000
15 1 16 14 p1 58 736 917 1.24592 917 0 0.00000 15 2 10 14 p1 58 456 565 1.23904 565 0 0.00000
16 1 17 15 p1 58 782 975 1.24680 975 0 0.00000 16 2 11 15 p1 58 502 623 1.24104 623 0 0.00000
17 1 18 16 p1 58 828 1033 1.24758 1033 0 0.00000 17 2 12 16 p1 58 548 681 1.24270 681 0 0.00000
18 1 19 17 p1 58 874 1091 1.24828 1091 0 0.00000 18 3 1 -1 base 47 46 47 1.02174 47 0 0.00000
19 1 20 18 p1 58 920 1149 1.24891 1149 0 0.00000 19 3 2 18 p1 58 92 105 1.14130 105 0 0.00000
20 1 21 19 p1 58 966 1207 1.24948 1207 0 0.00000 20 3 3 19 p1 58 138 163 1.18116 163 0 0.00000
21 1 22 20 p1 58 1012 1265 1.25000 1265 0 0.00000 21 3 4 20 p1 58 184 221 1.20109 221 0 0.00000
22 1 23 21 p1 58 1058 1323 1.25047 1323 0 0.00000 22 3 5 21 p1 58 230 279 1.21304 279 0 0.00000
23 1 24 22 p1 58 1104 1381 1.25091 1381 0 0.00000 23 3 6 22 p1 58 276 337 1.22101 337 0 0.00000
24 1 25 23 p1 58 1150 1439 1.25130 1439 0 0.00000 24 3 7 23 p1 58 322 395 1.22671 395 0 0.00000
25 1 26 24 p1 58 1196 1497 1.25167 1497 0 0.00000 25 3 8 24 p1 58 368 453 1.23098 453 0 0.00000
26 1 27 25 p1 58 1242 1555 1.25201 1555 0 0.00000 26 3 9 25 p1 58 414 511 1.23430 511 0 0.00000
27 1 28 26 p1 58 1288 1613 1.25233 1613 0 0.00000 27 3 10 26 p1 58 460 569 1.23696 569 0 0.00000
28 1 29 27 p1 58 1334 1671 1.25262 1671 0 0.00000 28 3 11 27 p1 58 506 627 1.23913 627 0 0.00000
29 1 30 28 p1 58 1380 1729 1.25290 1729 0 0.00000 29 3 12 28 p1 58 552 685 1.24094 685 0 0.00000
30 1 31 29 p1 58 1426 1787 1.25316 1787 0 0.00000 30 3 13 29 p1 58 598 743 1.24247 743 0 0.00000
31 2 1 -1 base 46 45 46 1.02222 46 0 0.00000 31 3 14 30 p1 58 644 801 1.24379 801 0 0.00000
32 2 2 31 p1 57 90 103 1.14444 103 0 0.00000 32 3 15 31 p1 58 690 859 1.24493 859 0 0.00000
33 2 3 32 p1 57 135 160 1.18519 160 0 0.00000 33 3 16 32 p1 58 736 917 1.24592 917 0 0.00000
34 2 4 33 p1 57 180 217 1.20556 217 0 0.00000 34 3 17 33 p1 58 782 975 1.24680 975 0 0.00000
35 2 5 34 p1 57 225 274 1.21778 274 0 0.00000 35 3 18 34 p1 58 828 1033 1.24758 1033 0 0.00000
36 2 6 35 p1 57 270 331 1.22593 331 0 0.00000 36 3 19 35 p1 58 874 1091 1.24828 1091 0 0.00000
37 2 7 36 p1 58 316 389 1.23101 389 0 0.00000 37 3 20 36 p1 58 920 1149 1.24891 1149 0 0.00000
38 2 8 37 p1 58 362 447 1.23481 447 0 0.00000 38 3 21 37 p1 58 966 1207 1.24948 1207 0 0.00000
39 3 1 -1 base 46 45 46 1.02222 46 0 0.00000 39 3 22 38 p1 58 1012 1265 1.25000 1265 0 0.00000
40 3 2 39 p1 57 90 103 1.14444 103 0 0.00000 40 3 23 39 p1 58 1058 1323 1.25047 1323 0 0.00000
41 3 3 40 p1 57 135 160 1.18519 160 0 0.00000 41 3 24 40 p1 58 1104 1381 1.25091 1381 0 0.00000
42 3 4 41 p1 57 180 217 1.20556 217 0 0.00000 42 3 25 41 p1 58 1150 1439 1.25130 1439 0 0.00000
43 3 5 42 p1 58 226 275 1.21681 275 0 0.00000 43 3 26 42 p1 58 1196 1497 1.25167 1497 0 0.00000
44 3 6 43 p1 58 272 333 1.22426 333 0 0.00000 44 3 27 43 p1 58 1242 1555 1.25201 1555 0 0.00000
45 3 7 44 p1 58 318 391 1.22956 391 0 0.00000 45 3 28 44 p1 58 1288 1613 1.25233 1613 0 0.00000
46 3 8 45 p1 58 364 449 1.23352 449 0 0.00000 46 3 29 45 p1 58 1334 1671 1.25262 1671 0 0.00000
47 3 9 46 p1 58 410 507 1.23659 507 0 0.00000 47 3 30 46 p1 58 1380 1729 1.25290 1729 0 0.00000
48 3 10 47 p1 58 456 565 1.23904 565 0 0.00000 48 3 31 47 p1 58 1426 1787 1.25316 1787 0 0.00000
49 3 11 48 p1 58 502 623 1.24104 623 0 0.00000 49 1 7 5 p1 58 316 389 1.23101 2857 2468 6.34447
50 3 12 49 p1 58 548 681 1.24270 681 0 0.00000 50 1 8 49 p1 58 362 447 1.23481 2915 2468 5.52125
51 3 13 50 p1 58 594 739 1.24411 739 0 0.00000 51 2 13 17 p1 58 594 739 1.24411 2642 1903 2.57510
52 3 14 51 p1 58 640 797 1.24531 797 0 0.00000 52 2 14 51 p1 58 640 797 1.24531 2700 1903 2.38770
53 4 1 -1 base 0 0 0 0.00000 0 0 0.00000 53 4 1 -1 base 0 0 0 0.00000 0 0 0.00000
54 5 1 -1 base 369 640 369 0.57656 369 0 0.00000 54 5 1 -1 base 369 640 369 0.57656 369 0 0.00000