This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10695

cache: avoid warming the fnodetags cache after clone
ClosedPublic

Authored by marmoute on May 7 2021, 5:31 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG8be95673eb8a: cache: avoid warming the fnodetags cache after clone
Summary

That cache can quite expensive to compute on large repository as not that `hg
clone` is warming all cache, this can introduces a significant slowdown for
clone time[1]. As a stop gap measure introduce a quick fix for
that on stable, skipping the fnodetags cache post-clone.

[1] https://www.mercurial-scm.org/pipermail/mercurial/2021-April/052679.html

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.May 7 2021, 5:31 AM
baymax updated this revision to Diff 27966.May 17 2021, 4:27 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)
⚠ This patch is intended for stable ⚠

Alphare accepted this revision.May 17 2021, 5:09 AM
This revision is now accepted and ready to land.May 17 2021, 5:09 AM
marmoute added a commit: rHG8be95673eb8a: cache: avoid warming the fnodetags cache after clone.May 17 2021, 5:15 AM
Closed by commit rHG8be95673eb8a: cache: avoid warming the fnodetags cache after clone (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 27967

mercurial/hg.py

bookmarks.activate(destrepo, update) bookmarks.activate(destrepo, update)
if destlock is not None: if destlock is not None:
release(destlock) release(destlock)
# here is a tiny windows were someone could end up writing the # here is a tiny windows were someone could end up writing the
# repository before the cache are sure to be warm. This is "fine" # repository before the cache are sure to be warm. This is "fine"
# as the only "bad" outcome would be some slowness. That potential # as the only "bad" outcome would be some slowness. That potential
# slowness already affect reader. # slowness already affect reader.
with destrepo.lock(): with destrepo.lock():
destrepo.updatecaches(full=True) destrepo.updatecaches(full=b"post-clone")
finally: finally:
release(srclock, destlock) release(srclock, destlock)
if cleandir is not None: if cleandir is not None:
shutil.rmtree(cleandir, True) shutil.rmtree(cleandir, True)
if srcpeer is not None: if srcpeer is not None:
srcpeer.close() srcpeer.close()
if destpeer and destpeer.local() is None: if destpeer and destpeer.local() is None:
destpeer.close() destpeer.close()

mercurial/localrepo.py

"""warm appropriate caches """warm appropriate caches
If this function is called after a transaction closed. The transaction If this function is called after a transaction closed. The transaction
will be available in the 'tr' argument. This can be used to selectively will be available in the 'tr' argument. This can be used to selectively
update caches relevant to the changes in that transaction. update caches relevant to the changes in that transaction.
If 'full' is set, make sure all caches the function knows about have If 'full' is set, make sure all caches the function knows about have
up-to-date data. Even the ones usually loaded more lazily. up-to-date data. Even the ones usually loaded more lazily.
The `full` argument can take a special "post-clone" value. In this case
the cache warming is made after a clone and of the slower cache might
be skipped, namely the `.fnodetags` one. This argument is 5.8 specific
as we plan for a cleaner way to deal with this for 5.9.
""" """
if tr is not None and tr.hookargs.get(b'source') == b'strip': if tr is not None and tr.hookargs.get(b'source') == b'strip':
# During strip, many caches are invalid but # During strip, many caches are invalid but
# later call to `destroyed` will refresh them. # later call to `destroyed` will refresh them.
return return
if tr is None or tr.changes[b'origrepolen'] < len(self): if tr is None or tr.changes[b'origrepolen'] < len(self):
# accessing the 'served' branchmap should refresh all the others, # accessing the 'served' branchmap should refresh all the others,
for r in unfi.changelog: for r in unfi.changelog:
rbc.branchinfo(r) rbc.branchinfo(r)
rbc.write() rbc.write()
# ensure the working copy parents are in the manifestfulltextcache # ensure the working copy parents are in the manifestfulltextcache
for ctx in self[b'.'].parents(): for ctx in self[b'.'].parents():
ctx.manifest() # accessing the manifest is enough ctx.manifest() # accessing the manifest is enough
if not full == b"post-clone":
# accessing fnode cache warms the cache # accessing fnode cache warms the cache
tagsmod.fnoderevs(self.ui, unfi, unfi.changelog.revs()) tagsmod.fnoderevs(self.ui, unfi, unfi.changelog.revs())
# accessing tags warm the cache # accessing tags warm the cache
self.tags() self.tags()
self.filtered(b'served').tags() self.filtered(b'served').tags()
# The `full` arg is documented as updating even the lazily-loaded # The `full` arg is documented as updating even the lazily-loaded
# caches immediately, so we're forcing a write to cause these caches # caches immediately, so we're forcing a write to cause these caches
# to be warmed up even if they haven't explicitly been requested # to be warmed up even if they haven't explicitly been requested
# yet (if they've never been used by hg, they won't ever have been # yet (if they've never been used by hg, they won't ever have been

tests/test-clone-uncompressed.t

$ ls -1 clone1/.hg/cache $ ls -1 clone1/.hg/cache
branch2-base branch2-base
branch2-immutable branch2-immutable
branch2-served branch2-served
branch2-served.hidden branch2-served.hidden
branch2-visible branch2-visible
branch2-visible-hidden branch2-visible-hidden
hgtagsfnodes1
rbc-names-v1 rbc-names-v1
rbc-revs-v1 rbc-revs-v1
tags2 tags2
tags2-served tags2-served
$ cat server/errors.txt $ cat server/errors.txt
#endif #endif
getbundle requests with stream=1 are uncompressed getbundle requests with stream=1 are uncompressed

tests/test-clone.t

$ ls .hg/cache $ ls .hg/cache
branch2-base branch2-base
branch2-immutable branch2-immutable
branch2-served branch2-served
branch2-served.hidden branch2-served.hidden
branch2-visible branch2-visible
branch2-visible-hidden branch2-visible-hidden
hgtagsfnodes1
rbc-names-v1 rbc-names-v1
rbc-revs-v1 rbc-revs-v1
tags2 tags2
tags2-served tags2-served
$ cat a $ cat a
a a
$ hg verify $ hg verify
$ ls .hg/cache $ ls .hg/cache
branch2-base branch2-base
branch2-immutable branch2-immutable
branch2-served branch2-served
branch2-served.hidden branch2-served.hidden
branch2-visible branch2-visible
branch2-visible-hidden branch2-visible-hidden
hgtagsfnodes1
rbc-names-v1 rbc-names-v1
rbc-revs-v1 rbc-revs-v1
tags2 tags2
tags2-served tags2-served
$ cat a 2>/dev/null || echo "a not present" $ cat a 2>/dev/null || echo "a not present"
a not present a not present
$ hg verify $ hg verify

tests/test-hardlinks.t

2 r4/.hg/00changelog.i 2 r4/.hg/00changelog.i
2 r4/.hg/branch 2 r4/.hg/branch
2 r4/.hg/cache/branch2-base 2 r4/.hg/cache/branch2-base
2 r4/.hg/cache/branch2-immutable 2 r4/.hg/cache/branch2-immutable
2 r4/.hg/cache/branch2-served 2 r4/.hg/cache/branch2-served
2 r4/.hg/cache/branch2-served.hidden 2 r4/.hg/cache/branch2-served.hidden
2 r4/.hg/cache/branch2-visible 2 r4/.hg/cache/branch2-visible
2 r4/.hg/cache/branch2-visible-hidden 2 r4/.hg/cache/branch2-visible-hidden
2 r4/.hg/cache/hgtagsfnodes1
2 r4/.hg/cache/rbc-names-v1 2 r4/.hg/cache/rbc-names-v1
2 r4/.hg/cache/rbc-revs-v1 2 r4/.hg/cache/rbc-revs-v1
2 r4/.hg/cache/tags2 2 r4/.hg/cache/tags2
2 r4/.hg/cache/tags2-served 2 r4/.hg/cache/tags2-served
2 r4/.hg/dirstate 2 r4/.hg/dirstate
2 r4/.hg/fsmonitor.state (fsmonitor !) 2 r4/.hg/fsmonitor.state (fsmonitor !)
2 r4/.hg/hgrc 2 r4/.hg/hgrc
2 r4/.hg/last-message.txt 2 r4/.hg/last-message.txt
2 r4/.hg/00changelog.i 2 r4/.hg/00changelog.i
1 r4/.hg/branch 1 r4/.hg/branch
2 r4/.hg/cache/branch2-base 2 r4/.hg/cache/branch2-base
2 r4/.hg/cache/branch2-immutable 2 r4/.hg/cache/branch2-immutable
2 r4/.hg/cache/branch2-served 2 r4/.hg/cache/branch2-served
2 r4/.hg/cache/branch2-served.hidden 2 r4/.hg/cache/branch2-served.hidden
2 r4/.hg/cache/branch2-visible 2 r4/.hg/cache/branch2-visible
2 r4/.hg/cache/branch2-visible-hidden 2 r4/.hg/cache/branch2-visible-hidden
2 r4/.hg/cache/hgtagsfnodes1
2 r4/.hg/cache/rbc-names-v1 2 r4/.hg/cache/rbc-names-v1
2 r4/.hg/cache/rbc-revs-v1 2 r4/.hg/cache/rbc-revs-v1
2 r4/.hg/cache/tags2 2 r4/.hg/cache/tags2
2 r4/.hg/cache/tags2-served 2 r4/.hg/cache/tags2-served
1 r4/.hg/dirstate 1 r4/.hg/dirstate
1 r4/.hg/fsmonitor.state (fsmonitor !) 1 r4/.hg/fsmonitor.state (fsmonitor !)
2 r4/.hg/hgrc 2 r4/.hg/hgrc
2 r4/.hg/last-message.txt 2 r4/.hg/last-message.txt

tests/test-share.t

1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ ls -1 ../repo2-clone/.hg/cache $ ls -1 ../repo2-clone/.hg/cache
branch2-base branch2-base
branch2-immutable branch2-immutable
branch2-served branch2-served
branch2-served.hidden branch2-served.hidden
branch2-visible branch2-visible
branch2-visible-hidden branch2-visible-hidden
hgtagsfnodes1
rbc-names-v1 rbc-names-v1
rbc-revs-v1 rbc-revs-v1
tags2 tags2
tags2-served tags2-served
tags2-visible tags2-visible
Some sed versions appends newline, some don't, and some just fails Some sed versions appends newline, some don't, and some just fails

tests/test-ssh.t

default 0:1160648e36ce default 0:1160648e36ce
$ cd $TESTTMP $ cd $TESTTMP
clone bookmarks via stream clone bookmarks via stream
$ hg -R local-stream book mybook $ hg -R local-stream book mybook
$ hg clone -e "\"$PYTHON\" \"$TESTDIR/dummyssh\"" --stream ssh://user@dummy/local-stream stream2 $ hg clone -e "\"$PYTHON\" \"$TESTDIR/dummyssh\"" --stream ssh://user@dummy/local-stream stream2
streaming all changes streaming all changes
16 files to transfer, * of data (glob) 15 files to transfer, * of data (glob)
transferred * in * seconds (*) (glob) transferred * in * seconds (*) (glob)
updating to branch default updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd stream2 $ cd stream2
$ hg book $ hg book
mybook 0:1160648e36ce mybook 0:1160648e36ce
$ cd $TESTTMP $ cd $TESTTMP
$ rm -rf local-stream stream2 $ rm -rf local-stream stream2

tests/test-tags.t

rbc-revs-v1 rbc-revs-v1
tags2 tags2
tags2-served tags2-served
Cache should contain the head only, even though other nodes have tags data Cache should contain the head only, even though other nodes have tags data
$ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1 $ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1
tagsclient/.hg/cache/hgtagsfnodes1: size=96 tagsclient/.hg/cache/hgtagsfnodes1: size=96
0000: 96 ee 1d 73 00 00 00 00 00 00 00 00 00 00 00 00 |...s............| 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: 00 00 00 00 00 00 00 00 c4 da b0 c2 94 65 e1 c6 |.............e..| 0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0020: 0d f7 f0 dd 32 04 ea 57 78 c8 97 97 79 fc d5 95 |....2..Wx...y...| 0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0030: f6 3c c8 fe 94 65 e1 c6 0d f7 f0 dd 32 04 ea 57 |.<...e......2..W| 0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0040: 78 c8 97 97 79 fc d5 95 40 f0 35 8c 19 e0 a7 d3 |x...y...@.5.....| 0040: ff ff ff ff ff ff ff ff 40 f0 35 8c 19 e0 a7 d3 |........@.5.....|
0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.| 0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.|
Running hg tags should produce tags2* file and not change cache Running hg tags should produce tags2* file and not change cache
$ hg -R tagsclient tags $ hg -R tagsclient tags
tip 3:40f0358cb314 tip 3:40f0358cb314
0.2 2:f63cc8fe54e4 0.2 2:f63cc8fe54e4
0.1 0:96ee1d7354c4 0.1 0:96ee1d7354c4
rbc-names-v1 rbc-names-v1
rbc-revs-v1 rbc-revs-v1
tags2 tags2
tags2-served tags2-served
tags2-visible tags2-visible
$ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1 $ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1
tagsclient/.hg/cache/hgtagsfnodes1: size=96 tagsclient/.hg/cache/hgtagsfnodes1: size=96
0000: 96 ee 1d 73 00 00 00 00 00 00 00 00 00 00 00 00 |...s............| 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: 00 00 00 00 00 00 00 00 c4 da b0 c2 94 65 e1 c6 |.............e..| 0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0020: 0d f7 f0 dd 32 04 ea 57 78 c8 97 97 79 fc d5 95 |....2..Wx...y...| 0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0030: f6 3c c8 fe 94 65 e1 c6 0d f7 f0 dd 32 04 ea 57 |.<...e......2..W| 0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0040: 78 c8 97 97 79 fc d5 95 40 f0 35 8c 19 e0 a7 d3 |x...y...@.5.....| 0040: ff ff ff ff ff ff ff ff 40 f0 35 8c 19 e0 a7 d3 |........@.5.....|
0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.| 0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.|
Check that the bundle includes cache data Check that the bundle includes cache data
$ hg -R tagsclient bundle --all ./test-cache-in-bundle-all-rev.hg $ hg -R tagsclient bundle --all ./test-cache-in-bundle-all-rev.hg
4 changesets found 4 changesets found
$ hg debugbundle ./test-cache-in-bundle-all-rev.hg $ hg debugbundle ./test-cache-in-bundle-all-rev.hg
Stream params: {Compression: BZ} Stream params: {Compression: BZ}

tests/test-treemanifest.t

crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 4 changesets with 18 changes to 8 files checked 4 changesets with 18 changes to 8 files
Stream clone with basicstore Stream clone with basicstore
$ hg clone --config experimental.changegroup3=True --stream -U \ $ hg clone --config experimental.changegroup3=True --stream -U \
> http://localhost:$HGPORT1 stream-clone-basicstore > http://localhost:$HGPORT1 stream-clone-basicstore
streaming all changes streaming all changes
29 files to transfer, * of data (glob) 28 files to transfer, * of data (glob)
transferred * in * seconds (*) (glob) transferred * in * seconds (*) (glob)
$ hg -R stream-clone-basicstore verify $ hg -R stream-clone-basicstore verify
checking changesets checking changesets
checking manifests checking manifests
checking directory manifests checking directory manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 4 changesets with 18 changes to 8 files checked 4 changesets with 18 changes to 8 files
Stream clone with encodedstore Stream clone with encodedstore
$ hg clone --config experimental.changegroup3=True --stream -U \ $ hg clone --config experimental.changegroup3=True --stream -U \
> http://localhost:$HGPORT2 stream-clone-encodedstore > http://localhost:$HGPORT2 stream-clone-encodedstore
streaming all changes streaming all changes
29 files to transfer, * of data (glob) 28 files to transfer, * of data (glob)
transferred * in * seconds (*) (glob) transferred * in * seconds (*) (glob)
$ hg -R stream-clone-encodedstore verify $ hg -R stream-clone-encodedstore verify
checking changesets checking changesets
checking manifests checking manifests
checking directory manifests checking directory manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 4 changesets with 18 changes to 8 files checked 4 changesets with 18 changes to 8 files