This is an archive of the discontinued Mercurial Phabricator instance.

branchheads: store wdir-dependent caches in wcache (issue6181)
AbandonedPublic

Authored by spectral on Aug 2 2019, 9:26 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

Previously, all branch2-* caches were stored in .hg/cache, which is shared
across repos when using hg share. This can cause cache thrashing when the
repos are accessed concurrently, since some of the caches depend on wdir state
(which is NOT shared across the repos).

There's already a cache directory for caches that depend on wdir state, wcache,
so let's just put the caches there.

This change does not clean up any existing caches in .hg/cache that are now
moved to the new location, and it does not re-use the caches from that location
when constructing them in the wcache directory. Upgrades to versions of
Mercurial that have this commit will need to regenerate their branchheads once, and
this may be an expensive operation.

The format of these cache files is not changing - forwards/backwards
compatibility remains; if a newer repo (with these in wcache) is accessed by an
older hg, the older hg will either see the files that remain in .hg/cache, and
update them there, or will not see the files at all and will fall back to a
slower path (and then likely cache them in .hg/cache). If it's then accessed by
a newer hg, the work done by the older hg is ignored, and the wcache copies are
brought up to date.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

spectral created this revision.Aug 2 2019, 9:26 PM

The change looks good to me, However we probably want to introduce a new filter level 'wdir-independent-visible' to ensure we a good branchcache cache in .hg/cache that most share can use and that will be kept up to date. This also means we need to make sure it is warmed after transaction.

spectral updated this revision to Diff 16130.Aug 5 2019, 9:07 PM

The change looks good to me, However we probably want to introduce a new filter level 'wdir-independent-visible' to ensure we a good branchcache cache in .hg/cache that most share can use and that will be kept up to date. This also means we need to make sure it is warmed after transaction.

I agree such a thing would be useful, and pretty easy to add (just needs to not remove items considered "pinned" during computehidden, but I'm a little concerned about the proliferation of these things. I think ideally we'd end up with:

visible-hidden > visible > wdir-independent-visible > wdir-independent-served > wdir-independent-served.hidden > immutable > base
and
served > wdir-independent-served > wdir-independent-served.hidden > immutable > base
and
served.hidden > wdir-independent-served.hidden > immutable > base

I think?

(Not sure if we need a wdir-independent-visible-hidden, I still don't really understand visible-hidden, I just know that updating it actually breaks a Lot of stuff, so I avoided doing so in the first commit in the stack. :))

The change looks good to me, However we probably want to introduce a new filter level 'wdir-independent-visible' to ensure we a good branchcache cache in .hg/cache that most share can use and that will be kept up to date. This also means we need to make sure it is warmed after transaction.

I agree such a thing would be useful, and pretty easy to add (just needs to not remove items considered "pinned" during computehidden, but I'm a little concerned about the proliferation of these things. I think ideally we'd end up with:
visible-hidden > visible > wdir-independent-visible > wdir-independent-served > wdir-independent-served.hidden > immutable > base
and
served > wdir-independent-served > wdir-independent-served.hidden > immutable > base
and
served.hidden > wdir-independent-served.hidden > immutable > base
I think?

Yeah, this makes a lot of them, but the alternative (having non-up-to-date cache in the share-source would a significant issue). In the current situation you fix one bug, reintroducing another. I wonder if we have a "simpler or saner" way to introduce these various disctinction. Let me think about it a bit.

(Not sure if we need a wdir-independent-visible-hidden, I still don't really understand visible-hidden, I just know that updating it actually breaks a Lot of stuff, so I avoided doing so in the first commit in the stack. :))

visible-hidden still exclude all the internal phase commit that we never want to expose to anyone. (yeah the name is … confusing)

spectral planned changes to this revision.Sep 5 2019, 3:06 PM

I'm not sure what the current state of this is, but I don't want it lingering in the review queue while I'm not actively working on it.

spectral abandoned this revision.Apr 19 2021, 3:04 PM