( )⚙ D10551 dirstate-tree: Paralellize the status algorithm with Rayon

This is an archive of the discontinued Mercurial Phabricator instance.

dirstate-tree: Paralellize the status algorithm with Rayon
ClosedPublic

Authored by SimonSapin on May 3 2021, 6:25 AM.

Details

Summary

The rayon crate exposes "parallel iterators" that work like normal iterators
but dispatch work on different items to an implicit global thread pool.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

SimonSapin created this revision.May 3 2021, 6:25 AM
Alphare accepted this revision.May 3 2021, 10:48 AM
Alphare added a subscriber: Alphare.

I wonder which of using std::Mutex vs a crossbeam channel is faster. But this can be fine-tuned when we actually have most of the work done.

This revision is now accepted and ready to land.May 3 2021, 10:48 AM

Yes, this is worth benchmarking later. I think a mutex is at least not horribly slow because each thread spend a lot of time in readdir and stat without holding the mutex, and taking a mutex without contention should be fast (a few atomic operations). If the system’s / standard library’s mutex isn’t fast (does a syscall every time) we can try https://crates.io/crates/parking_lot. And channels also need synchronization: https://github.com/zesterer/flume is basically Arc<Mutex<VecDeque<T>> and has benchmarks claiming it’s competitive with crossbeam-channel. It’s possible that crossbeam-channel will still be faster (as far as I understand it has more fine-grained synchronization, with atomic metadata for each slot in the queue). Or maybe none of this is a bottleneck in the first place.

baymax updated this revision to Diff 27627.May 6 2021, 12:03 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)