Page MenuHomePhabricator

nodemap: introduce append-only incremental update of the persisten data

Authored by marmoute on Jan 15 2020, 9:55 AM.



Rewriting the full nodemap for each transaction has a cost we would like to
avoid. We introduce a new way to write persistent nodemap data by adding new
information at the end for file. Any new and updated block as added at the end
of the file. The last block is the new root node.

With this method, some of the block already on disk get "dereferenced" and
become dead data. In later changesets, We'll start tracking the amount of dead
data to eventually re-generate a full nodemap.

Diff Detail

rHG Mercurial
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Jan 15 2020, 9:55 AM
martinvonz added inline comments.

Which patch introduced the C code? I have not looked at all patches yet, but I thought the idea was that this entire stack was just pure Python and then you'd add a Rust version (not C) in a separate series.

marmoute added inline comments.Jan 17 2020, 5:19 AM

There are no C code (and is non planned). This should be the C path or the cext policy.

marmoute updated this revision to Diff 19767.Jan 31 2020, 10:29 AM
marmoute updated this revision to Diff 19795.Jan 31 2020, 6:00 PM
marmoute updated this revision to Diff 19839.Feb 2 2020, 3:19 AM

small doc update on .#s[1]

marmoute updated this revision to Diff 19898.Feb 4 2020, 7:25 PM

rebase to latest default

This revision was not accepted when it landed; it landed in state Needs Review.
This revision was automatically updated to reflect the committed changes.