Page MenuHomePhabricator

joerg.sonnenberger (Joerg Sonnenberger)
User

Projects

User does not belong to any projects.

User Details

User Since
Dec 6 2017, 9:25 AM (156 w, 3 d)

Recent Activity

Today

joerg.sonnenberger added a comment to D8574: hooks: teach reject_new_heads about phase transitions.

Superseded by D9525

Sat, Dec 5, 5:52 PM
joerg.sonnenberger created D9525: singlehead: introduce option to restrict to public changes.
Sat, Dec 5, 5:37 PM

Yesterday

joerg.sonnenberger added inline comments to D9518: wireprotov2: re-add tuple around `bundle2` check.
Fri, Dec 4, 7:41 AM

Thu, Dec 3

joerg.sonnenberger added a comment to D8784: tags: adjust file node cache to have space for 32 bytes node ids.

Will be done differently.

Thu, Dec 3, 4:46 PM
joerg.sonnenberger retitled D8480: bundle: optional advisory obsolescence parts from bundle: make obsolescence parts optional to bundle: optional advisory obsolescence parts.
Thu, Dec 3, 4:33 PM

Wed, Dec 2

joerg.sonnenberger updated the diff for D9480: node: import symbols explicitly.
Wed, Dec 2, 2:24 PM
joerg.sonnenberger updated the diff for D9480: node: import symbols explicitly.
Wed, Dec 2, 2:19 PM
joerg.sonnenberger committed rHGcb623dedb17c: relnotes: document better memory use for unbundle.
relnotes: document better memory use for unbundle
Wed, Dec 2, 3:04 AM
joerg.sonnenberger closed D9481: relnotes: document better memory use for unbundle.
Wed, Dec 2, 3:04 AM

Tue, Dec 1

joerg.sonnenberger created D9481: relnotes: document better memory use for unbundle.
Tue, Dec 1, 4:19 PM
joerg.sonnenberger created D9480: node: import symbols explicitly.
Tue, Dec 1, 4:12 PM

Mon, Nov 30

joerg.sonnenberger created D9465: [POC] node: replace nullid and friends with nodeconstants class.
Mon, Nov 30, 6:24 AM

Sun, Nov 29

joerg.sonnenberger closed D9443: tests: simplify and extend pull-bundle test using debugbuilddag.
Sun, Nov 29, 1:40 PM
joerg.sonnenberger committed rHG35d5d1d8bcf0: tests: simplify and extend pull-bundle test using debugbuilddag.
tests: simplify and extend pull-bundle test using debugbuilddag
Sun, Nov 29, 1:40 PM

Sat, Nov 28

joerg.sonnenberger created D9450: cext: isolate hash size in the revlog handling in a single place.
Sat, Nov 28, 5:40 PM
joerg.sonnenberger added a comment to D8693: sidedata: test changes for incoming and local changes [POC].

The non-POC part is superseded by D9445.

Sat, Nov 28, 9:06 AM
joerg.sonnenberger created D9445: sidedata: send the correct revision data for wireproto v2.
Sat, Nov 28, 9:05 AM
joerg.sonnenberger created D9443: tests: simplify and extend pull-bundle test using debugbuilddag.
Sat, Nov 28, 7:55 AM

Fri, Nov 27

joerg.sonnenberger closed D9162: revlog: store new index entries as binary.
Fri, Nov 27, 6:34 AM
joerg.sonnenberger committed rHG0ce15a8c7b8b: revlog: store new index entries as binary.
revlog: store new index entries as binary
Fri, Nov 27, 6:34 AM

Sat, Nov 21

joerg.sonnenberger added a comment to D9283: bundle: optional multithreaded compression, ATM zstd-only.

I was looking specifically at bzip2 for a bit. There are essentially two kinds of threaded compressors for it. pbzip2 is the more common and creates effectively multiple independent streams. That's not handled transparently by Python's bz2, so it would break all existing clients, making this a big no-go. Sourceforge has a more proper implementation for POSIX platforms (http://bzip2smp.sourceforge.net/) which doesn't have that problem and it would be nice if someone re-implemented the idea for modern libbz2. It can be done cleaner too. While this doesn't allow multi-threaded decompression for multi-stream-aware clients, it does work with all bzip2 decoders. Sadly the way it is done can't be from Python without re-implementing a good chunk of bz2 as it hooks deeply into the implementation. So in short, it would be possible to provide it as C extension and possibly even vendored, but it is more work than I currently want to do. I haven't looked into the state of pigz.

Sat, Nov 21, 4:24 PM

Thu, Nov 19

joerg.sonnenberger committed rHGec73a6a75985: transaction: split new files into a separate set.
transaction: split new files into a separate set
Thu, Nov 19, 6:40 PM
joerg.sonnenberger committed rHGa985c4fb23ca: transaction: change list of journal entries into a dictionary.
transaction: change list of journal entries into a dictionary
Thu, Nov 19, 6:40 PM
joerg.sonnenberger committed rHGa6f08085edfe: transaction: rename find to findoffset and drop backup file support.
transaction: rename find to findoffset and drop backup file support
Thu, Nov 19, 6:40 PM
joerg.sonnenberger committed rHG63edc384d3b7: transaction: drop per-file extra data support.
transaction: drop per-file extra data support
Thu, Nov 19, 6:40 PM

Wed, Nov 18

joerg.sonnenberger closed D9278: transaction: split new files into a separate set.
Wed, Nov 18, 6:07 PM
joerg.sonnenberger closed D9275: transaction: drop per-file extra data support.
Wed, Nov 18, 6:07 PM
joerg.sonnenberger committed rHGfae02ffcbae8: transaction: split new files into a separate set.
transaction: split new files into a separate set
Wed, Nov 18, 6:07 PM
joerg.sonnenberger closed D9276: transaction: rename find to findoffset and drop backup file support.
Wed, Nov 18, 6:07 PM
joerg.sonnenberger committed rHG5b1214c16a39: transaction: change list of journal entries into a dictionary.
transaction: change list of journal entries into a dictionary
Wed, Nov 18, 6:07 PM
joerg.sonnenberger closed D9277: transaction: change list of journal entries into a dictionary.
Wed, Nov 18, 6:07 PM
joerg.sonnenberger committed rHG0516dfea14e4: transaction: rename find to findoffset and drop backup file support.
transaction: rename find to findoffset and drop backup file support
Wed, Nov 18, 6:07 PM
joerg.sonnenberger committed rHG203f178a9dc8: transaction: drop per-file extra data support.
transaction: drop per-file extra data support
Wed, Nov 18, 6:07 PM
joerg.sonnenberger added a comment to D9277: transaction: change list of journal entries into a dictionary.

The last update implemented the option to read to journal from disk, in case that wasn't clear from the description change.

Wed, Nov 18, 11:52 AM
joerg.sonnenberger added inline comments to D9283: bundle: optional multithreaded compression, ATM zstd-only.
Wed, Nov 18, 11:50 AM

Tue, Nov 10

joerg.sonnenberger added a comment to D9162: revlog: store new index entries as binary.

Please retry, the parent changes are all merged at this point.

Tue, Nov 10, 8:44 AM

Sun, Nov 8

joerg.sonnenberger added a comment to D9284: purge: move extension into core mercurial.

I don't think just aborting when no option is given is a good UX. That feels even worse than making it interactive by default as discussed during the sprint.

Sun, Nov 8, 8:08 PM
joerg.sonnenberger committed rHG793976f9029c: relnotes: drop 5.6 release entries from next.
relnotes: drop 5.6 release entries from next
Sun, Nov 8, 7:37 PM
joerg.sonnenberger closed D9282: relnotes: drop 5.6 release entries from next.
Sun, Nov 8, 7:37 PM
joerg.sonnenberger added a comment to D9284: purge: move extension into core mercurial.

The proposal during the spring was to ask by default if it is not loaded as extension. Or to make it more explicit:
(1) Check if there is anything to do. If no, just return success.
(2) Check if the extension was enabled. If yes, remove and exit.
(3) Check if the "--confirm" (or "--force"?) flag was given. If not, ask for confirmation to remove X files.
(4) Remove files.

Sun, Nov 8, 6:55 PM
joerg.sonnenberger added a comment to D9285: strip: move into core.

My memory from the discussion was that we explicitly wanted to keep it as extension as it normal users shouldn't use this and it is considered dangerous?

Sun, Nov 8, 6:48 PM
joerg.sonnenberger created D9283: bundle: optional multithreaded compression, ATM zstd-only.
Sun, Nov 8, 2:50 PM
joerg.sonnenberger created D9282: relnotes: drop 5.6 release entries from next.
Sun, Nov 8, 2:12 PM
joerg.sonnenberger updated the diff for D9278: transaction: split new files into a separate set.
Sun, Nov 8, 10:00 AM
joerg.sonnenberger updated the summary of D9277: transaction: change list of journal entries into a dictionary.
Sun, Nov 8, 9:29 AM

Sat, Nov 7

joerg.sonnenberger added a comment to D9277: transaction: change list of journal entries into a dictionary.

I understand the concern and agree. I see two options of dealing with this without breaking the intention of the change here.
(1) Reverse the order. This would fix the immediate problem, but is more risky for similar classes of issues in other places, e.g. tree manifest chains.
(2) Adopt the logic from D9237 to read back the journal to recover the original order. Since both _playback and repair.strip will only ever be called once per transaction, it doesn't alter the performance much.

Sat, Nov 7, 6:44 PM
joerg.sonnenberger added a comment to D9278: transaction: split new files into a separate set.

I've been wondering about that question as well. In theory, we could run into O(n^2) behavior when there are many files that end up being split. I've decided to take the step back from the full dropping in D9237 as most transactions are much smaller than the initial clone and therefore are not as sensitive to the memory use.

Sat, Nov 7, 6:41 PM
joerg.sonnenberger created D9275: transaction: drop per-file extra data support.
Sat, Nov 7, 4:32 PM
joerg.sonnenberger created D9276: transaction: rename find to findoffset and drop backup file support.
Sat, Nov 7, 4:32 PM
joerg.sonnenberger created D9278: transaction: split new files into a separate set.
Sat, Nov 7, 4:32 PM
joerg.sonnenberger created D9277: transaction: change list of journal entries into a dictionary.
Sat, Nov 7, 4:32 PM
joerg.sonnenberger added inline comments to D9274: internals: start to document how .hg/ works.
Sat, Nov 7, 3:56 PM
joerg.sonnenberger added a comment to D9237: transaction: only keep file names in-memory for journal [WIP].

I'll cut this into smaller parts that are easier to review and decide on which steps are too far.

Sat, Nov 7, 1:09 PM

Fri, Nov 6

joerg.sonnenberger added a comment to D9237: transaction: only keep file names in-memory for journal [WIP].

My understanding of the transaction processing is that we keep track of all files modified. For each file, we remember the old size. When we have to rollback the transaction, we iterate over that list and truncate it at that point. Now this part works as before, just that we don't keep the file name -> offset table in memory but read back the on-disk representation. I don't think this makes a performance difference either compared to everything else going on. There is one more user of the offsets and that's the logic in revlog for moving from inline data storage to separated data files. With the current patch, it would be O(n^2) worst case as it reparses the whole file.

Fri, Nov 6, 2:36 PM

Nov 3 2020

joerg.sonnenberger closed D9161: unionrepo: don't insert index tuples with None as int field.
Nov 3 2020, 1:56 PM
joerg.sonnenberger committed rHGb6f4a1df91bd: unionrepo: don't insert index tuples with None as int field.
unionrepo: don't insert index tuples with None as int field
Nov 3 2020, 1:56 PM
joerg.sonnenberger closed D9160: bundlerepo: don't insert index tuples with full nodes as linkrev.
Nov 3 2020, 1:55 PM
joerg.sonnenberger committed rHG88d5abec8f61: bundlerepo: don't insert index tuples with full nodes as linkrev.
bundlerepo: don't insert index tuples with full nodes as linkrev
Nov 3 2020, 1:55 PM

Nov 2 2020

joerg.sonnenberger updated the summary of D9161: unionrepo: don't insert index tuples with None as int field.
Nov 2 2020, 5:14 PM
joerg.sonnenberger updated the summary of D9160: bundlerepo: don't insert index tuples with full nodes as linkrev.
Nov 2 2020, 5:05 PM
joerg.sonnenberger committed rHG4404f129341e: revlog: don't cache parsed tuples in the C module.
revlog: don't cache parsed tuples in the C module
Nov 2 2020, 2:52 PM

Oct 30 2020

joerg.sonnenberger closed D9155: revlog: don't cache parsed tuples in the C module.
Oct 30 2020, 2:42 PM
joerg.sonnenberger committed rHG9c6f9a1a0d1c: revlog: don't cache parsed tuples in the C module.
revlog: don't cache parsed tuples in the C module
Oct 30 2020, 2:42 PM
joerg.sonnenberger committed rHG5b6c0af021da: utils: helper function to print top memory allocation site.
utils: helper function to print top memory allocation site
Oct 30 2020, 1:03 PM

Oct 29 2020

joerg.sonnenberger committed rHG4f50d0303557: utils: helper function to print top memory allocation site.
utils: helper function to print top memory allocation site
Oct 29 2020, 4:48 PM
joerg.sonnenberger closed D9236: utils: helper function to print top memory allocation site.
Oct 29 2020, 4:48 PM
joerg.sonnenberger updated the diff for D9236: utils: helper function to print top memory allocation site.
Oct 29 2020, 4:47 PM
joerg.sonnenberger added a comment to D9155: revlog: don't cache parsed tuples in the C module.

Sorry, let me retract that. Wrong review, this one can be pulled out in isolation.

Oct 29 2020, 4:24 PM
joerg.sonnenberger committed rHGc4ab93849383: relnotes: mention improved memory use and underlaying API changes.
relnotes: mention improved memory use and underlaying API changes
Oct 29 2020, 10:35 AM
joerg.sonnenberger closed D9258: relnotes: mention improved memory use and underlaying API changes.
Oct 29 2020, 10:35 AM
joerg.sonnenberger created D9258: relnotes: mention improved memory use and underlaying API changes.
Oct 29 2020, 8:29 AM

Oct 28 2020

joerg.sonnenberger closed D9231: revlog: extend addgroup() with callback for duplicates.
Oct 28 2020, 10:02 PM
joerg.sonnenberger closed D9233: phases: convert registernew users to use revision sets.
Oct 28 2020, 10:01 PM
joerg.sonnenberger committed rHG5d65e04b6a80: phases: convert registernew users to use revision sets.
phases: convert registernew users to use revision sets
Oct 28 2020, 10:01 PM
joerg.sonnenberger committed rHG09735cde6275: phases: allow registration and boundary advancement with revision sets.
phases: allow registration and boundary advancement with revision sets
Oct 28 2020, 10:00 PM
joerg.sonnenberger closed D9232: phases: allow registration and boundary advancement with revision sets.
Oct 28 2020, 10:00 PM
joerg.sonnenberger committed rHGa5206e71c536: revlog: extend addgroup() with callback for duplicates.
revlog: extend addgroup() with callback for duplicates
Oct 28 2020, 10:00 PM
joerg.sonnenberger closed D9235: revlog: use LRU for the chain cache.
Oct 28 2020, 9:59 PM
joerg.sonnenberger committed rHG8719a5b68419: revlog: use LRU for the chain cache.
revlog: use LRU for the chain cache
Oct 28 2020, 9:58 PM

Oct 27 2020

joerg.sonnenberger added a comment to D9160: bundlerepo: don't insert index tuples with full nodes as linkrev.

The index has a documented format, but it wasn't enforced so far. With the pending child review, that would change. The case of cs==node is seen when we send full changesets in the bundle. In that case we used to leak the full node id into the index.

Oct 27 2020, 8:39 PM
joerg.sonnenberger added a comment to D9155: revlog: don't cache parsed tuples in the C module.

No, the two parents are necessary because they store invalid data in the index, i.e. they violate the interface contract, but it wasn't enforced so far.

Oct 27 2020, 8:28 PM

Oct 23 2020

joerg.sonnenberger updated the diff for D9233: phases: convert registernew users to use revision sets.
Oct 23 2020, 2:36 PM
joerg.sonnenberger retitled D9232: phases: allow registration and boundary advancement with revision sets from phases: allow registration and boundary advancement with revsion sets to phases: allow registration and boundary advancement with revision sets.
Oct 23 2020, 2:34 PM

Oct 22 2020

joerg.sonnenberger added inline comments to D9232: phases: allow registration and boundary advancement with revision sets.
Oct 22 2020, 3:49 PM

Oct 21 2020

joerg.sonnenberger created D9237: transaction: only keep file names in-memory for journal [WIP].
Oct 21 2020, 5:45 PM
joerg.sonnenberger created D9236: utils: helper function to print top memory allocation site.
Oct 21 2020, 4:30 PM

Oct 20 2020

joerg.sonnenberger created D9235: revlog: use LRU for the chain cache.
Oct 20 2020, 8:18 PM
joerg.sonnenberger created D9231: revlog: extend addgroup() with callback for duplicates.
Oct 20 2020, 11:34 AM
joerg.sonnenberger created D9233: phases: convert registernew users to use revision sets.
Oct 20 2020, 11:33 AM
joerg.sonnenberger created D9232: phases: allow registration and boundary advancement with revision sets.
Oct 20 2020, 11:33 AM

Oct 15 2020

joerg.sonnenberger committed rHG44d84b726c8a: unbundle: free temporary objects after use.
unbundle: free temporary objects after use
Oct 15 2020, 3:06 PM

Oct 14 2020

joerg.sonnenberger retitled D9162: revlog: store new index entries as binary from [WIP] revlog: store new index entries as binary to revlog: store new index entries as binary.
Oct 14 2020, 4:29 PM
joerg.sonnenberger retitled D9155: revlog: don't cache parsed tuples in the C module from [WIP] revlog: don't cache parsed tuples in the C module to revlog: don't cache parsed tuples in the C module.
Oct 14 2020, 4:28 PM
joerg.sonnenberger added a comment to D9153: unbundle: free temporary objects after use.

Re-submitted the change in a way that the Python2.7 nested scope issue is avoided.

Oct 14 2020, 8:19 AM
joerg.sonnenberger updated the summary of D9153: unbundle: free temporary objects after use.
Oct 14 2020, 8:16 AM

Oct 12 2020

joerg.sonnenberger updated the summary of D9153: unbundle: free temporary objects after use.
Oct 12 2020, 6:36 PM
joerg.sonnenberger updated the summary of D9153: unbundle: free temporary objects after use.
Oct 12 2020, 4:16 PM
joerg.sonnenberger closed D9153: unbundle: free temporary objects after use.
Oct 12 2020, 2:11 PM
joerg.sonnenberger committed rHG3aa7b0c1b97c: unbundle: free temporary objects after use.
unbundle: free temporary objects after use
Oct 12 2020, 2:11 PM

Oct 9 2020

joerg.sonnenberger added a comment to D9155: revlog: don't cache parsed tuples in the C module.

Basically, because the cache in the current form is very expensive in terms of memory use. For a large repository, it can easily consume 100MB+ for something like 2% runtime gain, which doesn't seem a good trade-off.

Oct 9 2020, 5:49 AM