This is an archive of the discontinued Mercurial Phabricator instance.

infinitepush: override processparts
ClosedPublic

Authored by durham on Sep 14 2017, 5:17 PM.
Tags
None
Subscribers

Details

Reviewers
stash
Group Reviewers
Restricted Project
Commits
rFBHGX0cb55aec2779: infinitepush: override processparts
Summary

Upstream Mercurial now has a function that handles the iteration over bundle
parts. As part of making infinitepush handle more part types, let's override
processparts and completely take over part processing.

Initially the part processing will just mimic the existing handlers, but there's
a config flag that causes it to simply redirect the received parts into the new
bundle. This will be useful later for storing all bundle parts (like tree packs
and obsmarkers).

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durham created this revision.Sep 14 2017, 5:17 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptSep 14 2017, 5:17 PM
singhsrb added inline comments.
infinitepush/__init__.py
76

nit: IInstructs -> Instructs

durham updated this revision to Diff 1840.Sep 14 2017, 8:06 PM
durham marked an inline comment as done.Sep 14 2017, 8:06 PM
stash added a comment.Sep 15 2017, 1:05 PM

I've just updated to the latest hg-crew commit and can't find processparts() function in bundle2. Am I missing something?

stash added inline comments.Sep 15 2017, 1:19 PM
infinitepush/__init__.py
981–982

But this won't be possible to do, since we have parts that pass infinitepush bookmarks, and they need to be handled differently - we can't save them to the bundle. So we always need to handle some parts differently, don't we?

stash added inline comments.Sep 15 2017, 1:21 PM
infinitepush/__init__.py
1014–1016

I think it's possible to send infinitepush bundle, which without changegroup part. For example, infinitepush bundle, that just sends infinitepush bookmarks. We don't want to raise exception in that case, do we?

stash requested changes to this revision.Sep 15 2017, 1:24 PM

To your queue to answer questions

This revision now requires changes to proceed.Sep 15 2017, 1:24 PM

processparts is in a pending diff under review in upstream

infinitepush/__init__.py
981–982

Sure, some parts can be handled special, like how scratch branch parts are handled above. But this logic here opens the doors for migrating to not using custom parts, so it could be a completely server side extension (except for the backup push commands).

1014–1016

The storebundle logic only works if it has a changegroup part params available, so I assume it only worked if a changegroup part was sent. Is there a test for sending just bookmarks? If not, what would the repro be? Put a bookmark on master and push a scratch bookmark with no commits?

stash added inline comments.Sep 18 2017, 11:04 AM
infinitepush/__init__.py
77

Actually, you comment is written twice.

1014–1016

There are tests that cover this case. The problem is that infinitepush backups don't use standard push codepath to avoid taking locks. That means that line "bundler.addparam("infinitepush", "True")" is skipped and new logic is not tested on infinitepush backups tests.

If you add this diff:

diff --git a/infinitepush/backupcommands.py b/infinitepush/backupcommands.py
--- a/infinitepush/backupcommands.py
+++ b/infinitepush/backupcommands.py
@@ -424,6 +424,7 @@
     wrapfunction(changegroup.cg2packer, 'deltaparent', _deltaparent)
     try:
         bundler = _createbundler(ui, repo, other)
+        bundler.addparam("infinitepush", "True")
         backup = False
         if outgoing and outgoing.missing:
             backup = True

and run test-infinitepush-backup.t, then you'll see test failures.

It seems that new logic is tested on normal (i.e. non pushbackup) tests, but please double check it.

stash added inline comments.Sep 18 2017, 11:11 AM
infinitepush/__init__.py
1000–1002

BTW, why do you need it?

Also this code is not covered with tests. If we need it, can we cover it with tests?

durham added inline comments.Sep 20 2017, 1:11 PM
infinitepush/__init__.py
1000–1002

Why do we need handleallparts? So we can store other parts, like treepacks and obsmarkers, without having to create custom part types for them. In the long run, allowing the server to take over handling each part will allow us to send normal part types to the server and the server can decide what to do with them.

The last patch in the stack adds a test that runs this code, though it doesn't set the config. The config isn't really ready to be used, but having it available to flip helps us figure out where new bundle parts aren't working with infinitepush.

1014–1016

Good catch. Will fix

durham updated this revision to Diff 1941.Sep 20 2017, 1:46 PM
stash added inline comments.Sep 21 2017, 3:41 AM
infinitepush/__init__.py
1000–1002

No, sorry, I was interested in this code:

op.records.add(part.type, {
    'return': 1,
})

Why do you need to set return?

stash accepted this revision.Sep 21 2017, 3:53 AM
stash added inline comments.
infinitepush/__init__.py
1031–1032

Can you delete '@bundle2.parthandler(scratchbookmarksparttype)' line below and do:

if scratchbookpart:
  bundle2scratchbranch(op, part)

?

If you delete all '@bundle2.parthandler' then we'd have caught pushbackup bug.
Is there a reason why we keep these handlers?

1130–1132

Since you are handling it in processparts() can you remove these two lines

@bundle2.parthandler(scratchbranchparttype,
                     ('bookmark', 'bookprevnode' 'create', 'force',
                      'pushbackbookmarks', 'cgversion'))

?

This revision is now accepted and ready to land.Sep 21 2017, 3:53 AM
durham added inline comments.Sep 21 2017, 12:17 PM
infinitepush/__init__.py
1031–1032

For backwards compatibility. If we ship this to the servers, and there are still some old hg clients out there, they would be pushing without the infinitepush=True flag.

I think we'll get around to remove the part handlers later, when we stop sending custom part types.

This revision was automatically updated to reflect the committed changes.