Page MenuHomePhabricator

narrow: use narrow_widen wireproto command to widen in case of ellipses
ClosedPublic

Authored by pulkit on May 22 2019, 5:47 PM.

Details

Summary

Few releases ago, we introduce narrow_widen wireproto command to be used to widen
narrow repositories. Before this patch, that was used in non-ellipses cases
only. In ellipses cases, we still do exchange.pull() which can pull more data
than required.

After this patch, the client will first check whether server supports doing
ellipses widening using wireproto command or not by checking server's wireproto
capability. If the server is upto date and support latest ellipses capability,
we call the wireproto command. Otherwise we fallback to exchange.pull() like
before.

The compat code make sure that things works even if one of the client or server
is old. The initial version of this patch does not had this compat code. It's
added to help Google release things smoothly internally. I plan to drop the
compat code before the upcoming major release.

Due to change to wireproto command, the code looks a bit dirty, next patches
will clean that up.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

pulkit created this revision.May 22 2019, 5:47 PM

This is going to break our server. Can you make it work with existing servers for a while (behave differently depending on capability, I suppose), so we can get a chance to transition? I would think it's not going to be terribly hard, but let me know if it would take more time than you have and I'll try to help.

This is going to break our server. Can you make it work with existing servers for a while (behave differently depending on capability, I suppose), so we can get a chance to transition? I would think it's not going to be terribly hard, but let me know if it would take more time than you have and I'll try to help.

Sure, will send an updated version soon.

pulkit edited the summary of this revision. (Show Details)May 28 2019, 9:42 AM
pulkit updated this revision to Diff 15271.
martinvonz added inline comments.Jun 3 2019, 7:47 PM
hgext/narrow/narrowcommands.py
149

Also need to check old capability here

257–258

nit: any(cap in remotecap for cap in wireprototypes.SUPPORTED_ELLIPSESCAP)

282

Should keep this in the "old" case

303–305

Is the known set needed when not using ellipses? Conversely, commonheads shouldn't be needed when using ellipses, but perhaps it's still useful to have it there (it's usually way smaller, so it's much less of a concern). These things can be fixed in a separate patch, of course.

rdamazio added inline comments.
hgext/narrow/narrowcommands.py
306

Being nosy here: does this mean that it'll download the whole bundle into memory before it starts applying it, rather than streaming it into the store? Bundles, especially in repos using narrow, can be very large, so that would not be ideal.

pulkit added inline comments.Jun 4 2019, 11:12 AM
hgext/narrow/narrowcommands.py
149

Only need to check the old capability here. Will update.

282

We don't need this in old cases. This is only required when widening non-ellipses cases as that will end us in a situation with an empty changegroup.

303–305

No, the known set is not needed in non-ellipses cases. We should not compute that also as in non-ellipses repos, computing known just adding all local commits which can be in millions. The server logic completely rely on common in non-ellipses cases.

pulkit added inline comments.
hgext/narrow/narrowcommands.py
306

I believe it streams rather than downloading the whole bundle into memory. getbundle() uses the same function and I think we do streaming there. https://www.mercurial-scm.org/repo/hg-committed/file/127937874395/mercurial/exchange.py#l1757

I also asked @marmoute and he also think that it does streaming.

pulkit updated this revision to Diff 15334.Jun 4 2019, 11:23 AM
pulkit updated this revision to Diff 15336.Jun 4 2019, 1:52 PM
This revision was automatically updated to reflect the committed changes.

Hopefully not too late

hgext/narrow/narrowcommands.py
306

The difference from the code you're pointing too is that you're calling .result() here, which blocks until the results are fully streamed in (see httppeer.py) - so I really think this is NOT streaming.

pulkit added inline comments.Jun 5 2019, 11:53 AM
hgext/narrow/narrowcommands.py
306
  1. I see a .result() there too https://www.mercurial-scm.org/repo/hg-committed/file/127937874395/mercurial/exchange.py#l1751
  1. I spent some time understanding whether this is streaming or not, it looks like it is not. I will try to understand more the related code path and update about my findings.