This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4668

fileserverclient: don't wait for all batches getfiles before processing
ClosedPublic

Authored by martinvonz on Sep 20 2018, 1:10 AM.
Tags
None
Subscribers
None

Details

Reviewers
None
Group Reviewers
Restricted Project
Commits
rFBHGX3566090ff7e3: fileserverclient: don't wait for all batches getfiles before processing
Summary

The "new" peer.commandexecutor() (that I added support for in
3bfc430c40eb) waits for all responses to arrive in its exit()
method. I didn't expect that, so I put the processing of the futures
outside the context manager scope. To fix that, we simply need to move
the processing inside the context manager. Note that the command
executor modifies the futures' result() methods so the first result()
call will trigger executor.sendcommands(), so we don't need to
explicitly do that.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

martinvonz created this revision.Sep 20 2018, 1:10 AM
Herald added a reviewer: Restricted Project. · View Herald TranscriptSep 20 2018, 1:10 AM
Closed by commit rFBHGX3566090ff7e3: fileserverclient: don't wait for all batches getfiles before processing (authored by martinvonz). · Explain WhySep 20 2018, 4:11 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mremotefilelog/fileserverclient.py (14 lines)

Diff 11241

remotefilelog/fileserverclient.py

# progress. # progress.
# #
# Over ssh, iterbatch isn't streamy because batch() wasn't # Over ssh, iterbatch isn't streamy because batch() wasn't
# explicitly designed as a streaming method. In the future we # explicitly designed as a streaming method. In the future we
# should probably introduce a streambatch() method upstream and # should probably introduce a streambatch() method upstream and
# use that for this. # use that for this.
if (util.safehasattr(remote, 'commandexecutor')): if (util.safehasattr(remote, 'commandexecutor')):
futures = []
with remote.commandexecutor() as e: with remote.commandexecutor() as e:
futures = []
for m in missed: for m in missed:
futures.append(e.callcommand('getfile', { futures.append(e.callcommand('getfile', {
'file': idmap[m], 'file': idmap[m],
'node': m[-40:] 'node': m[-40:]
})) }))
for m, future in itertools.izip(missed, futures): for m, future in itertools.izip(missed, futures):
r = future.result() r = future.result()
file_ = idmap[m] file_ = idmap[m]
node = m[-40:] node = m[-40:]
receivemissing(io.BytesIO('%d\n%s' % (len(r), r)), file_, node) receivemissing(io.BytesIO('%d\n%s' % (len(r), r)), file_, node)
progresstick() progresstick()
return return
if (getattr(remote, 'iterbatch', False) and remote.capable('httppostargs') if (getattr(remote, 'iterbatch', False) and remote.capable('httppostargs')
and isinstance(remote, httppeer.httppeer)): and isinstance(remote, httppeer.httppeer)):
b = remote.iterbatch() b = remote.iterbatch()
for m in missed: for m in missed:
file_ = idmap[m] file_ = idmap[m]
node = m[-40:] node = m[-40:]