( )⚙ D10696 remotefilelog: prevent file download to deadlock on windows

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10696

remotefilelog: prevent file download to deadlock on windows
AbandonedPublic

Authored by klevesque on May 7 2021, 9:02 AM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

When fetching a big file on Windows, the buffer gets full, which results on a
deadlock. To prevent this, we fetch the data hunk by chunk until all the data
has been retrieved.

Diff Detail

Repository
rHG Mercurial
Branch
stable
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

klevesque created this revision.May 7 2021, 9:02 AM
klevesque abandoned this revision.May 14 2021, 5:13 PM

I found out the the bug was cause by the fact that I was using an old version of MSYS2. It doesn't happen with a recent one.

Revision Contents
Changeset List

Diff 27741

hgext/remotefilelog/fileserverclient.py

def receivemissing(self, pipe, filename, node): def receivemissing(self, pipe, filename, node):
line = pipe.readline()[:-1] line = pipe.readline()[:-1]
if not line: if not line:
raise error.ResponseError( raise error.ResponseError(
_(b"error downloading file contents:"), _(b"error downloading file contents:"),
_(b"connection closed early"), _(b"connection closed early"),
) )
size = int(line) size = int(line)
data = pipe.read(size) data = _readhugestream(pipe, size)
if len(data) != size: if len(data) != size:
raise error.ResponseError( raise error.ResponseError(
_(b"error downloading file contents:"), _(b"error downloading file contents:"),
_(b"only received %s of %s bytes") % (len(data), size), _(b"only received %s of %s bytes") % (len(data), size),
) )
self.writedata.addremotefilelognode( self.writedata.addremotefilelognode(
filename, bin(node), zlib.decompress(data) filename, bin(node), zlib.decompress(data)
def logstacktrace(self): def logstacktrace(self):
import traceback import traceback
self.ui.log( self.ui.log(
b'remotefilelog', b'remotefilelog',
b'excess remotefilelog fetching:\n%s\n', b'excess remotefilelog fetching:\n%s\n',
b''.join(pycompat.sysbytes(s) for s in traceback.format_stack()), b''.join(pycompat.sysbytes(s) for s in traceback.format_stack()),
) )
if pycompat.iswindows:
# read() deadlock on Windows when size is too big
def _readhugestream(stream, size):
maxchunksize = 4 * 1024
data = bytearray()
while len(data) < size:
datachunk = stream.read(min(size - len(data), maxchunksize))
data += datachunk
if not datachunk:
# Prevents a deadlock in case the pipe is empty
break
return pycompat.bytestr(data)
else:
def _readhugestream(stream, size):
return stream.read(size)