This is an archive of the discontinued Mercurial Phabricator instance.

worker: fix `_blockingreader.read()` to really block
AbandonedPublic

Authored by mharbison72 on Wed, May 18, 12:36 PM.

Details

Reviewers
mjacob
Group Reviewers
hg-reviewers
Summary

Maybe I'm missing something simple, but the help for io.BytesIO.readinto says:

Returns number of bytes read (0 for EOF), or None if the object
is set not to block and has no data to read.

and io.BytesIO.read says:

Return an empty bytes object at EOF.

That would seem to mean that if the _first_ internal readinto() of the
nonblocking self._wrapped returns None because no data is available, the caller
is tricked that EOF has been reached by returning an empty bytes object.

Diff Detail

Repository
rHG Mercurial
Branch
stable
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

mharbison72 created this revision.Wed, May 18, 12:36 PM

CI fails[1] with this series, but I think it's unrelated- its only chg, and test-http-bad-server.t fails for me locally in a different way, but identically with and without this series applied.

--- /mnt/c/Users/Matt/hg/tests/test-http-bad-server.t
+++ /mnt/c/Users/Matt/hg/tests/test-http-bad-server.t.err
@@ -42,7 +42,7 @@
   $ cat hg.pid > $DAEMON_PIDS

   $ hg clone http://localhost:$HGPORT/ clone
-  abort: error: (\$ECONNRESET\$|\$EADDRNOTAVAIL\$) (re)
+  abort: error: bad HTTP status line: ''
   [100]

[1] https://foss.heptapod.net/mercurial/mercurial-devel/-/jobs/552052

mjacob requested changes to this revision.Thu, May 19, 10:17 PM
mjacob added a subscriber: mjacob.

self._wrapped is not non-blocking, therefore self._wrapped.readinto() won’t ever return None, therefore this change only adds dead code.

The name of the class _blockingreader is misleading.

This revision now requires changes to proceed.Thu, May 19, 10:17 PM
mharbison72 abandoned this revision.Thu, May 19, 11:04 PM

self._wrapped is not non-blocking, therefore self._wrapped.readinto() won’t ever return None, therefore this change only adds dead code.
The name of the class _blockingreader is misleading.

I'm totally lost then- what's the point of this wrapper?

pickle.load() requires that the passed file’s read() returns exactly as many bytes as was passed to its argument and readinto() fills the passed buffer completely; otherwise it raises the “pickle data was truncated” error or EOFError if unambiguously encountering EOF.

For buffered streams this is the case if they are non-interactive and EOF is not reached (which we handle specifically) (according to https://docs.python.org/3/library/io.html#io.BufferedIOBase.read).

However, we can’t use buffered streams because the selector checks whether the unbuffered stream is readable while we need to check that the stream passed to pickle.load() is readable (the bug resulting from this was fixed in 12491abf93bd87b057cb6826e36606afa1cee88a).

_blockingreader wraps the unbuffered stream to meet the expectations of pickle.load() while not reading too much data to create the problem described in the previous paragraph.

pickle.load() requires that the passed file’s read() returns exactly as many bytes as was passed to its argument and readinto() fills the passed buffer completely; otherwise it raises the “pickle data was truncated” error or EOFError if unambiguously encountering EOF.
For buffered streams this is the case if they are non-interactive and EOF is not reached (which we handle specifically) (according to https://docs.python.org/3/library/io.html#io.BufferedIOBase.read).
However, we can’t use buffered streams because the selector checks whether the unbuffered stream is readable while we need to check that the stream passed to pickle.load() is readable (the bug resulting from this was fixed in 12491abf93bd87b057cb6826e36606afa1cee88a).
_blockingreader wraps the unbuffered stream to meet the expectations of pickle.load() while not reading too much data to create the problem described in the previous paragraph.

Interresting. Can I convince you to send a patch adding this explanation to the class docstring ? (and maybe fixing the class name)

Interresting. Can I convince you to send a patch adding this explanation to the class docstring ? (and maybe fixing the class name)

Yes, I planned to do so. Should this go to the default branch?

Interresting. Can I convince you to send a patch adding this explanation to the class docstring ? (and maybe fixing the class name)

Yes, I planned to do so. Should this go to the default branch?

Documentation changes are OK for stable, but renames aren't, so putting everything on default is probably easiest.

Interresting. Can I convince you to send a patch adding this explanation to the class docstring ? (and maybe fixing the class name)

Yes, I planned to do so. Should this go to the default branch?

Documentation changes are OK for stable, but renames aren't, so putting everything on default is probably easiest.

Yeah overall the philosophy of stable includes "restrict churn to minimum".

mjacob added a comment.EditedSat, May 21, 10:27 PM

Okay, I sent some patches adding documentation and comments (to the mailing list).