Here's the way we communicate between hg binary and an ssh binary in fileserverclient:
while not done: write STEP lines to PIPEI read all responses from PIPEO
where PIPEI and PIPEO are communication pipes between hg and ssh, and STEP
is a configurable number of requests for file data (each looking like filename:hash).
In the past STEP used to be always 10000 and it just worked for us, because the default
Linux pipe capacity (65536, I call it the pipe buffer size) was always enough.
However, it is pretty obvious that it is possible for this thing to deadlock
if the total size of STEP lines is bigger than the buffer size. This is what
happened on Windows, where default buffer size is 4096 bytes, so we made STEP
configurable and set it to 100 on Windows, which proved to work.
A better solution is to make PIPEI have a larger buffer, which is possible if
we create the pipe manually, instead of asking subprocess to do so for us.
Later we can also get rid of the STEP altogether by just counting how many bites
we've written in each loop iteration.
Should we just default to a large size for windows? I know other companies have hit this as well.