This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2720

debugcommands: introduce actions to perform deterministic reads
ClosedPublic

Authored by indygreg on Mar 7 2018, 8:36 PM.

Download Raw Diff

Details

Reviewers

durin42

Group Reviewers

hg-reviewers

Commits

rHG143219fc2620: debugcommands: introduce actions to perform deterministic reads

Summary

"readavailable" is useful as a debugging device to see what data is
available on a pipe. But the mechanism isn't deterministic because
what's available on a pipe is highly conditional on timing, system
load, OS behavior, etc. This makes it not suitable for tests.

We introduce "ereadline," "read," and "eread" for performing
deterministic I/O operations (at least on blocking file descriptors).

We stop short of converting existing consumers of "readavailable"
in tests because we're working out race conditions and deadlocks
on Windows. But the goal is to eventually move tests away from
"readavailable" to these new APIs.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 7 2018, 8:36 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptMar 7 2018, 8:36 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

indygreg added a parent revision: D2719: wireproto: formalize permissions checking as part of protocol interface.Mar 7 2018, 8:37 PM

indygreg added a child revision: D2721: util: observable proxy objects for sockets.Mar 8 2018, 12:31 AM

This patch seems to deadlock on Windows when running test-ssh-proto-unbundle.t. There are two python processes idling:

c:\Python27\python.exe c:/Users/Matt/projects/hg/hg --verbose debugwireproto --localssh --noreadstderr
CWD: C:\Users\Matt\AppData\Local\Temp\hgtests.1mtivg\child1\test-ssh-proto-unbundle.t\no-bundle1\

and

c:\Python27\python.exe c:/Users/Matt/projects/hg/hg -R C:\Users\Matt\AppData\Local\Temp\hgtests.1mtivg\child1\test-ssh-proto-unbundle.t\no-bundle1 debugserve --sshstdio
CWD: C:\Users\Matt\AppData\Local\Temp\hgtests.1mtivg\child1\test-ssh-proto-unbundle.t\no-bundle1\

Past experience says that something needs to be flushed- stdout (and maybe stderr?) are full buffered on Windows IIRC. The following failure with this patch might also point to that:

--- c:/Users/Matt/projects/hg/tests/test-ssh-proto.t
+++ c:/Users/Matt/projects/hg/tests/test-ssh-proto.t.err
@@ -100,12 +100,12 @@
   $ hg debugserve --sshstdio --logiofd 1 << EOF
   > hello
   > EOF
-  o> write(4) -> 4:
-  o>     384\n
-  o> write(384) -> 384:
-  o>     capabilities: lookup branchmap pushkey known getbundle unbundlehash batch changegroupsubset streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$unbundle=HG10GZ,HG10BZ,HG10UN\n
   384
   capabilities: lookup branchmap pushkey known getbundle unbundlehash batch changegroupsubset streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN
+  o> write(4) -> 4:
+  o>     384\n
+  o> write(384) -> 384:
+  o>     capabilities: lookup branchmap pushkey known getbundle unbundlehash batch changegroupsubset streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN\n
   o> flush() -> None

   $ hg debugserve --sshstdio --logiofile $TESTTMP/io << EOF

In D2720#44221, @mharbison72 wrote:

This patch seems to deadlock on Windows when running test-ssh-proto-unbundle.t. There are two python processes idling:
Past experience says that something needs to be flushed- stdout (and maybe stderr?) are full buffered on Windows IIRC. The following failure with this patch might also point to that:

Hmmm.

We explicitly create the pipes between the processes as unbuffered (bufsize=0). That being said, I would not at all be surprised if we need to throw a flush() in here somewhere.

Could you please run the test with run-tests.py -d and see where it is deadlocking?

The test failure due to the o> block is likely due to buffering or a race condition when writing to stdout of the hg debugwireproto process. I'll look into that after we fix the deadlock (since fixing might change order of things and I don't want to debug this in a system that is changing).

In D2720#44288, @indygreg wrote:

In D2720#44221, @mharbison72 wrote:

This patch seems to deadlock on Windows when running test-ssh-proto-unbundle.t. There are two python processes idling:
Past experience says that something needs to be flushed- stdout (and maybe stderr?) are full buffered on Windows IIRC. The following failure with this patch might also point to that:

Hmmm.
We explicitly create the pipes between the processes as unbuffered (bufsize=0). That being said, I would not at all be surprised if we need to throw a flush() in here somewhere.
Could you please run the test with run-tests.py -d and see where it is deadlocking?
The test failure due to the o> block is likely due to buffering or a race condition when writing to stdout of the hg debugwireproto process. I'll look into that after we fix the deadlock (since fixing might change order of things and I don't want to debug this in a system that is changing).

It looks like the middle of line 84. Here's the relevant output, from the start of the stuck command on 43 (the off-by-one in the SALT line is odd, but the other lines are line that too, so I assume it's a zero based index thing):

+ echo SALT1520615870 42 0
SALT1520615870 42 0
+ debugwireproto
++ cat -
+ commands='command unbundle
# This is "force" in hex.
    heads 666f726365
    PUSHFILE ../initial.v1.hg
eread 115'
+ echo 'testing ssh1'
testing ssh1
++ hg log -r tip -T '{node}'
+ tip=0000000000000000000000000000000000000000
+ echo 'command unbundle
# This is "force" in hex.
    heads 666f726365
    PUSHFILE ../initial.v1.hg
eread 115'
+ hg --verbose debugwireproto --localssh --noreadstderr
creating ssh peer from handshake results
i> write(104) -> 104:
i>     hello\n
i>     between\n
i>     pairs 81\n
i>     0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
i> flush() -> None
o> readline() -> 4:
o>     384\n
o> readline() -> 384:
o>     capabilities: lookup branchmap pushkey known getbundle unbundlehash batch changegroupsubset streamreqs=
generaldelta,revlogv1 bundle2=HG20%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror
%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Arem
ote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN\n
o> readline() -> 2:
o>     1\n
o> readline() -> 1:
o>     \n
sending unbundle command
i> write(9) -> 9:
i>     unbundle\n
i> write(9) -> 9:
i>     heads 10\n
i> write(10) -> 10: 666f726365
i> flush() -> None
o> readline() -> 2:
o>     0\n
i> write(4) -> 4:
i>     426\n
i> write(426) -> 426:
i>     HG10UN\x00\x00\x00\x9eh\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00>cba485ca3678256e044428f70f58291196f6e9de\n
i>     test\n
i>     0 0\n
i>     foo\n
i>     \n
i>     initial\x00\x00\x00\x00\x00\x00\x00\x8d\xcb\xa4\x85\xca6x%n\x04D(\xf7\x0fX)\x11\x96\xf6\xe9\xde\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00-foo\x00362fef284ce2ca02aecc8de6d5e8a1c3af0556fe\n
i>     \x00\x00\x00\x00\x00\x00\x00\x07foo\x00\x00\x00b6/\xef(L\xe2\xca\x02\xae\xcc\x8d\xe6\xd5\xe8\xa1\xc3\xa
f\x05V\xfe\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\

In D2720#44293, @mharbison72 wrote:
i> write(4) -> 4:
i> 426\n
i> write(426) -> 426:
i> HG10UN\x00\x00\x00\x9eh\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00>cba485ca3678256e044428f70f58291196f6e9de\n
i> test\n
i> 0 0\n
i> foo\n
i> \n
i> initial\x00\x00\x00\x00\x00\x00\x00\x8d\xcb\xa4\x85\xca6x%n\x04D(\xf7\x0fX)\x11\x96\xf6\xe9\xde\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00-foo\x00362fef284ce2ca02aecc8de6d5e8a1c3af0556fe\n
i> \x00\x00\x00\x00\x00\x00\x00\x07foo\x00\x00\x00b6/\xef(L\xe2\xca\x02\xae\xcc\x8d\xe6\xd5\xe8\xa1\xc3\xa
f\x05V\xfe\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\

This is a wonky place to hang! For context, this is processing a command unbundle line. The lines that follow should be:

i> write(2) -> 2:
i>     0\n
i> flush() -> None
o> readline() -> 2:
o>     0\n
o> readline() -> 2:
o>     1\n
o> read(1) -> 1: 0
result: 0
remote output: 
e> read(115) -> 115:
e>     abort: incompatible Mercurial client; bundle2 required\n
e>     (see https://www.mercurial-scm.org/wiki/IncompatibleClient)\n

What happens under the hood during pushes is we send out chunks containing the bundle followed by an empty chunk. That code is:

for d in iter(lambda: fp.read(4096), ''):
    self._writeframed(d)
self._writeframed("", flush=True)

That empty chunk is apparently not getting sent. Or its logging is not getting written/printed.

This patch shouldn't have changed any behavior with regard to this part of the I/O. So I'm scratching my head over how this caused deadlock. Are you sure you can bisect it to this patch.

In D2720#44350, @indygreg wrote:
In D2720#44293, @mharbison72 wrote:
i> write(4) -> 4:
i> 426\n
i> write(426) -> 426:
i> HG10UN\x00\x00\x00\x9eh\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00>cba485ca3678256e044428f70f58291196f6e9de\n
i> test\n
i> 0 0\n
i> foo\n
i> \n
i> initial\x00\x00\x00\x00\x00\x00\x00\x8d\xcb\xa4\x85\xca6x%n\x04D(\xf7\x0fX)\x11\x96\xf6\xe9\xde\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00-foo\x00362fef284ce2ca02aecc8de6d5e8a1c3af0556fe\n
i> \x00\x00\x00\x00\x00\x00\x00\x07foo\x00\x00\x00b6/\xef(L\xe2\xca\x02\xae\xcc\x8d\xe6\xd5\xe8\xa1\xc3\xa
f\x05V\xfe\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\
This is a wonky place to hang! For context, this is processing a command unbundle line. The lines that follow should be:
i> write(2) -> 2:
i>     0\n
i> flush() -> None
o> readline() -> 2:
o>     0\n
o> readline() -> 2:
o>     1\n
o> read(1) -> 1: 0
result: 0
remote output: 
e> read(115) -> 115:
e>     abort: incompatible Mercurial client; bundle2 required\n
e>     (see https://www.mercurial-scm.org/wiki/IncompatibleClient)\n
What happens under the hood during pushes is we send out chunks containing the bundle followed by an empty chunk. That code is:
for d in iter(lambda: fp.read(4096), ''):
    self._writeframed(d)
self._writeframed("", flush=True)
That empty chunk is apparently not getting sent. Or its logging is not getting written/printed.
This patch shouldn't have changed any behavior with regard to this part of the I/O. So I'm scratching my head over how this caused deadlock. Are you sure you can bisect it to this patch.

Yep. I imported this with hg phapread --stack, and can run the D2719 in 50 seconds or so. The debug run of this I let hang out for 20 minutes before killing it.

Here's the neighboring failure from D2719, because I can't explain the AWOL output. (But that shouldn't have stopped stdout above mid stream.)

--- e:/Projects/hg/tests/test-ssh-proto-unbundle.t
+++ e:/Projects/hg/tests/test-ssh-proto-unbundle.t.err
@@ -93,9 +93,6 @@
   o> read(1) -> 1: 0
   result: 0
   remote output:
-  e> read(-1) -> 115:
-  e>     abort: incompatible Mercurial client; bundle2 required\n
-  e>     (see https://www.mercurial-scm.org/wiki/IncompatibleClient)\n

   testing ssh2
   creating ssh peer from handshake results
@@ -143,9 +140,6 @@
   o> read(1) -> 1: 0
   result: 0
   remote output:
-  e> read(-1) -> 115:
-  e>     abort: incompatible Mercurial client; bundle2 required\n
-  e>     (see https://www.mercurial-scm.org/wiki/IncompatibleClient)\n

   $ cd ..

My only explanation is this is stdout output buffering and things are really hanging on the next read operation.

Maybe try sprinkling some ui.fout.flush() and/or util.stdout.flush() and/or sys.stdout.flush() calls throughout the debug command? Maybe at the first thing in the loop that evaluates commands to execute?

In D2720#44396, @indygreg wrote:

My only explanation is this is stdout output buffering and things are really hanging on the next read operation.
Maybe try sprinkling some ui.fout.flush() and/or util.stdout.flush() and/or sys.stdout.flush() calls throughout the debug command? Maybe at the first thing in the loop that evaluates commands to execute?

Both util.stdout.flush() and ui.fout.flush() get a little further in the same command; sys.stdout.flush() has no effect, when added at the top of the loop. But then it stalls on remote output:

...
o> readline() -> 2:
o>     0\n
i> write(4) -> 4:
i>     426\n
i> write(426) -> 426:
i>     HG10UN\x00\x00\x00\x9eh\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
x00>cba485ca3678256e044428f70f58291196f6e9de\n
i>     test\n
i>     0 0\n
i>     foo\n
i>     \n
i>     initial\x00\x00\x00\x00\x00\x00\x00\x8d\xcb\xa4\x85\xca6x%n\x04D(\xf7\x0fX)\x11\x96\xf6\xe9\xde\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00-foo\x00362fef284ce2ca02aecc8de6d5e8a1c3af0556fe\n
i>     \x00\x00\x00\x00\x00\x00\x00\x07foo\x00\x00\x00b6/\xef(L\xe2\xca\x02\xae\xcc\x8d\xe6\xd5\xe8\xa1\xc3\xa
f\x05V\xfe\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00h\x98b\x13\xbdD\x85\xeaQS55\xe3\xfc\x9ex\x00zq\x1f
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x020\n
i>     \x00\x00\x00\x00\x00\x00\x00\x00
i> write(2) -> 2:
i>     0\n
i> flush() -> None
o> readline() -> 2:
o>     0\n
o> readline() -> 2:
o>     1\n
o> read(1) -> 1: 0
result: 0
remote output:

So I added a self.flush() inside ui._write() because I figured this is a problem on the remote end (and probably not localized like the debug command), but that didn't do anything. (ui._write_err() already flushes, and has a comment about stderr may be buffered on Windows.)

IDK if this means anything, but when it is stuck and I hit Ctrl + C, instead of terminating the test, the test simply continues with the script, and then hangs on the next 'remote output:' line. Typically, Ctrl + C ends the test runner.

IDK if this means anything, but when it is stuck and I hit Ctrl+C, instead of terminating the test, the test simply continues with the script, and then hangs on the next 'remote output:' line. Typically, Ctrl+C ends the test runner.

A couple of more tidbits on this:

When I hit Ctrl+C, it *does* print the shell prompt, but the process keeps printing test output after it, so I missed it. When it jams, I can hit Ctrl+C again to get it going again.
I did that all the way through the test, and eventually it got to the last one, which prints:

i> write(2) -> 2:
i>     0\n
i> flush() -> None
o> readline() -> 2:
o>     0\n
o> readline() -> 2:
o>     1\n
o> read(1) -> 1: 1
result: 1
remote output:
e> read(152) -> 152:
e>     adding changesets\n
e>     adding manifests\n
e>     adding file changes\n
e>     added 1 changesets with 1 changes to 1 files\n
e>     ui.write 1\n
e>     ui.write_err 1\n
e>     ui.write 2\n
e>     ui.write_err 2\n
+ '[' -n 1 ']'
+ hg --config extensions.strip= strip --no-backup -r 'all() - ::0000000000000000000000000000000000000000'
+ echo SALT1520635194 2034 0
SALT1520635194 2034 0

Note the exit code is 1 here. So is it possible that the debug command is having trouble reading bytes from an empty stderr in the initial cases?

mharbison72 mentioned this in rHGd7d427362119: util: forward __bool__()/__nonzero__() on fileobjectproxy.Mar 10 2018, 1:09 PM

mharbison72 mentioned this in rHG6bdea0efdab5: util: forward __bool__()/__nonzero__() on fileobjectproxy.Mar 10 2018, 1:21 PM

At this point, I've narrowed it down to stderr. I misunderstood the 'noreadstderr', and wondered why we were bothering to pipe stderr if True. So I set it to os.devnull, and the test sails through. (But it does drop all "e>" lines, so that's not an actual solution.)

Is there some way to send a signal to a python process to get it to dump its stacks, like Ctrl+\ in java? Sprinkling random prints and flushes hasn't been fruitful so far.

See the contrib/showstack.py extension for registering SIGQUIT and SIGINFO signals to dump the active stack. Not sure if that works on Windows though.

mharbison72 mentioned this in rHG9c636ec1ef37: hook: ensure stderr is flushed when an exception is raised, for test stability.Mar 11 2018, 5:58 AM

indygreg edited the summary of this revision. (Show Details)Mar 12 2018, 8:07 PM

indygreg updated this revision to Diff 6976.

indygreg added a child revision: D2726: debugcommands: support connecting to HTTP peers.Mar 12 2018, 8:09 PM

indygreg removed a child revision: D2721: util: observable proxy objects for sockets.

I removed the test changes from this. So we should be able to land it without causing chaos to Windows tests. It may conflict with other patches that @yuja and @mharbison72 have in flight though.

Sounds good to me. I think the only thing I have inflight is the patch with explicit flushes in the observer. I haven't tried Yuya's series yet, because I'm not sure where it fits into the other stuff inflight. I don't mind mopping this up after it lands.

indygreg added a child revision: D2839: tests: use $HTTP_DATE$ for Date header.Mar 13 2018, 3:11 PM

indygreg removed a child revision: D2726: debugcommands: support connecting to HTTP peers.Mar 13 2018, 3:20 PM

mharbison72 mentioned this in rHGd683c7367989: wireproto: explicitly flush stdio to prevent stalls on Windows.Mar 19 2018, 9:46 AM

durin42 accepted this revision.Mar 19 2018, 9:35 PM

This revision is now accepted and ready to land.Mar 19 2018, 9:35 PM

Closed by commit rHG143219fc2620: debugcommands: introduce actions to perform deterministic reads (authored by indygreg). · Explain WhyMar 19 2018, 9:40 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/debugcommands.py (23 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	6714		Mar 7 2018, 8:36 PM	★	★
Diff 2	6976		Mar 12 2018, 8:07 PM	★	★
Diff 3	7153	rHG143219fc2620ce59423200f7b4c830c42c2c3279	Mar 12 2018, 6:49 PM	★	★

Status	Author	Revision
Closed	indygreg	D2987 stringutil: add function to pretty print an object
Closed	indygreg	D2986 wireproto: add frame flag to denote payloads as CBOR
Closed	indygreg	D2985 wireproto: implement custom __repr__ for frame
Closed	indygreg	D2984 keepalive: implement readinto()
Closed	indygreg	D2983 wireproto: port protocol handler to zope.interface
Closed	indygreg	D2982 wireproto: separate commands tables for version 1 and 2 commands
Closed	indygreg	D2981 wireproto: mark SSHv2 as a version 1 transport
Closed	indygreg	D2979 wireproto: stop aliasing wire protocol types (API)
Closed	indygreg	D2951 wireproto: use CBOR for command requests
Closed	indygreg	D2902 wireproto: define frame to represent progress updates
Closed	indygreg	D2948 wireproto: syntax for encoding CBOR into frames
Closed	indygreg	D2947 wireproto: explicit API to create outgoing streams
Closed	indygreg	D2907 wireproto: add streams to frame-based protocol
Closed	indygreg	D2906 wireproto: start to associate frame generation with a stream
Closed	indygreg	D2950 tests: fix duplicate and failing test
Closed	indygreg	D2978 cbor: import CBORDecoder and CBOREncoder
Closed	indygreg	D2949 setup: install cbor packages
Abandoned	indygreg	D2885 RFC: use Redis to cache file data
Changes Planned	indygreg	D2884 wireproto: experimental command to emit file data
Abandoned	indygreg	D2883 revlogstore: create and implement an interface for repo files storage
Closed	indygreg	D2901 wireproto: explicitly track which requests are active
Closed	indygreg	D2900 wireproto: use named arguments when passing around frame data
Closed	indygreg	D2899 wireproto: define attr-based classes for representing frames
Closed	indygreg	D2872 wireproto: define human output side channel frame
Closed	indygreg	D2871 wireproto: service multiple command requests per HTTP request
Closed	indygreg	D2870 wireproto: support for receiving multiple requests
Closed	indygreg	D2869 wireproto: add request IDs to frames
Closed	indygreg	D2860 wireproto: buffer output frames when in half duplex mode
Closed	indygreg	D2858 wireproto: define and implement responses in framing protocol
Closed	indygreg	D2857 wireproto: implement basic command dispatching for HTTPv2
Closed	indygreg	D2856 wireproto: nominally don't expose "batch" to version 2 wire transports
Closed	indygreg	D2852 wireproto: implement basic frame reading and processing
Closed	indygreg	D2851 wireproto: define and implement protocol for issuing requests
Closed	indygreg	D2868 util: prefer "bytesio" to "stringio"
Closed	indygreg	D2850 wireproto: define content negotiation for HTTPv2
Closed	indygreg	D2849 hgweb: also set Content-Type header
Closed	indygreg	D2837 wireproto: require POST for all HTTPv2 requests
Closed	indygreg	D2836 wireproto: define permissions-based routing of HTTPv2 wire protocol
Closed	indygreg	D2834 wireproto: support /api/* URL space for exposing APIs
Closed	indygreg	D2843 url: support suppressing Accept header
Closed	indygreg	D2842 util: don't log low-level I/O calls for HTTP peer
Closed	indygreg	D2841 debugcommands: support sending HTTP requests with debugwireproto
Closed	indygreg	D2726 debugcommands: support connecting to HTTP peers
Closed	indygreg	D2722 url: add HTTP handler that uses a proxied socket
Closed	indygreg	D2721 util: observable proxy objects for sockets
Closed	indygreg	D2840 hgweb: allow defining Server response header for HTTP server
Closed	indygreg	D2839 tests: use $HTTP_DATE$ for Date header
Closed	indygreg	D2720 debugcommands: introduce actions to perform deterministic reads
Closed	indygreg	D2725 httppeer: refactor how httppeer is created (API)
Closed	indygreg	D2724 httppeer: alias url as urlmod
Closed	indygreg	D2723 httppeer: consolidate _requestbuilder assignments and document
Closed	indygreg	D2832 hgweb: remove wsgirequest (API)
Closed	indygreg	D2831 hgweb: store the raw WSGI environment dict
Closed	indygreg	D2830 hgweb: remove dead wsgirequest code
Closed	indygreg	D2829 hgweb: port to new response API
Closed	indygreg	D2828 hgweb: pass modern request type into templater()
Closed	indygreg	D2827 hgweb: use modern response type for index generation
Closed	indygreg	D2826 hgweb: don't pass wsgireq to makeindex and other functions
Closed	indygreg	D2825 hgweb: replace PATH_INFO with dispatchpath
Closed	indygreg	D2824 hgweb: rewrite path generation for index entries
Closed	indygreg	D2823 hgweb: construct {url} with req.apppath
Closed	indygreg	D2822 hgweb: support constructing URLs from an alternate base URL
Closed	indygreg	D2821 hgweb: clarify that apppath begins with a forward slash
Closed	indygreg	D2820 hgweb: change how dispatch path is reported
Closed	indygreg	D2819 hgweb: refactor repository name URL parsing
Closed	indygreg	D2818 tests: add test coverage for parsing WSGI requests
Closed	indygreg	D2817 hgweb: construct static URL like hgweb does
Closed	indygreg	D2816 hgweb: remove unused **map argument
Closed	indygreg	D2815 hgweb: extract entries() to standalone function
Closed	indygreg	D2814 hgweb: move rawentries() to a standalone function
Closed	indygreg	D2813 hgweb: move archivelist to standalone function
Closed	indygreg	D2812 hgweb: move readallowed to a standalone function
Closed	indygreg	D2805 hgweb: remove some use of wsgireq in hgwebdir
Closed	indygreg	D2804 hgweb: fix a bug due to variable name typo
Closed	indygreg	D2803 hgweb: stop passing req and tmpl into @webcommand functions (API)
Closed	indygreg	D2802 hgweb: pass modern request type into various webutil functions (API)
Closed	indygreg	D2801 hgweb: don't redundantly pass templater with requestcontext (API)
Closed	indygreg	D2800 hgweb: use templater on requestcontext instance
Closed	indygreg	D2799 hgweb: add a sendtemplate() helper function
Closed	indygreg	D2798 hgweb: use web.req instead of req.req
Closed	indygreg	D2797 hgweb: stop setting headers on wsgirequest
Closed	indygreg	D2796 hgweb: always return iterable from @webcommand functions (API)
Closed	indygreg	D2795 hgweb: send errors using new response API
Closed	indygreg	D2794 hgweb: refactor 304 handling code
Closed	indygreg	D2793 hgweb: transition permissions hooks to modern request type (API)
Closed	indygreg	D2792 hgweb: port archive command to modern response API
Closed	indygreg	D2791 hgweb: refactor fake file object proxy for archiving
Closed	indygreg	D2790 tests: additional test coverage of archive web command
Closed	indygreg	D2789 hgweb: port static file handling to new response API
Closed	indygreg	D2788 hgweb: remove one-off routing for file?style=raw
Closed	indygreg	D2787 hgweb: port most @webcommand to use modern response type
Closed	indygreg	D2786 hgweb: support using new response object for web commands
Closed	indygreg	D2785 hgweb: inline caching() and port to modern mechanisms
Closed	indygreg	D2784 hgweb: expose repo name on parsedrequest
Closed	indygreg	D2783 hgweb: expose URL scheme and REMOTE_* attributes
Closed	indygreg	D2782 hgweb: remove wsgirequest.form (API)
Closed	indygreg	D2781 hgweb: perform all parameter lookup via qsparams
Closed	indygreg	D2780 hgweb: set variables in qsparams
Closed	indygreg	D2779 hgweb: use our new request object for "style" parameter
Closed	indygreg	D2776 hgweb: use a multidict for holding query string parameters
Closed	indygreg	D2775 hgweb: create dedicated type for WSGI responses
Closed	indygreg	D2778 tests: add test for a wire protocol request to wrong base URL
Closed	indygreg	D2773 hgweb: remove support for short query string based aliases (BC)
Closed	indygreg	D2774 hgweb: remove support for POST form data (BC)
Closed	indygreg	D2771 hgweb: expose input stream on parsed WSGI request object
Closed	indygreg	D2770 hgweb: make parsedrequest part of wsgirequest
Closed	indygreg	D2769 hgweb: refactor the request draining code
Closed	indygreg	D2768 hgweb: use a capped reader for WSGI input stream
Closed	indygreg	D2767 hgweb: document continuereader
Closed	indygreg	D2749 hgweb: remove wsgirequest.__iter__
Closed	indygreg	D2748 hgweb: remove wsgirequest.read()
Closed	indygreg	D2747 hgweb: remove unused methods on wsgirequest
Closed	indygreg	D2746 wireprotoserver: remove unused argument from _handlehttperror()
Closed	indygreg	D2745 hgweb: store and use request method on parsed request
Closed	indygreg	D2744 hgweb: handle CONTENT_LENGTH
Closed	indygreg	D2743 wireprotoserver: access headers through parsed request
Closed	indygreg	D2742 hgweb: parse and store HTTP request headers
Closed	indygreg	D2741 wireprotoserver: remove broken optimization for non-httplib client
Closed	indygreg	D2740 wireprotoserver: move all wire protocol handling logic out of hgweb
Closed	indygreg	D2739 hgweb: use parsed request to construct query parameters
Closed	indygreg	D2738 hgweb: only recognize wire protocol commands from query string (BC)
Closed	indygreg	D2737 hgweb: teach WSGI parser about query strings
Closed	indygreg	D2736 hgweb: use the parsed application path directly
Closed	indygreg	D2735 hgweb: use computed base URL from parsed request
Closed	indygreg	D2734 hgweb: parse WSGI request into a data structure
Closed	indygreg	D2733 hgweb: always use "?" when writing session vars
Closed	indygreg	D2732 hgweb: rename req to wsgireq
Closed	indygreg	D2731 hgweb: validate WSGI environment dict
Closed	indygreg	D2730 hgweb: ensure all wsgi environment values are str

Diff 7153

mercurial/debugcommands.py

	If the connection to the server encompasses multiple pipes, we poll both			If the connection to the server encompasses multiple pipes, we poll both
	pipes and read available data.			pipes and read available data.

	readline			readline
	--------			--------

	Read a line of output from the server. If there are multiple output			Read a line of output from the server. If there are multiple output
	pipes, reads only the main pipe.			pipes, reads only the main pipe.

				ereadline
				---------

				Like ``readline``, but read from the stderr pipe, if available.

				read <X>
				--------

				``read()`` N bytes from the server's main output pipe.

				eread <X>
				---------

				``read()`` N bytes from the server's stderr pipe, if available.
	"""			"""
	opts = pycompat.byteskwargs(opts)			opts = pycompat.byteskwargs(opts)

	if opts['localssh'] and not repo:			if opts['localssh'] and not repo:
	raise error.Abort(_('--localssh requires a repository'))			raise error.Abort(_('--localssh requires a repository'))

	if opts['peer'] and opts['peer'] not in ('raw', 'ssh1', 'ssh2'):			if opts['peer'] and opts['peer'] not in ('raw', 'ssh1', 'ssh2'):
	raise error.Abort(_('invalid value for --peer'),			raise error.Abort(_('invalid value for --peer'),
	elif action == 'close':			elif action == 'close':
	peer.close()			peer.close()
	elif action == 'readavailable':			elif action == 'readavailable':
	stdin.close()			stdin.close()
	stdout.read()			stdout.read()
	stderr.read()			stderr.read()
	elif action == 'readline':			elif action == 'readline':
	stdout.readline()			stdout.readline()
				elif action == 'ereadline':
				stderr.readline()
				elif action.startswith('read '):
				count = int(action.split(' ', 1)[1])
				stdout.read(count)
				elif action.startswith('eread '):
				count = int(action.split(' ', 1)[1])
				stderr.read(count)
	else:			else:
	raise error.Abort(_('unknown action: %s') % action)			raise error.Abort(_('unknown action: %s') % action)

	if batchedcommands is not None:			if batchedcommands is not None:
	raise error.Abort(_('unclosed "batchbegin" request'))			raise error.Abort(_('unclosed "batchbegin" request'))

	if peer:			if peer:
	peer.close()			peer.close()

	if proc:			if proc:
	proc.kill()			proc.kill()