( )⚙ D2769 hgweb: refactor the request draining code

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2769

hgweb: refactor the request draining code
ClosedPublic

Authored by indygreg on Mar 9 2018, 8:23 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG2cdf47e14c30: hgweb: refactor the request draining code
Summary

The previous code for draining was only invoked in a few places in
the wire protocol. Behavior wasn't consist. Furthermore, it was
difficult to reason about.

With us converting the input stream to a capped reader, it is now
safe to always drain the input stream when its size is known because
we can never overrun the input and read into the next HTTP request.
The only question is "should we?"

This commit changes the draining code so every request is examined.
Draining now kicks in for a few requests where it wouldn't before.
But I think the code is sufficiently restricted so the behavior is
safe. Possibly the most dangerous part of this code is the issuing
of Connection: close for POST and PUT requests that don't have a
Content-Length. I don't think there are any such uses in our WSGI
application, so this should be safe.

In the near future, I plan to significantly refactor the WSGI
response handling. I anticipate this code evolving a bit. So any
minor regressions around draining or connection closing behavior
might be fixed as a result of that work.

All tests pass with this change. That scares me a bit because it
means we are lacking low-level tests for the HTTP protocol.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 9 2018, 8:23 PM
indygreg updated this revision to Diff 6831.Mar 10 2018, 3:03 PM
durin42 accepted this revision.Mar 12 2018, 4:57 PM
This revision is now accepted and ready to land.Mar 12 2018, 4:57 PM
Closed by commit rHG2cdf47e14c30: hgweb: refactor the request draining code (authored by indygreg). · Explain WhyMar 12 2018, 5:32 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/hgweb/request.py (53 lines)
Mmercurial/wireprotoserver.py (18 lines)

Diff 6907

mercurial/hgweb/request.py

self.env = wsgienv self.env = wsgienv
self.form = normalize(cgi.parse(self.inp, self.form = normalize(cgi.parse(self.inp,
self.env, self.env,
keep_blank_values=1)) keep_blank_values=1))
self._start_response = start_response self._start_response = start_response
self.server_write = None self.server_write = None
self.headers = [] self.headers = []
def drain(self):
'''need to read all data from request, httplib is half-duplex'''
length = int(self.env.get('CONTENT_LENGTH') or 0)
for s in util.filechunkiter(self.inp, limit=length):
pass
def respond(self, status, type, filename=None, body=None): def respond(self, status, type, filename=None, body=None):
if not isinstance(type, str): if not isinstance(type, str):
type = pycompat.sysstr(type) type = pycompat.sysstr(type)
if self._start_response is not None: if self._start_response is not None:
self.headers.append((r'Content-Type', type)) self.headers.append((r'Content-Type', type))
if filename: if filename:
filename = (filename.rpartition('/')[-1] filename = (filename.rpartition('/')[-1]
.replace('\\', '\\\\').replace('"', '\\"')) .replace('\\', '\\\\').replace('"', '\\"'))
k in ('Date', 'ETag', 'Expires', k in ('Date', 'ETag', 'Expires',
'Cache-Control', 'Vary')] 'Cache-Control', 'Vary')]
status = statusmessage(status.code, pycompat.bytestr(status)) status = statusmessage(status.code, pycompat.bytestr(status))
elif status == 200: elif status == 200:
status = '200 Script output follows' status = '200 Script output follows'
elif isinstance(status, int): elif isinstance(status, int):
status = statusmessage(status) status = statusmessage(status)
# Various HTTP clients (notably httplib) won't read the HTTP
# response until the HTTP request has been sent in full. If servers
# (us) send a response before the HTTP request has been fully sent,
# the connection may deadlock because neither end is reading.
#
# We work around this by "draining" the request data before
# sending any response in some conditions.
drain = False
close = False
# If the client sent Expect: 100-continue, we assume it is smart
# enough to deal with the server sending a response before reading
# the request. (httplib doesn't do this.)
if self.env.get(r'HTTP_EXPECT', r'').lower() == r'100-continue':
pass
# Only tend to request methods that have bodies. Strictly speaking,
# we should sniff for a body. But this is fine for our existing
# WSGI applications.
elif self.env[r'REQUEST_METHOD'] not in (r'POST', r'PUT'):
pass
else:
# If we don't know how much data to read, there's no guarantee
# that we can drain the request responsibly. The WSGI
# specification only says that servers *should* ensure the
# input stream doesn't overrun the actual request. So there's
# no guarantee that reading until EOF won't corrupt the stream
# state.
if not isinstance(self.inp, util.cappedreader):
close = True
else:
# We /could/ only drain certain HTTP response codes. But 200
# and non-200 wire protocol responses both require draining.
# Since we have a capped reader in place for all situations
# where we drain, it is safe to read from that stream. We'll
# either do a drain or no-op if we're already at EOF.
drain = True
if close:
self.headers.append((r'Connection', r'Close'))
if drain:
assert isinstance(self.inp, util.cappedreader)
while True:
chunk = self.inp.read(32768)
if not chunk:
break
self.server_write = self._start_response( self.server_write = self._start_response(
pycompat.sysstr(status), self.headers) pycompat.sysstr(status), self.headers)
self._start_response = None self._start_response = None
self.headers = [] self.headers = []
if body is not None: if body is not None:
self.write(body) self.write(body)
self.server_write = None self.server_write = None

mercurial/wireprotoserver.py

wsgireq.respond(HTTP_OK, mediatype) wsgireq.respond(HTTP_OK, mediatype)
return gen return gen
elif isinstance(rsp, wireprototypes.pushres): elif isinstance(rsp, wireprototypes.pushres):
rsp = '%d\n%s' % (rsp.res, rsp.output) rsp = '%d\n%s' % (rsp.res, rsp.output)
wsgireq.respond(HTTP_OK, HGTYPE, body=rsp) wsgireq.respond(HTTP_OK, HGTYPE, body=rsp)
return [] return []
elif isinstance(rsp, wireprototypes.pusherr): elif isinstance(rsp, wireprototypes.pusherr):
# This is the httplib workaround documented in _handlehttperror().
wsgireq.drain()
rsp = '0\n%s\n' % rsp.res rsp = '0\n%s\n' % rsp.res
wsgireq.respond(HTTP_OK, HGTYPE, body=rsp) wsgireq.respond(HTTP_OK, HGTYPE, body=rsp)
return [] return []
elif isinstance(rsp, wireprototypes.ooberror): elif isinstance(rsp, wireprototypes.ooberror):
rsp = rsp.message rsp = rsp.message
wsgireq.respond(HTTP_OK, HGERRTYPE, body=rsp) wsgireq.respond(HTTP_OK, HGERRTYPE, body=rsp)
return [] return []
raise error.ProgrammingError('hgweb.protocol internal failure', rsp) raise error.ProgrammingError('hgweb.protocol internal failure', rsp)
def _handlehttperror(e, wsgireq, req): def _handlehttperror(e, wsgireq, req):
"""Called when an ErrorResponse is raised during HTTP request processing.""" """Called when an ErrorResponse is raised during HTTP request processing."""
# Clients using Python's httplib are stateful: the HTTP client
# won't process an HTTP response until all request data is
# sent to the server. The intent of this code is to ensure
# we always read HTTP request data from the client, thus
# ensuring httplib transitions to a state that allows it to read
# the HTTP response. In other words, it helps prevent deadlocks
# on clients using httplib.
if (req.method == 'POST' and
# But not if Expect: 100-continue is being used.
(req.headers.get('Expect', '').lower() != '100-continue')):
wsgireq.drain()
else:
wsgireq.headers.append((r'Connection', r'Close'))
# TODO This response body assumes the failed command was # TODO This response body assumes the failed command was
# "unbundle." That assumption is not always valid. # "unbundle." That assumption is not always valid.
wsgireq.respond(e, HGTYPE, body='0\n%s\n' % pycompat.bytestr(e)) wsgireq.respond(e, HGTYPE, body='0\n%s\n' % pycompat.bytestr(e))
return '' return ''
def _sshv1respondbytes(fout, value): def _sshv1respondbytes(fout, value):
"""Send a bytes response for protocol version 1.""" """Send a bytes response for protocol version 1."""