( )⚙ D2734 hgweb: parse WSGI request into a data structure

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2734

hgweb: parse WSGI request into a data structure
ClosedPublic

Authored by indygreg on Mar 8 2018, 8:06 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG69b2d0900cd7: hgweb: parse WSGI request into a data structure
rHG3e13a21ce6b5: hgweb: parse WSGI request into a data structure
Summary

Currently, our WSGI applications (hgweb_mod and hgwebdir_mod) process
the raw WSGI request instance themselves. This means they have to
talk in terms of system strings. And they need to know details
about what's in the WSGI request. And in the case of hgweb_mod, it
is doing some very funky things with URL parsing to impact
dispatching. The code is difficult to read and maintain.

This commit introduces parsing of the WSGI request into a higher-level
and easier-to-reason-about data structure.

To prove it works, we hook it up to hgweb_mod and use it for populating
the relative URL on the request instance.

We hold off on using it in more places because the logic in hgweb_mod
is crazy and I don't want to involve those changes with review of
the parsing code.

The URL construction code has variations that use the HTTP: Host header
(the canonical WSGI way of reconstructing the URL) and with the use
of SERVER_NAME. We need to differentiate because hgweb is currently
using SERVER_NAME for URL construction.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 8 2018, 8:06 PM
durin42 accepted this revision.Mar 9 2018, 1:59 PM
This revision is now accepted and ready to land.Mar 9 2018, 1:59 PM
Closed by commit rHG3e13a21ce6b5: hgweb: parse WSGI request into a data structure (authored by indygreg). · Explain WhyMar 9 2018, 2:29 PM
This revision was automatically updated to reflect the committed changes.
yuja added a subscriber: yuja.Mar 10 2018, 4:23 AM
yuja added inline comments.
mercurial/hgweb/hgwebdir_mod.py
232

We'll need something to silence pyflakes saying "local variable 'req' is
assigned to but never used."

mercurial/hgweb/request.py
125

Missed assignment fullurl = ?

pulkit added a subscriber: pulkit.Mar 10 2018, 4:45 AM
pulkit added inline comments.
mercurial/hgweb/request.py
14

Chunk from your wip?

indygreg added inline comments.Mar 10 2018, 1:18 PM
mercurial/hgweb/request.py
14

Yes. But there is a TODO below to track enabling it. I fully intend to enable this once we can. That will likely require doing away with the REPO_NAME hack in hgwebdir. That will likely happen towards the end of this series. I'm a few hours away from getting there I think :)

indygreg marked 4 inline comments as done.Mar 10 2018, 1:22 PM

I amended hg-committed with the fixes for the issues @yuja found.

Revision Contents
Changeset List

Diff 6778

mercurial/hgweb/hgweb_mod.py

""" """
with self._obtainrepo() as repo: with self._obtainrepo() as repo:
profile = repo.ui.configbool('profiling', 'enabled') profile = repo.ui.configbool('profiling', 'enabled')
with profiling.profile(repo.ui, enabled=profile): with profiling.profile(repo.ui, enabled=profile):
for r in self._runwsgi(wsgireq, repo): for r in self._runwsgi(wsgireq, repo):
yield r yield r
def _runwsgi(self, wsgireq, repo): def _runwsgi(self, wsgireq, repo):
req = requestmod.parserequestfromenv(wsgireq.env)
rctx = requestcontext(self, repo) rctx = requestcontext(self, repo)
# This state is global across all threads. # This state is global across all threads.
encoding.encoding = rctx.config('web', 'encoding') encoding.encoding = rctx.config('web', 'encoding')
rctx.repo.ui.environ = wsgireq.env rctx.repo.ui.environ = wsgireq.env
if rctx.csp: if rctx.csp:
# hgwebdir may have added CSP header. Since we generate our own, # hgwebdir may have added CSP header. Since we generate our own,
# replace it. # replace it.
wsgireq.headers = [h for h in wsgireq.headers wsgireq.headers = [h for h in wsgireq.headers
if h[0] != 'Content-Security-Policy'] if h[0] != 'Content-Security-Policy']
wsgireq.headers.append(('Content-Security-Policy', rctx.csp)) wsgireq.headers.append(('Content-Security-Policy', rctx.csp))
# work with CGI variables to create coherent structure wsgireq.url = pycompat.sysstr(req.apppath)
# use SCRIPT_NAME, PATH_INFO and QUERY_STRING as well as our REPO_NAME
wsgireq.url = wsgireq.env[r'SCRIPT_NAME']
if not wsgireq.url.endswith(r'/'):
wsgireq.url += r'/'
if wsgireq.env.get('REPO_NAME'):
wsgireq.url += wsgireq.env[r'REPO_NAME'] + r'/'
if r'PATH_INFO' in wsgireq.env: if r'PATH_INFO' in wsgireq.env:
parts = wsgireq.env[r'PATH_INFO'].strip(r'/').split(r'/') parts = wsgireq.env[r'PATH_INFO'].strip(r'/').split(r'/')
repo_parts = wsgireq.env.get(r'REPO_NAME', r'').split(r'/') repo_parts = wsgireq.env.get(r'REPO_NAME', r'').split(r'/')
if parts[:len(repo_parts)] == repo_parts: if parts[:len(repo_parts)] == repo_parts:
parts = parts[len(repo_parts):] parts = parts[len(repo_parts):]
query = r'/'.join(parts) query = r'/'.join(parts)
else: else:

mercurial/hgweb/hgwebdir_mod.py

def run_wsgi(self, wsgireq): def run_wsgi(self, wsgireq):
profile = self.ui.configbool('profiling', 'enabled') profile = self.ui.configbool('profiling', 'enabled')
with profiling.profile(self.ui, enabled=profile): with profiling.profile(self.ui, enabled=profile):
for r in self._runwsgi(wsgireq): for r in self._runwsgi(wsgireq):
yield r yield r
def _runwsgi(self, wsgireq): def _runwsgi(self, wsgireq):
req = requestmod.parserequestfromenv(wsgireq.env)
yujaUnsubmitted
Done

We'll need something to silence pyflakes saying "local variable 'req' is
assigned to but never used."

yuja: We'll need something to silence pyflakes saying "local variable 'req' is assigned to but never…
try: try:
self.refresh() self.refresh()
csp, nonce = cspvalues(self.ui) csp, nonce = cspvalues(self.ui)
if csp: if csp:
wsgireq.headers.append(('Content-Security-Policy', csp)) wsgireq.headers.append(('Content-Security-Policy', csp))
virtual = wsgireq.env.get("PATH_INFO", "").strip('/') virtual = wsgireq.env.get("PATH_INFO", "").strip('/')

mercurial/hgweb/request.py

# hgweb/request.py - An http request from either CGI or the standalone server. # hgweb/request.py - An http request from either CGI or the standalone server.
# #
# Copyright 21 May 2005 - (c) 2005 Jake Edge <jake@edge2.net> # Copyright 21 May 2005 - (c) 2005 Jake Edge <jake@edge2.net>
# Copyright 2005, 2006 Matt Mackall <mpm@selenic.com> # Copyright 2005, 2006 Matt Mackall <mpm@selenic.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import cgi import cgi
import errno import errno
import socket import socket
#import wsgiref.validate
pulkitUnsubmitted
Done

Chunk from your wip?

pulkit: Chunk from your wip?
indygregAuthorUnsubmitted
Done

Yes. But there is a TODO below to track enabling it. I fully intend to enable this once we can. That will likely require doing away with the REPO_NAME hack in hgwebdir. That will likely happen towards the end of this series. I'm a few hours away from getting there I think :)

indygreg: Yes. But there is a TODO below to track enabling it. I fully intend to enable this once we can.
from .common import ( from .common import (
ErrorResponse, ErrorResponse,
HTTP_NOT_MODIFIED, HTTP_NOT_MODIFIED,
statusmessage, statusmessage,
) )
from ..thirdparty import (
attr,
)
from .. import ( from .. import (
pycompat, pycompat,
util, util,
) )
shortcuts = { shortcuts = {
'cl': [('cmd', ['changelog']), ('rev', None)], 'cl': [('cmd', ['changelog']), ('rev', None)],
'sl': [('cmd', ['shortlog']), ('rev', None)], 'sl': [('cmd', ['shortlog']), ('rev', None)],
del form[k] del form[k]
# And strip the values # And strip the values
bytesform = {} bytesform = {}
for k, v in form.iteritems(): for k, v in form.iteritems():
bytesform[pycompat.bytesurl(k)] = [ bytesform[pycompat.bytesurl(k)] = [
pycompat.bytesurl(i.strip()) for i in v] pycompat.bytesurl(i.strip()) for i in v]
return bytesform return bytesform
@attr.s(frozen=True)
class parsedrequest(object):
"""Represents a parsed WSGI request / static HTTP request parameters."""
# Full URL for this request.
url = attr.ib()
# URL without any path components. Just <proto>://<host><port>.
baseurl = attr.ib()
# Advertised URL. Like ``url`` and ``baseurl`` but uses SERVER_NAME instead
# of HTTP: Host header for hostname. This is likely what clients used.
advertisedurl = attr.ib()
advertisedbaseurl = attr.ib()
# WSGI application path.
apppath = attr.ib()
# List of path parts to be used for dispatch.
dispatchparts = attr.ib()
# URL path component (no query string) used for dispatch.
dispatchpath = attr.ib()
# Raw query string (part after "?" in URL).
querystring = attr.ib()
def parserequestfromenv(env):
"""Parse URL components from environment variables.
WSGI defines request attributes via environment variables. This function
parses the environment variables into a data structure.
"""
# PEP-0333 defines the WSGI spec and is a useful reference for this code.
# We first validate that the incoming object conforms with the WSGI spec.
# We only want to be dealing with spec-conforming WSGI implementations.
# TODO enable this once we fix internal violations.
#wsgiref.validate.check_environ(env)
# PEP-0333 states that environment keys and values are native strings
# (bytes on Python 2 and str on Python 3). The code points for the Unicode
# strings on Python 3 must be between \00000-\000FF. We deal with bytes
# in Mercurial, so mass convert string keys and values to bytes.
if pycompat.ispy3:
env = {k.encode('latin-1'): v for k, v in env.iteritems()}
env = {k: v.encode('latin-1') if isinstance(v, str) else v
for k, v in env.iteritems()}
# https://www.python.org/dev/peps/pep-0333/#environ-variables defines
# the environment variables.
# https://www.python.org/dev/peps/pep-0333/#url-reconstruction defines
# how URLs are reconstructed.
fullurl = env['wsgi.url_scheme'] + '://'
advertisedfullurl = fullurl
def addport(s):
if env['wsgi.url_scheme'] == 'https':
if env['SERVER_PORT'] != '443':
s += ':' + env['SERVER_PORT']
else:
if env['SERVER_PORT'] != '80':
s += ':' + env['SERVER_PORT']
return s
if env.get('HTTP_HOST'):
fullurl += env['HTTP_HOST']
else:
fullurl += env['SERVER_NAME']
addport(fullurl)
yujaUnsubmitted
Done

Missed assignment fullurl = ?

yuja: Missed assignment `fullurl =` ?
advertisedfullurl += env['SERVER_NAME']
advertisedfullurl = addport(advertisedfullurl)
baseurl = fullurl
advertisedbaseurl = advertisedfullurl
fullurl += util.urlreq.quote(env.get('SCRIPT_NAME', ''))
advertisedfullurl += util.urlreq.quote(env.get('SCRIPT_NAME', ''))
fullurl += util.urlreq.quote(env.get('PATH_INFO', ''))
advertisedfullurl += util.urlreq.quote(env.get('PATH_INFO', ''))
if env.get('QUERY_STRING'):
fullurl += '?' + env['QUERY_STRING']
advertisedfullurl += '?' + env['QUERY_STRING']
# When dispatching requests, we look at the URL components (PATH_INFO
# and QUERY_STRING) after the application root (SCRIPT_NAME). But hgwebdir
# has the concept of "virtual" repositories. This is defined via REPO_NAME.
# If REPO_NAME is defined, we append it to SCRIPT_NAME to form a new app
# root. We also exclude its path components from PATH_INFO when resolving
# the dispatch path.
# TODO the use of trailing slashes in apppath is arguably wrong. We need it
# to appease low-level parts of hgweb_mod for now.
apppath = env['SCRIPT_NAME']
if not apppath.endswith('/'):
apppath += '/'
if env.get('REPO_NAME'):
apppath += env.get('REPO_NAME') + '/'
if 'PATH_INFO' in env:
dispatchparts = env['PATH_INFO'].strip('/').split('/')
# Strip out repo parts.
repoparts = env.get('REPO_NAME', '').split('/')
if dispatchparts[:len(repoparts)] == repoparts:
dispatchparts = dispatchparts[len(repoparts):]
else:
dispatchparts = []
dispatchpath = '/'.join(dispatchparts)
querystring = env.get('QUERY_STRING', '')
return parsedrequest(url=fullurl, baseurl=baseurl,
advertisedurl=advertisedfullurl,
advertisedbaseurl=advertisedbaseurl,
apppath=apppath,
dispatchparts=dispatchparts, dispatchpath=dispatchpath,
querystring=querystring)
class wsgirequest(object): class wsgirequest(object):
"""Higher-level API for a WSGI request. """Higher-level API for a WSGI request.
WSGI applications are invoked with 2 arguments. They are used to WSGI applications are invoked with 2 arguments. They are used to
instantiate instances of this class, which provides higher-level APIs instantiate instances of this class, which provides higher-level APIs
for obtaining request parameters, writing HTTP output, etc. for obtaining request parameters, writing HTTP output, etc.
""" """
def __init__(self, wsgienv, start_response): def __init__(self, wsgienv, start_response):