This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2791

hgweb: refactor fake file object proxy for archiving
ClosedPublic

Authored by indygreg on Mar 11 2018, 12:23 AM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG16499427f6de: hgweb: refactor fake file object proxy for archiving
Summary

Python's zip file writer operates on a file object. When doing work,
it periodically calls write(), flush(), and tell() on that object.

In WSGI contexts, the start_response function returns a write()
function. That's a function to write data, not a full file object.
So, when the archival code was first introduced in 2b03c6733efa in
2006, someone invented a proxy "tellable" type that wrapped a file
object like object and kept track of write count so it could
implement tell() and satisfy zipfile's needs.

When our archival code runs, it attempts to tell() the destination
and if that fails, converts it to a "tellable" instance. Our WSGI
application passes the "wsgirequest" instance to the archival
function. It fails the tell() test and is converted to a "tellable."
It's worth noting that "wsgirequest" implements flush(), so
"tellable" doesn't.

This hackery all seems very specific to the WSGI code. So this commit
moves the "tellable" type and the conversion of the destination file
object into the WSGI code. There's a chance some other caller may be
passing a file object like object that doesn't implement tell(). But
I doubt it.

As part of the refactor, our new type implements flush() and doesn't
implement getattr. Given the intended limited use of this type,
I want things to fail fast if there is an attempt to access attributes
because I think it is important to document which attributes are being
used for what purposes.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 11 2018, 12:23 AM
durin42 accepted this revision.Mar 12 2018, 5:12 PM
This revision is now accepted and ready to land.Mar 12 2018, 5:12 PM
Closed by commit rHG16499427f6de: hgweb: refactor fake file object proxy for archiving (authored by indygreg). · Explain WhyMar 12 2018, 5:33 PM
This revision was automatically updated to reflect the committed changes.
martinvonz added a subscriber: martinvonz.Mar 15 2018, 5:54 PM
martinvonz added inline comments.
mercurial/hgweb/request.py
294

s/file object like object/file-like object/ ?
s/append only/append-only/ ?

319

IIUC, the old code would forward the call via getattr, but the new code ignores it. Any practical consequence? I don't know what flush() on a WSGI request does.

Revision Contents
Changeset List

Diff 6927

mercurial/archival.py

data = stringio(data) data = stringio(data)
self.z.addfile(i, data) self.z.addfile(i, data)
def done(self): def done(self):
self.z.close() self.z.close()
if self.fileobj: if self.fileobj:
self.fileobj.close() self.fileobj.close()
class tellable(object):
'''provide tell method for zipfile.ZipFile when writing to http
response file object.'''
def __init__(self, fp):
self.fp = fp
self.offset = 0
def __getattr__(self, key):
return getattr(self.fp, key)
def write(self, s):
self.fp.write(s)
self.offset += len(s)
def tell(self):
return self.offset
class zipit(object): class zipit(object):
'''write archive to zip file or stream. can write uncompressed, '''write archive to zip file or stream. can write uncompressed,
or compressed with deflate.''' or compressed with deflate.'''
def __init__(self, dest, mtime, compress=True): def __init__(self, dest, mtime, compress=True):
if not isinstance(dest, bytes):
try:
dest.tell()
except (AttributeError, IOError):
dest = tellable(dest)
self.z = zipfile.ZipFile(pycompat.fsdecode(dest), r'w', self.z = zipfile.ZipFile(pycompat.fsdecode(dest), r'w',
compress and zipfile.ZIP_DEFLATED or compress and zipfile.ZIP_DEFLATED or
zipfile.ZIP_STORED) zipfile.ZIP_STORED)
# Python's zipfile module emits deprecation warnings if we try # Python's zipfile module emits deprecation warnings if we try
# to store files with a date before 1980. # to store files with a date before 1980.
epoch = 315532800 # calendar.timegm((1980, 1, 1, 0, 0, 0, 1, 1, 0)) epoch = 315532800 # calendar.timegm((1980, 1, 1, 0, 0, 0, 1, 1, 0))
if mtime < epoch: if mtime < epoch:

mercurial/hgweb/request.py

dispatchparts=dispatchparts, dispatchpath=dispatchpath, dispatchparts=dispatchparts, dispatchpath=dispatchpath,
havepathinfo='PATH_INFO' in env, havepathinfo='PATH_INFO' in env,
reponame=env.get('REPO_NAME'), reponame=env.get('REPO_NAME'),
querystring=querystring, querystring=querystring,
qsparams=qsparams, qsparams=qsparams,
headers=headers, headers=headers,
bodyfh=bodyfh) bodyfh=bodyfh)
class offsettrackingwriter(object):
"""A file object like object that is append only and tracks write count.
martinvonzUnsubmitted
Not Done

s/file object like object/file-like object/ ?
s/append only/append-only/ ?

martinvonz: s/file object like object/file-like object/ ? s/append only/append-only/ ?
Instances are bound to a callable. This callable is called with data
whenever a ``write()`` is attempted.
Instances track the amount of written data so they can answer ``tell()``
requests.
The intent of this class is to wrap the ``write()`` function returned by
a WSGI ``start_response()`` function. Since ``write()`` is a callable and
not a file object, it doesn't implement other file object methods.
"""
def __init__(self, writefn):
self._write = writefn
self._offset = 0
def write(self, s):
res = self._write(s)
# Some Python objects don't report the number of bytes written.
if res is None:
self._offset += len(s)
else:
self._offset += res
def flush(self):
pass
martinvonzUnsubmitted
Not Done

IIUC, the old code would forward the call via getattr, but the new code ignores it. Any practical consequence? I don't know what flush() on a WSGI request does.

martinvonz: IIUC, the old code would forward the call via __getattr__, but the new code ignores it. Any…
def tell(self):
return self._offset
class wsgiresponse(object): class wsgiresponse(object):
"""Represents a response to a WSGI request. """Represents a response to a WSGI request.
A response consists of a status line, headers, and a body. A response consists of a status line, headers, and a body.
Consumers must populate the ``status`` and ``headers`` fields and Consumers must populate the ``status`` and ``headers`` fields and
make a call to a ``setbody*()`` method before the response can be make a call to a ``setbody*()`` method before the response can be
issued. issued.

mercurial/hgweb/webcommands.py

ErrorResponse, ErrorResponse,
HTTP_FORBIDDEN, HTTP_FORBIDDEN,
HTTP_NOT_FOUND, HTTP_NOT_FOUND,
HTTP_OK, HTTP_OK,
get_contact, get_contact,
paritygen, paritygen,
staticfile, staticfile,
) )
from . import (
request as requestmod,
)
from .. import ( from .. import (
archival, archival,
dagop, dagop,
encoding, encoding,
error, error,
graphmod, graphmod,
pycompat, pycompat,
headers = [ headers = [
('Content-Disposition', 'attachment; filename=%s%s' % (name, extension)) ('Content-Disposition', 'attachment; filename=%s%s' % (name, extension))
] ]
if encoding: if encoding:
headers.append(('Content-Encoding', encoding)) headers.append(('Content-Encoding', encoding))
req.headers.extend(headers) req.headers.extend(headers)
req.respond(HTTP_OK, mimetype) req.respond(HTTP_OK, mimetype)
archival.archive(web.repo, req, cnode, artype, prefix=name, bodyfh = requestmod.offsettrackingwriter(req.write)
archival.archive(web.repo, bodyfh, cnode, artype, prefix=name,
matchfn=match, matchfn=match,
subrepos=web.configbool("web", "archivesubrepos")) subrepos=web.configbool("web", "archivesubrepos"))
return [] return []
@webcommand('static') @webcommand('static')
def static(web, req, tmpl): def static(web, req, tmpl):
fname = req.req.qsparams['file'] fname = req.req.qsparams['file']