This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2833

tests: teach get-with-headers.py to ignore dynamic headers
AbandonedPublic

Authored by indygreg on Mar 12 2018, 8:07 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

The Date and Server HTTP headers are dynamic by nature. They are
also present on every response and are mostly advisory.

It isn't important for us to test for the presence of these
headers. In fact, all tests currently glob over the entirety
of their values.

Having to constantly glob these headers feels like more trouble
than it is worth. Let's teach get-with-headers.py to hide them
by default.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Mar 12 2018, 8:07 PM
mharbison72 added a subscriber: mharbison72.Mar 12 2018, 9:20 PM

I’ve got a patch locally that will pattern substitute away the date. It’s been useful for LFS, because I’ve been debug dumping the headers and JSON on the client side. I’ll try to submit that tonight.

indygreg abandoned this revision.Mar 13 2018, 12:46 PM

I like Matt's patch and will do something more deterministic.

Revision Contents
Changeset List

Diff 6978

tests/get-with-headers.py

parser.add_argument('--headeronly', action='store_true') parser.add_argument('--headeronly', action='store_true')
parser.add_argument('--json', action='store_true') parser.add_argument('--json', action='store_true')
parser.add_argument('--hgproto') parser.add_argument('--hgproto')
parser.add_argument('--requestheader', nargs='*', default=[], parser.add_argument('--requestheader', nargs='*', default=[],
help='Send an additional HTTP request header. Argument ' help='Send an additional HTTP request header. Argument '
'value is <header>=<value>') 'value is <header>=<value>')
parser.add_argument('--bodyfile', parser.add_argument('--bodyfile',
help='Write HTTP response body to a file') help='Write HTTP response body to a file')
parser.add_argument('--showdynamicheaders', action='store_true',
help='Show dynamic headers that are hidden by default')
parser.add_argument('host') parser.add_argument('host')
parser.add_argument('path') parser.add_argument('path')
parser.add_argument('show', nargs='*') parser.add_argument('show', nargs='*')
args = parser.parse_args() args = parser.parse_args()
twice = args.twice twice = args.twice
headeronly = args.headeronly headeronly = args.headeronly
conn = httplib.HTTPConnection(host) conn = httplib.HTTPConnection(host)
conn.request("GET", '/' + path, None, headers) conn.request("GET", '/' + path, None, headers)
response = conn.getresponse() response = conn.getresponse()
stdout.write(b'%d %s\n' % (response.status, stdout.write(b'%d %s\n' % (response.status,
response.reason.encode('ascii'))) response.reason.encode('ascii')))
if show[:1] == ['-']: if show[:1] == ['-']:
show = sorted(h for h, v in response.getheaders() show = sorted(h for h, v in response.getheaders()
if h.lower() not in show) if h.lower() not in show)
if not args.showdynamicheaders:
show = [h for h in show if h.lower() not in ('date', 'server')]
for h in [h.lower() for h in show]: for h in [h.lower() for h in show]:
if response.getheader(h, None) is not None: if response.getheader(h, None) is not None:
stdout.write(b"%s: %s\n" % (h.encode('ascii'), stdout.write(b"%s: %s\n" % (h.encode('ascii'),
response.getheader(h).encode('ascii'))) response.getheader(h).encode('ascii')))
if not headeronly: if not headeronly:
stdout.write(b'\n') stdout.write(b'\n')
data = response.read() data = response.read()

tests/test-archive.t

check http return codes check http return codes
$ test_archtype gz tar.gz tar.bz2 zip $ test_archtype gz tar.gz tar.bz2 zip
% gz allowed should give 200 % gz allowed should give 200
200 Script output follows 200 Script output follows
content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.gz content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.gz
content-type: application/x-gzip content-type: application/x-gzip
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=408, sha1=8fa06531bddecc365a9f5edb0f88b65974bfe505 body: size=408, sha1=8fa06531bddecc365a9f5edb0f88b65974bfe505
% tar.bz2 and zip disallowed should both give 403 % tar.bz2 and zip disallowed should both give 403
403 Archive type not allowed: bz2 403 Archive type not allowed: bz2
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352 body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352
403 Archive type not allowed: zip 403 Archive type not allowed: zip
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7 body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7
$ test_archtype bz2 tar.bz2 zip tar.gz $ test_archtype bz2 tar.bz2 zip tar.gz
% bz2 allowed should give 200 % bz2 allowed should give 200
200 Script output follows 200 Script output follows
content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.bz2 content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.bz2
content-type: application/x-bzip2 content-type: application/x-bzip2
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=426, sha1=8d87f5aba6e14f1bfea6c232985982c278b2fb0b body: size=426, sha1=8d87f5aba6e14f1bfea6c232985982c278b2fb0b
% zip and tar.gz disallowed should both give 403 % zip and tar.gz disallowed should both give 403
403 Archive type not allowed: zip 403 Archive type not allowed: zip
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7 body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7
403 Archive type not allowed: gz 403 Archive type not allowed: gz
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734 body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734
$ test_archtype zip zip tar.gz tar.bz2 $ test_archtype zip zip tar.gz tar.bz2
% zip allowed should give 200 % zip allowed should give 200
200 Script output follows 200 Script output follows
content-disposition: attachment; filename=test-archive-1701ef1f1510.zip content-disposition: attachment; filename=test-archive-1701ef1f1510.zip
content-type: application/zip content-type: application/zip
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1377, sha1=677b14d3d048778d5eb5552c14a67e6192068650 body: size=1377, sha1=677b14d3d048778d5eb5552c14a67e6192068650
% tar.gz and tar.bz2 disallowed should both give 403 % tar.gz and tar.bz2 disallowed should both give 403
403 Archive type not allowed: gz 403 Archive type not allowed: gz
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734 body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734
403 Archive type not allowed: bz2 403 Archive type not allowed: bz2
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352 body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352
check http return codes (with deprecated option) check http return codes (with deprecated option)
$ test_archtype_deprecated gz tar.gz tar.bz2 zip $ test_archtype_deprecated gz tar.gz tar.bz2 zip
% gz allowed should give 200 % gz allowed should give 200
200 Script output follows 200 Script output follows
content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.gz content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.gz
content-type: application/x-gzip content-type: application/x-gzip
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=408, sha1=8fa06531bddecc365a9f5edb0f88b65974bfe505 body: size=408, sha1=8fa06531bddecc365a9f5edb0f88b65974bfe505
% tar.bz2 and zip disallowed should both give 403 % tar.bz2 and zip disallowed should both give 403
403 Archive type not allowed: bz2 403 Archive type not allowed: bz2
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352 body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352
403 Archive type not allowed: zip 403 Archive type not allowed: zip
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7 body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7
$ test_archtype_deprecated bz2 tar.bz2 zip tar.gz $ test_archtype_deprecated bz2 tar.bz2 zip tar.gz
% bz2 allowed should give 200 % bz2 allowed should give 200
200 Script output follows 200 Script output follows
content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.bz2 content-disposition: attachment; filename=test-archive-1701ef1f1510.tar.bz2
content-type: application/x-bzip2 content-type: application/x-bzip2
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=426, sha1=8d87f5aba6e14f1bfea6c232985982c278b2fb0b body: size=426, sha1=8d87f5aba6e14f1bfea6c232985982c278b2fb0b
% zip and tar.gz disallowed should both give 403 % zip and tar.gz disallowed should both give 403
403 Archive type not allowed: zip 403 Archive type not allowed: zip
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7 body: size=1451, sha1=cbfa5574b337348bfd0564cc534474d002e7d6c7
403 Archive type not allowed: gz 403 Archive type not allowed: gz
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734 body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734
$ test_archtype_deprecated zip zip tar.gz tar.bz2 $ test_archtype_deprecated zip zip tar.gz tar.bz2
% zip allowed should give 200 % zip allowed should give 200
200 Script output follows 200 Script output follows
content-disposition: attachment; filename=test-archive-1701ef1f1510.zip content-disposition: attachment; filename=test-archive-1701ef1f1510.zip
content-type: application/zip content-type: application/zip
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1377, sha1=677b14d3d048778d5eb5552c14a67e6192068650 body: size=1377, sha1=677b14d3d048778d5eb5552c14a67e6192068650
% tar.gz and tar.bz2 disallowed should both give 403 % tar.gz and tar.bz2 disallowed should both give 403
403 Archive type not allowed: gz 403 Archive type not allowed: gz
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734 body: size=1450, sha1=71f0b12d59f85fdcfe8ff493e2dc66863f2f7734
403 Archive type not allowed: bz2 403 Archive type not allowed: bz2
content-type: text/html; charset=ascii content-type: text/html; charset=ascii
date: * (glob)
etag: W/"*" (glob) etag: W/"*" (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352 body: size=1451, sha1=4c5cf0f574446c44feb7f88f4e0e2a56bd92c352
$ echo "allow_archive = gz bz2 zip" >> .hg/hgrc $ echo "allow_archive = gz bz2 zip" >> .hg/hgrc
$ hg serve -p $HGPORT -d --pid-file=hg.pid -E errors.log $ hg serve -p $HGPORT -d --pid-file=hg.pid -E errors.log
$ cat hg.pid >> $DAEMON_PIDS $ cat hg.pid >> $DAEMON_PIDS

tests/test-hgweb-commands.t

lookup branchmap pushkey known getbundle unbundlehash batch changegroupsubset $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=$BUNDLE2_COMPRESSIONS$ lookup branchmap pushkey known getbundle unbundlehash batch changegroupsubset $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=$BUNDLE2_COMPRESSIONS$
wire protocol command to wrong base URL wire protocol command to wrong base URL
$ get-with-headers.py $LOCALIP:$HGPORT 'foo?cmd=capabilities' - $ get-with-headers.py $LOCALIP:$HGPORT 'foo?cmd=capabilities' -
404 Not Found 404 Not Found
content-length: 12 content-length: 12
content-type: application/mercurial-0.1 content-type: application/mercurial-0.1
date: * (glob)
server: * (glob)
0 0
Not Found Not Found
[1] [1]
heads heads
$ get-with-headers.py $LOCALIP:$HGPORT '?cmd=heads' $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=heads'

tests/test-http-protocol.t

$ hg serve -R server -p $HGPORT -d --pid-file hg.pid $ hg serve -R server -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS $ cat hg.pid > $DAEMON_PIDS
Server should send application/mercurial-0.1 to clients if no Accept is used Server should send application/mercurial-0.1 to clients if no Accept is used
$ get-with-headers.py --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' - $ get-with-headers.py --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' -
200 Script output follows 200 Script output follows
content-type: application/mercurial-0.1 content-type: application/mercurial-0.1
date: * (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
Server should send application/mercurial-0.1 when client says it wants it Server should send application/mercurial-0.1 when client says it wants it
$ get-with-headers.py --hgproto '0.1' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' - $ get-with-headers.py --hgproto '0.1' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' -
200 Script output follows 200 Script output follows
content-type: application/mercurial-0.1 content-type: application/mercurial-0.1
date: * (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
Server should send application/mercurial-0.2 when client says it wants it Server should send application/mercurial-0.2 when client says it wants it
$ get-with-headers.py --hgproto '0.2' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' - $ get-with-headers.py --hgproto '0.2' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' -
200 Script output follows 200 Script output follows
content-type: application/mercurial-0.2 content-type: application/mercurial-0.2
date: * (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
$ get-with-headers.py --hgproto '0.1 0.2' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' - $ get-with-headers.py --hgproto '0.1 0.2' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' -
200 Script output follows 200 Script output follows
content-type: application/mercurial-0.2 content-type: application/mercurial-0.2
date: * (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
Requesting a compression format that server doesn't support results will fall back to 0.1 Requesting a compression format that server doesn't support results will fall back to 0.1
$ get-with-headers.py --hgproto '0.2 comp=aa' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' - $ get-with-headers.py --hgproto '0.2 comp=aa' --headeronly $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' -
200 Script output follows 200 Script output follows
content-type: application/mercurial-0.1 content-type: application/mercurial-0.1
date: * (glob)
server: * (glob)
transfer-encoding: chunked transfer-encoding: chunked
#if zstd #if zstd
zstd is used if available zstd is used if available
$ get-with-headers.py --hgproto '0.2 comp=zstd' $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' > resp $ get-with-headers.py --hgproto '0.2 comp=zstd' $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' > resp
$ f --size --hexdump --bytes 36 --sha1 resp $ f --size --hexdump --bytes 36 --sha1 resp
resp: size=248, sha1=4d8d8f87fb82bd542ce52881fdc94f850748 resp: size=248, sha1=4d8d8f87fb82bd542ce52881fdc94f850748
0000: 32 30 30 20 53 63 72 69 70 74 20 6f 75 74 70 75 |200 Script outpu| 0000: 32 30 30 20 53 63 72 69 70 74 20 6f 75 74 70 75 |200 Script outpu|
0010: 74 20 66 6f 6c 6c 6f 77 73 0a 0a 04 7a 73 74 64 |t follows...zstd| 0010: 74 20 66 6f 6c 6c 6f 77 73 0a 0a 04 7a 73 74 64 |t follows...zstd|
0020: 28 b5 2f fd |(./.| 0020: 28 b5 2f fd |(./.|
#endif #endif
application/mercurial-0.2 is not yet used on non-streaming responses application/mercurial-0.2 is not yet used on non-streaming responses
$ get-with-headers.py --hgproto '0.2' $LOCALIP:$HGPORT '?cmd=heads' - $ get-with-headers.py --hgproto '0.2' $LOCALIP:$HGPORT '?cmd=heads' -
200 Script output follows 200 Script output follows
content-length: 41 content-length: 41
content-type: application/mercurial-0.1 content-type: application/mercurial-0.1
date: * (glob)
server: * (glob)
e93700bd72895c5addab234c56d4024b487a362f e93700bd72895c5addab234c56d4024b487a362f
Now test protocol preference usage Now test protocol preference usage
$ killdaemons.py $ killdaemons.py
$ hg serve --config server.compressionengines=none,zlib -R server -p $HGPORT -d --pid-file hg.pid $ hg serve --config server.compressionengines=none,zlib -R server -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS $ cat hg.pid > $DAEMON_PIDS