( )⚙ D2726 debugcommands: support connecting to HTTP peers

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2726

debugcommands: support connecting to HTTP peers
ClosedPublic

Authored by indygreg on Mar 8 2018, 12:31 AM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGfc8939825632: debugcommands: support connecting to HTTP peers
Summary

Now that we have the plumbing for logging socket activity, let's hook
it up to hg debugwireproto so we can collect low-level activity on
sockets.

The new code is a bit incomplete. But it is better than nothing:
hg debugwireproto is still heavily evolving.

The added test demonstrates some interesting behavior. For example,
we're calling socket.makefile() and doing I/O on that. TIL. We're also
sending an "Accept-Encoding: identity" request header.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Mar 8 2018, 12:31 AM
durin42 accepted this revision.Mar 19 2018, 9:36 PM
This revision is now accepted and ready to land.Mar 19 2018, 9:36 PM
Closed by commit rHGfc8939825632: debugcommands: support connecting to HTTP peers (authored by indygreg). · Explain WhyMar 19 2018, 9:40 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/debugcommands.py (58 lines)
Mtests/test-http-protocol.t (65 lines)

Diff 6721

mercurial/debugcommands.py

encoding, encoding,
error, error,
exchange, exchange,
extensions, extensions,
filemerge, filemerge,
fileset, fileset,
formatter, formatter,
hg, hg,
httppeer,
localrepo, localrepo,
lock as lockmod, lock as lockmod,
logcmdutil, logcmdutil,
merge as mergemod, merge as mergemod,
obsolete, obsolete,
obsutil, obsutil,
phases, phases,
policy, policy,
yield activeaction, blocklines yield activeaction, blocklines
@command('debugwireproto', @command('debugwireproto',
[ [
('', 'localssh', False, _('start an SSH server for this repo')), ('', 'localssh', False, _('start an SSH server for this repo')),
('', 'peer', '', _('construct a specific version of the peer')), ('', 'peer', '', _('construct a specific version of the peer')),
('', 'noreadstderr', False, _('do not read from stderr of the remote')), ('', 'noreadstderr', False, _('do not read from stderr of the remote')),
] + cmdutil.remoteopts, ] + cmdutil.remoteopts,
_('[REPO]'), _('[PATH]'),
optionalrepo=True) optionalrepo=True)
def debugwireproto(ui, repo, **opts): def debugwireproto(ui, repo, path=None, **opts):
"""send wire protocol commands to a server """send wire protocol commands to a server
This command can be used to issue wire protocol commands to remote This command can be used to issue wire protocol commands to remote
peers and to debug the raw data being exchanged. peers and to debug the raw data being exchanged.
``--localssh`` will start an SSH server against the current repository ``--localssh`` will start an SSH server against the current repository
and connect to that. By default, the connection will perform a handshake and connect to that. By default, the connection will perform a handshake
and establish an appropriate peer instance. and establish an appropriate peer instance.
if opts['localssh'] and not repo: if opts['localssh'] and not repo:
raise error.Abort(_('--localssh requires a repository')) raise error.Abort(_('--localssh requires a repository'))
if opts['peer'] and opts['peer'] not in ('raw', 'ssh1', 'ssh2'): if opts['peer'] and opts['peer'] not in ('raw', 'ssh1', 'ssh2'):
raise error.Abort(_('invalid value for --peer'), raise error.Abort(_('invalid value for --peer'),
hint=_('valid values are "raw", "ssh1", and "ssh2"')) hint=_('valid values are "raw", "ssh1", and "ssh2"'))
if path and opts['localssh']:
raise error.Abort(_('cannot specify --localssh with an explicit '
'path'))
if ui.interactive(): if ui.interactive():
ui.write(_('(waiting for commands on stdin)\n')) ui.write(_('(waiting for commands on stdin)\n'))
blocks = list(_parsewirelangblocks(ui.fin)) blocks = list(_parsewirelangblocks(ui.fin))
proc = None proc = None
stdin = None
stdout = None
stderr = None
if opts['localssh']: if opts['localssh']:
# We start the SSH server in its own process so there is process # We start the SSH server in its own process so there is process
# separation. This prevents a whole class of potential bugs around # separation. This prevents a whole class of potential bugs around
# shared state from interfering with server operation. # shared state from interfering with server operation.
args = util.hgcmd() + [ args = util.hgcmd() + [
'-R', repo.root, '-R', repo.root,
'debugserve', '--sshstdio', 'debugserve', '--sshstdio',
elif opts['peer'] == 'raw': elif opts['peer'] == 'raw':
ui.write(_('using raw connection to peer\n')) ui.write(_('using raw connection to peer\n'))
peer = None peer = None
else: else:
ui.write(_('creating ssh peer from handshake results\n')) ui.write(_('creating ssh peer from handshake results\n'))
peer = sshpeer.makepeer(ui, url, proc, stdin, stdout, stderr, peer = sshpeer.makepeer(ui, url, proc, stdin, stdout, stderr,
autoreadstderr=autoreadstderr) autoreadstderr=autoreadstderr)
elif path:
# We bypass hg.peer() so we can proxy the sockets.
# TODO consider not doing this because we skip
# ``hg.wirepeersetupfuncs`` and potentially other useful functionality.
u = util.url(path)
if u.scheme != 'http':
raise error.Abort(_('only http:// paths are currently supported'))
if opts['peer']:
raise error.Abort(_('--peer is not supported with HTTP peers'))
url, authinfo = u.authinfo()
openerargs = {}
# Turn pipes/sockets into observers so we can log I/O.
if ui.verbose:
openerargs = {
r'loggingfh': ui,
r'loggingname': b's',
r'loggingopts': {
r'logdata': True,
},
}
opener = urlmod.opener(ui, authinfo, **openerargs)
peer = httppeer.httppeer(ui, path, url, opener)
peer._fetchcaps()
# We /could/ populate stdin/stdout with sock.makefile()...
else: else:
raise error.Abort(_('only --localssh is currently supported')) raise error.Abort(_('unsupported connection configuration'))
batchedcommands = None batchedcommands = None
# Now perform actions based on the parsed wire language instructions. # Now perform actions based on the parsed wire language instructions.
for action, lines in blocks: for action, lines in blocks:
if action in ('raw', 'raw+'): if action in ('raw', 'raw+'):
if not stdin:
raise error.Abort(_('cannot call raw/raw+ on this peer'))
# Concatenate the data together. # Concatenate the data together.
data = ''.join(l.lstrip() for l in lines) data = ''.join(l.lstrip() for l in lines)
data = util.unescapestr(data) data = util.unescapestr(data)
stdin.write(data) stdin.write(data)
if action == 'raw+': if action == 'raw+':
stdin.flush() stdin.flush()
elif action == 'flush': elif action == 'flush':
if not stdin:
raise error.Abort(_('cannot call flush on this peer'))
stdin.flush() stdin.flush()
elif action.startswith('command'): elif action.startswith('command'):
if not peer: if not peer:
raise error.Abort(_('cannot send commands unless peer instance ' raise error.Abort(_('cannot send commands unless peer instance '
'is available')) 'is available'))
command = action.split(' ', 1)[1] command = action.split(' ', 1)[1]
len(batchedcommands)) len(batchedcommands))
for i, chunk in enumerate(peer._submitbatch(batchedcommands)): for i, chunk in enumerate(peer._submitbatch(batchedcommands)):
ui.status(_('response #%d: %s\n') % (i, util.escapedata(chunk))) ui.status(_('response #%d: %s\n') % (i, util.escapedata(chunk)))
batchedcommands = None batchedcommands = None
elif action == 'close': elif action == 'close':
peer.close() peer.close()
elif action == 'readavailable': elif action == 'readavailable':
if not stdout or not stderr:
raise error.Abort(_('readavailable not available on this peer'))
fds = [stdout.fileno(), stderr.fileno()] fds = [stdout.fileno(), stderr.fileno()]
try: try:
act = util.poll(fds) act = util.poll(fds)
except NotImplementedError: except NotImplementedError:
# non supported yet case, assume all have data. # non supported yet case, assume all have data.
act = fds act = fds
if stdout.fileno() in act: if stdout.fileno() in act:
util.readpipe(stdout) util.readpipe(stdout)
if stderr.fileno() in act: if stderr.fileno() in act:
util.readpipe(stderr) util.readpipe(stderr)
elif action == 'readline': elif action == 'readline':
if not stdout:
raise error.Abort(_('readline not available on this peer'))
stdout.readline() stdout.readline()
elif action == 'ereadline': elif action == 'ereadline':
if not stderr:
raise error.Abort(_('ereadline not available on this peer'))
stderr.readline() stderr.readline()
elif action.startswith('read '): elif action.startswith('read '):
count = int(action.split(' ', 1)[1]) count = int(action.split(' ', 1)[1])
if not stdout:
raise error.Abort(_('read not available on this peer'))
stdout.read(count) stdout.read(count)
elif action.startswith('eread '): elif action.startswith('eread '):
count = int(action.split(' ', 1)[1]) count = int(action.split(' ', 1)[1])
if not stderr:
raise error.Abort(_('eread not available on this peer'))
stderr.read(count) stderr.read(count)
else: else:
raise error.Abort(_('unknown action: %s') % action) raise error.Abort(_('unknown action: %s') % action)
if batchedcommands is not None: if batchedcommands is not None:
raise error.Abort(_('unclosed "batchbegin" request')) raise error.Abort(_('unclosed "batchbegin" request'))
if peer: if peer:
peer.close() peer.close()
if proc: if proc:
proc.kill() proc.kill()

tests/test-http-protocol.t

Client receives only supported format even if not server preferred format Client receives only supported format even if not server preferred format
$ get-with-headers.py --hgproto '0.2 comp=zlib' $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' > resp $ get-with-headers.py --hgproto '0.2 comp=zlib' $LOCALIP:$HGPORT '?cmd=getbundle&heads=e93700bd72895c5addab234c56d4024b487a362f&common=0000000000000000000000000000000000000000' > resp
$ f --size --hexdump --bytes 33 --sha1 resp $ f --size --hexdump --bytes 33 --sha1 resp
resp: size=232, sha1=a1c727f0c9693ca15742a75c30419bc36 resp: size=232, sha1=a1c727f0c9693ca15742a75c30419bc36
0000: 32 30 30 20 53 63 72 69 70 74 20 6f 75 74 70 75 |200 Script outpu| 0000: 32 30 30 20 53 63 72 69 70 74 20 6f 75 74 70 75 |200 Script outpu|
0010: 74 20 66 6f 6c 6c 6f 77 73 0a 0a 04 7a 6c 69 62 |t follows...zlib| 0010: 74 20 66 6f 6c 6c 6f 77 73 0a 0a 04 7a 6c 69 62 |t follows...zlib|
0020: 78 |x| 0020: 78 |x|
$ killdaemons.py
$ cd ..
Test listkeys for listing namespaces
$ hg init empty
$ hg -R empty serve -p $HGPORT -d --pid-file hg.pid
$ cat hg.pid > $DAEMON_PIDS
$ hg --verbose debugwireproto http://$LOCALIP:$HGPORT << EOF
> command listkeys
> namespace namespaces
> EOF
s> sendall(*, 0): (glob)
s> GET /?cmd=capabilities HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> accept: application/mercurial-0.1\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: mercurial/proto-1.0 (Mercurial *)\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> readline() -> 36:
s> HTTP/1.1 200 Script output follows\r\n
s> readline() -> *: (glob)
s> Server: *\r\n (glob)
s> readline() -> *: (glob)
s> Date: *\r\n (glob)
s> readline() -> 41:
s> Content-Type: application/mercurial-0.1\r\n
s> readline() -> 21:
s> Content-Length: *\r\n (glob)
s> readline() -> 2:
s> \r\n
s> read(*) -> *: lookup branchmap pushkey known getbundle unbundlehash batch changegroupsubset streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=$BUNDLE2_COMPRESSIONS$ (glob)
sending listkeys command
s> sendall(*, 0): (glob)
s> GET /?cmd=listkeys HTTP/1.1\r\n
s> Accept-Encoding: identity\r\n
s> vary: X-HgArg-1,X-HgProto-1\r\n
s> x-hgarg-1: namespace=namespaces\r\n
s> x-hgproto-1: 0.1 0.2 comp=$USUAL_COMPRESSIONS$\r\n
s> accept: application/mercurial-0.1\r\n
s> host: $LOCALIP:$HGPORT\r\n (glob)
s> user-agent: mercurial/proto-1.0 (Mercurial *)\r\n (glob)
s> \r\n
s> makefile('rb', None)
s> readline() -> 36:
s> HTTP/1.1 200 Script output follows\r\n
s> readline() -> 36:
s> Server: *\r\n (glob)
s> readline() -> *: (glob)
s> Date: *\r\n (glob)
s> readline() -> 41:
s> Content-Type: application/mercurial-0.1\r\n
s> readline() -> 20:
s> Content-Length: 30\r\n
s> readline() -> 2:
s> \r\n
s> read(30) -> 30:
s> bookmarks \n
s> namespaces \n
s> phases
response: bookmarks \nnamespaces \nphases
$ killdaemons.py