This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3381

wireprotov2: decode responses to their expected types
ClosedPublic

Authored by indygreg on Apr 14 2018, 8:21 PM.

Download Raw Diff

Details

Reviewers

durin42

Group Reviewers

hg-reviewers

Commits

rHGf7673845b167: wireprotov2: decode responses to their expected types

Summary

Callers of established wire protocol commands expect the
response from that command to be decoded into a data structure.
It's not very useful if callers get back a stream of bytes and
don't know how they should be interpreted - especially since that
stream of bytes varies by wire protocol and even the transport
within that protocol version.

This commit establishes decoding functions for various command
responses so callers of those commands get the response type
they expect.

In theory, this should make the version 2 HTTP peer usable for
various operations. But I haven't tested to confirm.

Diff Detail

Repository

rHG Mercurial

Lint

Lint Skipped

Unit

Unit Tests Skipped

Event Timeline

indygreg created this revision.Apr 14 2018, 8:21 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptApr 14 2018, 8:21 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

indygreg added a child revision: D3382: wireprotov2: define response data as CBOR.Apr 14 2018, 8:21 PM

durin42 accepted this revision.Apr 16 2018, 6:52 PM

This revision is now accepted and ready to land.Apr 16 2018, 6:52 PM

Closed by commit rHGf7673845b167: wireprotov2: decode responses to their expected types (authored by indygreg). · Explain WhyApr 16 2018, 7:10 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/wireprotov2peer.py (44 lines)
M			tests/test-wireproto-command-branchmap.t (2 lines)
M			tests/test-wireproto-command-heads.t (4 lines)
M			tests/test-wireproto-command-known.t (6 lines)
M			tests/test-wireproto-command-listkeys.t (6 lines)
M			tests/test-wireproto-command-lookup.t (2 lines)
M			tests/test-wireproto-command-pushkey.t (4 lines)

Status	Author	Revision
Closed	indygreg	D3388 wireprotov2: add support for more response types
Closed	indygreg	D3387 wireprotov2: remove support for sending bytes response
Closed	indygreg	D3386 wireprotov2: change behavior of error frame
Closed	indygreg	D3385 wireprotov2: change command response protocol to include a leading map
Closed	indygreg	D3384 wireprotov2: change frame type and name for command response
Closed	indygreg	D3383 wireprotov2: change frame type value for command data
Closed	indygreg	D3382 wireprotov2: define response data as CBOR
Closed	indygreg	D3381 wireprotov2: decode responses to their expected types
Closed	indygreg	D3380 wireprotov2: establish a type for representing command response
Closed	indygreg	D3379 wireprotov2: move response handling out of httppeer
Closed	indygreg	D3378 debugcommands: ability to suppress logging of handshake
Closed	indygreg	D3377 hg: pass command intents to repo/peer creation (API)
Closed	indygreg	D3376 registrar: replace "cmdtype" with an intent-based mechanism (API)

Diff 8293

mercurial/wireprotov2peer.py

	# wireprotov2peer.py - client side code for wire protocol version 2			# wireprotov2peer.py - client side code for wire protocol version 2
	#			#
	# Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>			# Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
	#			#
	# This software may be used and distributed according to the terms of the			# This software may be used and distributed according to the terms of the
	# GNU General Public License version 2 or any later version.			# GNU General Public License version 2 or any later version.

	from __future__ import absolute_import			from __future__ import absolute_import

	from .i18n import _			from .i18n import _
	from .thirdparty import (			from .thirdparty import (
	cbor,			cbor,
	)			)
	from . import (			from . import (
				encoding,
	error,			error,
	util,			util,
	wireprotoframing,			wireprotoframing,
	)			)

	class commandresponse(object):			class commandresponse(object):
	"""Represents the response to a command request."""			"""Represents the response to a command request."""


	if action == 'responsedata':			if action == 'responsedata':
	response.b.write(meta['data'])			response.b.write(meta['data'])

	if meta['cbor']:			if meta['cbor']:
	response.cbor = True			response.cbor = True

	if meta['eos']:			if meta['eos']:
	self._futures[frame.requestid].set_result(response)			# If the command has a decoder, resolve the future to the
				# decoded value. Otherwise resolve to the rich response object.
				decoder = COMMAND_DECODERS.get(response.command)

				result = decoder(response) if decoder else response

				self._futures[frame.requestid].set_result(result)

	del self._requests[frame.requestid]			del self._requests[frame.requestid]
	del self._futures[frame.requestid]			del self._futures[frame.requestid]

	else:			else:
	raise error.ProgrammingError(			raise error.ProgrammingError(
	'unhandled action from clientreactor: %s' % action)			'unhandled action from clientreactor: %s' % action)

				def decodebranchmap(resp):
				# Response should be a single CBOR map of branch name to array of nodes.
				bm = next(resp.cborobjects())

				return {encoding.tolocal(k): v for k, v in bm.items()}

				def decodeheads(resp):
				# Array of node bytestrings.
				return next(resp.cborobjects())

				def decodeknown(resp):
				# Bytestring where each byte is a 0 or 1.
				raw = next(resp.cborobjects())

				return [True if c == '1' else False for c in raw]

				def decodelistkeys(resp):
				# Map with bytestring keys and values.
				return next(resp.cborobjects())

				def decodelookup(resp):
				return next(resp.cborobjects())

				def decodepushkey(resp):
				return next(resp.cborobjects())

				COMMAND_DECODERS = {
				'branchmap': decodebranchmap,
				'heads': decodeheads,
				'known': decodeknown,
				'listkeys': decodelistkeys,
				'lookup': decodelookup,
				'pushkey': decodepushkey,
				}

tests/test-wireproto-command-branchmap.t

	s> 78\r\n			s> 78\r\n
	s> p\x00\x00\x01\x00\x02\x01F			s> p\x00\x00\x01\x00\x02\x01F
	s> \xa3Gbranch1\x81T\xb5\xfa\xac\xdf\xd2c7h\xcb1R3l\xc0\x953\x81&f\x88Gbranch2\x81T"Aa\xc7X\x9a\xa4\x8f\xa8:H\xfe\xff^\x95\xb5j\xe3\'\xfcGdefault\x82T&\x80Z\xba\x1e`\n			s> \xa3Gbranch1\x81T\xb5\xfa\xac\xdf\xd2c7h\xcb1R3l\xc0\x953\x81&f\x88Gbranch2\x81T"Aa\xc7X\x9a\xa4\x8f\xa8:H\xfe\xff^\x95\xb5j\xe3\'\xfcGdefault\x82T&\x80Z\xba\x1e`\n
	s> \x82\xe96a\x14\x9f#\x13\x86j"\x1a{T\xbe\x0e\xf7<\x17\xad\xe3\xfc\x89\xdcAp\x1e\xb9\xfc:\x91\xb5\x82\x82			s> \x82\xe96a\x14\x9f#\x13\x86j"\x1a{T\xbe\x0e\xf7<\x17\xad\xe3\xfc\x89\xdcAp\x1e\xb9\xfc:\x91\xb5\x82\x82
	s> \r\n			s> \r\n
	received frame(size=112; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=112; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [{b'branch1': [b'\xb5\xfa\xac\xdf\xd2c7h\xcb1R3l\xc0\x953\x81&f\x88'], b'branch2': [b'"Aa\xc7X\x9a\xa4\x8f\xa8:H\xfe\xff^\x95\xb5j\xe3\'\xfc'], b'default': [b'&\x80Z\xba\x1e`\n\x82\xe96a\x14\x9f#\x13\x86j"\x1a{', b'\xbe\x0e\xf7<\x17\xad\xe3\xfc\x89\xdcAp\x1e\xb9\xfc:\x91\xb5\x82\x82']}]			response: {b'branch1': [b'\xb5\xfa\xac\xdf\xd2c7h\xcb1R3l\xc0\x953\x81&f\x88'], b'branch2': [b'"Aa\xc7X\x9a\xa4\x8f\xa8:H\xfe\xff^\x95\xb5j\xe3\'\xfc'], b'default': [b'&\x80Z\xba\x1e`\n\x82\xe96a\x14\x9f#\x13\x86j"\x1a{', b'\xbe\x0e\xf7<\x17\xad\xe3\xfc\x89\xdcAp\x1e\xb9\xfc:\x91\xb5\x82\x82']}

	$ cat error.log			$ cat error.log

tests/test-wireproto-command-heads.t

	s> \r\n			s> \r\n
	s> 48\r\n			s> 48\r\n
	s> @\x00\x00\x01\x00\x02\x01F			s> @\x00\x00\x01\x00\x02\x01F
	s> \x83T\x1dok\x91\xd4J\xab\xa6\xd5\xe5\x80\xbc0\xa9\x94\x850\xdb\xe0\x0bT\xaeI.6\xb0\xc83\x9f\xfa\xf3(\xd0\x0b\x85\xb4R]\xe1\x16^T)Dm-\xc5A\x9c_\x97Dz\x8b\xc0b\xe4\xcc2\x8b\xf2A			s> \x83T\x1dok\x91\xd4J\xab\xa6\xd5\xe5\x80\xbc0\xa9\x94\x850\xdb\xe0\x0bT\xaeI.6\xb0\xc83\x9f\xfa\xf3(\xd0\x0b\x85\xb4R]\xe1\x16^T)Dm-\xc5A\x9c_\x97Dz\x8b\xc0b\xe4\xcc2\x8b\xf2A
	s> \r\n			s> \r\n
	received frame(size=64; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=64; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [[b'\x1dok\x91\xd4J\xab\xa6\xd5\xe5\x80\xbc0\xa9\x94\x850\xdb\xe0\x0b', b'\xaeI.6\xb0\xc83\x9f\xfa\xf3(\xd0\x0b\x85\xb4R]\xe1\x16^', b')Dm-\xc5A\x9c_\x97Dz\x8b\xc0b\xe4\xcc2\x8b\xf2A']]			response: [b'\x1dok\x91\xd4J\xab\xa6\xd5\xe5\x80\xbc0\xa9\x94\x850\xdb\xe0\x0b', b'\xaeI.6\xb0\xc83\x9f\xfa\xf3(\xd0\x0b\x85\xb4R]\xe1\x16^', b')Dm-\xc5A\x9c_\x97Dz\x8b\xc0b\xe4\xcc2\x8b\xf2A']

	Requesting just the public heads works			Requesting just the public heads works

	$ sendhttpv2peer << EOF			$ sendhttpv2peer << EOF
	> command heads			> command heads
	> publiconly 1			> publiconly 1
	> EOF			> EOF
	creating http peer for wire protocol version 2			creating http peer for wire protocol version 2
	s> \r\n			s> \r\n
	s> 1e\r\n			s> 1e\r\n
	s> \x16\x00\x00\x01\x00\x02\x01F			s> \x16\x00\x00\x01\x00\x02\x01F
	s> \x81Tx\xd2\xdc\xa46\xb2\xf5\xb1\x88\xac&~)\xb8\x1e\x07&m8\xfc			s> \x81Tx\xd2\xdc\xa46\xb2\xf5\xb1\x88\xac&~)\xb8\x1e\x07&m8\xfc
	s> \r\n			s> \r\n
	received frame(size=22; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=22; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [[b'x\xd2\xdc\xa46\xb2\xf5\xb1\x88\xac&~)\xb8\x1e\x07&m8\xfc']]			response: [b'x\xd2\xdc\xa46\xb2\xf5\xb1\x88\xac&~)\xb8\x1e\x07&m8\xfc']

	$ cat error.log			$ cat error.log

tests/test-wireproto-command-known.t

	s> \r\n			s> \r\n
	s> 9\r\n			s> 9\r\n
	s> \x01\x00\x00\x01\x00\x02\x01F			s> \x01\x00\x00\x01\x00\x02\x01F
	s> @			s> @
	s> \r\n			s> \r\n
	received frame(size=1; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=1; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [b'']			response: []

	Single known node works			Single known node works

	$ sendhttpv2peer << EOF			$ sendhttpv2peer << EOF
	> command known			> command known
	> nodes eval:[b'\x42\x6b\xad\xa5\xc6\x75\x98\xca\x65\x03\x6d\x57\xd9\xe4\xb6\x4b\x0c\x1c\xe7\xa0']			> nodes eval:[b'\x42\x6b\xad\xa5\xc6\x75\x98\xca\x65\x03\x6d\x57\xd9\xe4\xb6\x4b\x0c\x1c\xe7\xa0']
	> EOF			> EOF
	creating http peer for wire protocol version 2			creating http peer for wire protocol version 2
	s> \r\n			s> \r\n
	s> a\r\n			s> a\r\n
	s> \x02\x00\x00\x01\x00\x02\x01F			s> \x02\x00\x00\x01\x00\x02\x01F
	s> A1			s> A1
	s> \r\n			s> \r\n
	received frame(size=2; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=2; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [b'1']			response: [True]

	Multiple nodes works			Multiple nodes works

	$ sendhttpv2peer << EOF			$ sendhttpv2peer << EOF
	> command known			> command known
	> nodes eval:[b'\x42\x6b\xad\xa5\xc6\x75\x98\xca\x65\x03\x6d\x57\xd9\xe4\xb6\x4b\x0c\x1c\xe7\xa0', b'00000000000000000000', b'\x11\x24\x78\x96\x29\x61\x14\x71\x24\xed\xd4\x35\x49\xae\xdd\x1a\x33\x5e\x44\xbf']			> nodes eval:[b'\x42\x6b\xad\xa5\xc6\x75\x98\xca\x65\x03\x6d\x57\xd9\xe4\xb6\x4b\x0c\x1c\xe7\xa0', b'00000000000000000000', b'\x11\x24\x78\x96\x29\x61\x14\x71\x24\xed\xd4\x35\x49\xae\xdd\x1a\x33\x5e\x44\xbf']
	> EOF			> EOF
	creating http peer for wire protocol version 2			creating http peer for wire protocol version 2
	s> \r\n			s> \r\n
	s> c\r\n			s> c\r\n
	s> \x04\x00\x00\x01\x00\x02\x01F			s> \x04\x00\x00\x01\x00\x02\x01F
	s> C101			s> C101
	s> \r\n			s> \r\n
	received frame(size=4; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=4; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [b'101']			response: [True, False, True]

	$ cat error.log			$ cat error.log

tests/test-wireproto-command-listkeys.t

	s> \r\n			s> \r\n
	s> 28\r\n			s> 28\r\n
	s> \x00\x00\x01\x00\x02\x01F			s> \x00\x00\x01\x00\x02\x01F
	s> \xa3Fphases@Ibookmarks@Jnamespaces@			s> \xa3Fphases@Ibookmarks@Jnamespaces@
	s> \r\n			s> \r\n
	received frame(size=32; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=32; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [{b'bookmarks': b'', b'namespaces': b'', b'phases': b''}]			response: {b'bookmarks': b'', b'namespaces': b'', b'phases': b''}

	Request for phases works			Request for phases works

	$ sendhttpv2peer << EOF			$ sendhttpv2peer << EOF
	> command listkeys			> command listkeys
	> namespace phases			> namespace phases
	> EOF			> EOF
	creating http peer for wire protocol version 2			creating http peer for wire protocol version 2
	s> \r\n			s> \r\n
	s> 45\r\n			s> 45\r\n
	s> =\x00\x00\x01\x00\x02\x01F			s> =\x00\x00\x01\x00\x02\x01F
	s> \xa2JpublishingDTrueX(be0ef73c17ade3fc89dc41701eb9fc3a91b58282A1			s> \xa2JpublishingDTrueX(be0ef73c17ade3fc89dc41701eb9fc3a91b58282A1
	s> \r\n			s> \r\n
	received frame(size=61; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=61; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [{b'be0ef73c17ade3fc89dc41701eb9fc3a91b58282': b'1', b'publishing': b'True'}]			response: {b'be0ef73c17ade3fc89dc41701eb9fc3a91b58282': b'1', b'publishing': b'True'}

	Request for bookmarks works			Request for bookmarks works

	$ sendhttpv2peer << EOF			$ sendhttpv2peer << EOF
	> command listkeys			> command listkeys
	> namespace bookmarks			> namespace bookmarks
	> EOF			> EOF
	creating http peer for wire protocol version 2			creating http peer for wire protocol version 2
	s> \r\n			s> \r\n
	s> 35\r\n			s> 35\r\n
	s> -\x00\x00\x01\x00\x02\x01F			s> -\x00\x00\x01\x00\x02\x01F
	s> \xa1A@X(26805aba1e600a82e93661149f2313866a221a7b			s> \xa1A@X(26805aba1e600a82e93661149f2313866a221a7b
	s> \r\n			s> \r\n
	received frame(size=45; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=45; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [{b'@': b'26805aba1e600a82e93661149f2313866a221a7b'}]			response: {b'@': b'26805aba1e600a82e93661149f2313866a221a7b'}

	$ cat error.log			$ cat error.log

tests/test-wireproto-command-lookup.t

	s> \r\n			s> \r\n
	s> 1d\r\n			s> 1d\r\n
	s> *\x00\x01\x00\x02\x01F (glob)			s> *\x00\x01\x00\x02\x01F (glob)
	s> TBk\xad\xa5\xc6u\x98\xcae\x03mW\xd9\xe4\xb6K\x0c\x1c\xe7\xa0			s> TBk\xad\xa5\xc6u\x98\xcae\x03mW\xd9\xe4\xb6K\x0c\x1c\xe7\xa0
	s> \r\n			s> \r\n
	received frame(size=*; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor) (glob)			received frame(size=*; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor) (glob)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [b'Bk\xad\xa5\xc6u\x98\xcae\x03mW\xd9\xe4\xb6K\x0c\x1c\xe7\xa0']			response: b'Bk\xad\xa5\xc6u\x98\xcae\x03mW\xd9\xe4\xb6K\x0c\x1c\xe7\xa0'

	$ cat error.log			$ cat error.log

tests/test-wireproto-command-pushkey.t

	s> \r\n			s> \r\n
	s> 9\r\n			s> 9\r\n
	s> *\x00\x01\x00\x02\x01F (glob)			s> *\x00\x01\x00\x02\x01F (glob)
	s> \xf5			s> \xf5
	s> \r\n			s> \r\n
	received frame(size=*; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor) (glob)			received frame(size=*; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor) (glob)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [True]			response: True

	$ sendhttpv2peer << EOF			$ sendhttpv2peer << EOF
	> command listkeys			> command listkeys
	> namespace bookmarks			> namespace bookmarks
	> EOF			> EOF
	creating http peer for wire protocol version 2			creating http peer for wire protocol version 2
	sending listkeys command			sending listkeys command
	s> POST /api/exp-http-v2-0001/ro/listkeys HTTP/1.1\r\n			s> POST /api/exp-http-v2-0001/ro/listkeys HTTP/1.1\r\n
	s> \r\n			s> \r\n
	s> 35\r\n			s> 35\r\n
	s> -\x00\x00\x01\x00\x02\x01F			s> -\x00\x00\x01\x00\x02\x01F
	s> \xa1A@X(426bada5c67598ca65036d57d9e4b64b0c1ce7a0			s> \xa1A@X(426bada5c67598ca65036d57d9e4b64b0c1ce7a0
	s> \r\n			s> \r\n
	received frame(size=45; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)			received frame(size=45; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos\|cbor)
	s> 0\r\n			s> 0\r\n
	s> \r\n			s> \r\n
	response: [{b'@': b'426bada5c67598ca65036d57d9e4b64b0c1ce7a0'}]			response: {b'@': b'426bada5c67598ca65036d57d9e4b64b0c1ce7a0'}

	$ cat error.log			$ cat error.log

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	8293		Apr 14 2018, 8:21 PM	★	★
Diff 2	8324	rHGf7673845b167886277e21a8b0dbac9826b042921	Apr 14 2018, 2:49 PM	★	★