This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3175

commands: implement `export --format=X` with support for CBOR
AbandonedPublic

Authored by indygreg on Apr 6 2018, 6:10 PM.

Download Raw Diff

Details

Reviewers

yuja

Group Reviewers

hg-reviewers

Summary

What's better than having to parse patch files? Not having to parse
them.

The current text-based patch file format used by hg export is good
for humans to exchange. But for machines, it is better to use a
data format that is more structured.

We recently introduced support for CBOR. CBOR is a great, binary
preserving data format (unlike JSON), and I think we should use it
heavily for data interchange.

This commit teaches hg export to write data to CBOR. It adds a
--format argument. Hopefully this is the most controversial part of
this patch. I thought about using --template/-T. We already have
-Tjson. -Tcbor seems like a reasonable feature addition. That might
be the right way forward. I'm not sure. I didn't want to scope bloat
the patch. At least not initially. (The code for exporting a patch
is a bit wonky and I'm not very comfortable with the templating
layer.)

Diff Detail

Repository

rHG Mercurial

Lint

Lint Skipped

Unit

Unit Tests Skipped

Event Timeline

indygreg created this revision.Apr 6 2018, 6:10 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptApr 6 2018, 6:10 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

This is more of an RFC patch. My actual goal is support for ingesting CBOR patches via hg import. I figured it would be easier to test that if we had support for CBOR with hg export. And the reason I want CBOR support for patch ingestion is because it is safer: it reduces the surface area for injecting badness via patch parsing. See e.g. http://rachelbythebay.com/w/2018/04/05/bangpatch/. This implementation is still only partially there: I think a better approach would be to have structured data for the diffs so we don't need to parse those either. That would allow us to use binary data without escaping, not have to use inline text metadata for copy/renames, not have to worry about encoding of filenames, etc. All the problems with "parse a patch" go away and you are left with commit metadata and a series of splicing instructions, which should be pretty generic.

The I/O in the export code is pretty wonky. I'm kinda sad that cbor2 insists on binding an encoder/decoder to a file object. I *really* wish you could get straight bytes out of it without having to use io.BytesIO().

I suspect someone is going to tell me that hg export should use the templating layer. If someone does, I would appreciate help implementing that. Of course, for us to get CBOR with the templating layer, we'd have to teach the templating layer to emit CBOR. Since we can emit JSON, CBOR seems reasonable. I'm just not too familiar with that code though.

I'm broadly in favor. Agree it might make sense to just add generic templating to hg export.

OK, hg export -Tjson appears mostly working. Perhaps we can add
cborfromatter for -Tcbor.

This revision now requires changes to proceed.Apr 12 2018, 10:52 AM

indygreg abandoned this revision.Apr 12 2018, 9:41 PM

Revision Contents
Changeset List

			Path	Packages
M			mercurial/cmdutil.py (72 lines)
M			mercurial/commands.py (11 lines)
M			mercurial/utils/stringutil.py (4 lines)
M			tests/test-completion.t (2 lines)
M			tests/test-export.t (45 lines)

Status	Author	Revision
Abandoned	indygreg	D3175 commands: implement `export --format=X` with support for CBOR
Abandoned	indygreg	D3174 cmdutil: refactor I/O for exporting
Abandoned	indygreg	D3173 cmdutil: drop unused rev arguments from _exportsingle()
Abandoned	indygreg	D3172 cborutil: implement streaming arrays and maps

Diff 7815

mercurial/cmdutil.py


	from .i18n import _			from .i18n import _
	from .node import (			from .node import (
	hex,			hex,
	nullid,			nullid,
	nullrev,			nullrev,
	short,			short,
	)			)
				from .thirdparty import (
				cbor,
				)
	from . import (			from . import (
	bookmarks,			bookmarks,
	changelog,			changelog,
	copies,			copies,
	crecord as crecordmod,			crecord as crecordmod,
	dirstateguard,			dirstateguard,
	encoding,			encoding,
	error,			error,
	subrepoutil,			subrepoutil,
	templatekw,			templatekw,
	templater,			templater,
	util,			util,
	vfs as vfsmod,			vfs as vfsmod,
	)			)

	from .utils import (			from .utils import (
				cborutil,
	dateutil,			dateutil,
	stringutil,			stringutil,
	)			)

	stringio = util.stringio			stringio = util.stringio

	# templates of common command options			# templates of common command options

	if header is not None:			if header is not None:
	yield '# %s\n' % header, None			yield '# %s\n' % header, None
	yield ctx.description().rstrip(), None			yield ctx.description().rstrip(), None
	yield "\n\n", None			yield "\n\n", None

	for chunk, label in patch.diffui(repo, prev, node, match, opts=diffopts):			for chunk, label in patch.diffui(repo, prev, node, match, opts=diffopts):
	yield chunk, label			yield chunk, label

				def _exportsinglecbor(repo, ctx, match, switch_parent, seqno, diffopts,
				encoder):
				"""Export a single patch to CBOR."""
				node = scmutil.binnode(ctx)
				parents = [p.node() for p in ctx.parents() if p]
				branch = ctx.branch()
				if switch_parent:
				parents.reverse()

				if parents:
				prev = parents[0]
				else:
				prev = nullid

				with cborutil.streammap(encoder) as writeitem:
				writeitem(b'version', 1)
				writeitem(b'user', encoding.fromlocal(ctx.user()))
				writeitem(b'time', ctx.date()[0])
				writeitem(b'timezone', ctx.date()[1])
				writeitem(b'branch', branch)
				writeitem(b'node', node)
				writeitem(b'parents', parents)

				extras = []
				for headerid in extraexport:
				header = extraexportmap[headerid](seqno, ctx)
				if header:
				extras.append(header)

				if extras:
				writeitem(b'extras', extras)

				writeitem(b'description',
				encoding.fromlocal(ctx.description().rstrip()))

				encoder.encode(b'patchchunks')
				chunks = patch.diff(repo, prev, node, match, opts=diffopts)
				cborutil.streamarrayitems(encoder, chunks)

	def export(repo, revs, fntemplate='hg-%h.patch', fp=None, switch_parent=False,			def export(repo, revs, fntemplate='hg-%h.patch', fp=None, switch_parent=False,
	opts=None, match=None):			opts=None, match=None, fmt='hg'):
	'''export changesets as hg patches			'''export changesets as hg patches

	Args:			Args:
	repo: The repository from which we're exporting revisions.			repo: The repository from which we're exporting revisions.
	revs: A list of revisions to export as revision numbers.			revs: A list of revisions to export as revision numbers.
	fntemplate: An optional string to use for generating patch file names.			fntemplate: An optional string to use for generating patch file names.
	fp: An optional file-like object to which patches should be written.			fp: An optional file-like object to which patches should be written.
	switch_parent: If True, show diffs against second parent when not nullid.			switch_parent: If True, show diffs against second parent when not nullid.
	Default is false, which always shows diff against p1.			Default is false, which always shows diff against p1.
	opts: diff options to use for generating the patch.			opts: diff options to use for generating the patch.
	match: If specified, only export changes to files matching this matcher.			match: If specified, only export changes to files matching this matcher.
				fmt: Format to export the patch to. ``hg`` is the default Mercurial
				patch format. ``cbor`` encodes to CBOR.

	Returns:			Returns:
	Nothing.			Nothing.

	Side Effect:			Side Effect:
	"HG Changeset Patch" data is emitted to one of the following			"HG Changeset Patch" data is emitted to one of the following
	destinations:			destinations:
	fp is specified: All revs are written to the specified			fp is specified: All revs are written to the specified

	for s, label in _exportsingle(repo, ctx, match, switch_parent, seqno,			for s, label in _exportsingle(repo, ctx, match, switch_parent, seqno,
	opts):			opts):
	if supportslabel and label:			if supportslabel and label:
	fp.write(s, label=label)			fp.write(s, label=label)
	else:			else:
	fp.write(s)			fp.write(s)

				if fmt == 'cbor' and allfp:
				allcbor = cbor.CBOREncoder(allfp)
				cborutil.beginindefinitearray(allcbor)
				else:
				allcbor = None

	for seqno, rev in enumerate(revs, 1):			for seqno, rev in enumerate(revs, 1):
	ctx = repo[rev]			ctx = repo[rev]

	if not allfp:			if not allfp:
	assert fntemplate			assert fntemplate

	with makefileobj(ctx, fntemplate, mode='wb', modemap=filemode,			with makefileobj(ctx, fntemplate, mode='wb', modemap=filemode,
	total=total, seqno=seqno, revwidth=revwidth) as fp:			total=total, seqno=seqno, revwidth=revwidth) as fp:
				if fmt == 'cbor':
				encoder = cbor.CBOREncoder(fp)
				with cborutil.streamarray(encoder):
				_exportsinglecbor(repo, ctx, match, switch_parent,
				seqno, opts, encoder)
				else:
	doexport(ctx, fp, fp.name)			doexport(ctx, fp, fp.name)
	else:			else:
				if fmt == 'cbor':
				_exportsinglecbor(repo, ctx, match, switch_parent, seqno, opts,
				allcbor)
				else:
	doexport(ctx, allfp, dest)			doexport(ctx, allfp, dest)

				if allcbor:
				cborutil.endindefinite(allcbor)

	def showmarker(fm, marker, index=None):			def showmarker(fm, marker, index=None):
	"""utility function to display obsolescence marker in a readable way			"""utility function to display obsolescence marker in a readable way

	To be used by debug function."""			To be used by debug function."""
	if index is not None:			if index is not None:
	fm.write('index', '%i ', index)			fm.write('index', '%i ', index)
	fm.write('prednode', '%s ', hex(marker.prednode()))			fm.write('prednode', '%s ', hex(marker.prednode()))
	succs = marker.succnodes()			succs = marker.succnodes()

mercurial/commands.py

	listsubrepos=opts.get('subrepos'),			listsubrepos=opts.get('subrepos'),
	root=opts.get('root'))			root=opts.get('root'))

	@command('^export',			@command('^export',
	[('o', 'output', '',			[('o', 'output', '',
	_('print output to file with formatted name'), _('FORMAT')),			_('print output to file with formatted name'), _('FORMAT')),
	('', 'switch-parent', None, _('diff against the second parent')),			('', 'switch-parent', None, _('diff against the second parent')),
	('r', 'rev', [], _('revisions to export'), _('REV')),			('r', 'rev', [], _('revisions to export'), _('REV')),
				('', 'format', 'hg', _('format of written data (EXPERIMENTAL)')),
	] + diffopts,			] + diffopts,
	_('[OPTION]... [-o OUTFILESPEC] [-r] [REV]...'), cmdtype=readonly)			_('[OPTION]... [-o OUTFILESPEC] [-r] [REV]...'), cmdtype=readonly)
	def export(ui, repo, changesets, *opts):			def export(ui, repo, changesets, *opts):
	"""dump the header and diffs for one or more changesets			"""dump the header and diffs for one or more changesets

	Print the changeset header and diffs for one or more revisions.			Print the changeset header and diffs for one or more revisions.
	If no revision is given, the parent of the working directory is used.			If no revision is given, the parent of the working directory is used.

	opts = pycompat.byteskwargs(opts)			opts = pycompat.byteskwargs(opts)
	changesets += tuple(opts.get('rev', []))			changesets += tuple(opts.get('rev', []))
	if not changesets:			if not changesets:
	changesets = ['.']			changesets = ['.']
	repo = scmutil.unhidehashlikerevs(repo, changesets, 'nowarn')			repo = scmutil.unhidehashlikerevs(repo, changesets, 'nowarn')
	revs = scmutil.revrange(repo, changesets)			revs = scmutil.revrange(repo, changesets)
	if not revs:			if not revs:
	raise error.Abort(_("export requires at least one changeset"))			raise error.Abort(_("export requires at least one changeset"))

				if opts['format'] not in ('hg', 'cbor'):
				raise error.Abort(_('invalid value for --format: only "hg" and "cbor" '
				'are supported'))

	if len(revs) > 1:			if len(revs) > 1:
	ui.note(_('exporting patches:\n'))			ui.note(_('exporting patches:\n'))
	else:			else:
	ui.note(_('exporting patch:\n'))			ui.note(_('exporting patch:\n'))
	ui.pager('export')			ui.pager('export')
	cmdutil.export(repo, revs, fntemplate=opts.get('output'),			cmdutil.export(repo, revs, fntemplate=opts.get('output'),
	switch_parent=opts.get('switch_parent'),			switch_parent=opts.get('switch_parent'),
	opts=patch.diffallopts(ui, opts))			opts=patch.diffallopts(ui, opts),
				fmt=opts['format'])

	@command('files',			@command('files',
	[('r', 'rev', '', _('search the repository as it is in REV'), _('REV')),			[('r', 'rev', '', _('search the repository as it is in REV'), _('REV')),
	('0', 'print0', None, _('end filenames with NUL, for use with xargs')),			('0', 'print0', None, _('end filenames with NUL, for use with xargs')),
	] + walkopts + formatteropts + subrepoopts,			] + walkopts + formatteropts + subrepoopts,
	_('[OPTION]... [FILE]...'), cmdtype=readonly)			_('[OPTION]... [FILE]...'), cmdtype=readonly)
	def files(ui, repo, pats, *opts):			def files(ui, repo, pats, *opts):
	"""list tracked files			"""list tracked files

mercurial/utils/stringutil.py

	return "b'%s'" % escapestr(o)			return "b'%s'" % escapestr(o)
	elif isinstance(o, list):			elif isinstance(o, list):
	return '[%s]' % (b', '.join(pprint(a) for a in o))			return '[%s]' % (b', '.join(pprint(a) for a in o))
	elif isinstance(o, dict):			elif isinstance(o, dict):
	return '{%s}' % (b', '.join(			return '{%s}' % (b', '.join(
	'%s: %s' % (pprint(k), pprint(v)) for k, v in sorted(o.items())))			'%s: %s' % (pprint(k), pprint(v)) for k, v in sorted(o.items())))
	elif isinstance(o, bool):			elif isinstance(o, bool):
	return b'True' if o else b'False'			return b'True' if o else b'False'
				elif isinstance(o, int):
				return '%d' % o
				elif isinstance(o, float):
				return b'%f' % o
	else:			else:
	raise error.ProgrammingError('do not know how to format %r' % o)			raise error.ProgrammingError('do not know how to format %r' % o)

	def binary(s):			def binary(s):
	"""return true if a string is binary data"""			"""return true if a string is binary data"""
	return bool(s and '\0' in s)			return bool(s and '\0' in s)

	def stringmatcher(pattern, casesensitive=True):			def stringmatcher(pattern, casesensitive=True):

tests/test-completion.t


	Show all commands + options			Show all commands + options
	$ hg debugcommands			$ hg debugcommands
	add: include, exclude, subrepos, dry-run			add: include, exclude, subrepos, dry-run
	annotate: rev, follow, no-follow, text, user, file, date, number, changeset, line-number, skip, ignore-all-space, ignore-space-change, ignore-blank-lines, ignore-space-at-eol, include, exclude, template			annotate: rev, follow, no-follow, text, user, file, date, number, changeset, line-number, skip, ignore-all-space, ignore-space-change, ignore-blank-lines, ignore-space-at-eol, include, exclude, template
	clone: noupdate, updaterev, rev, branch, pull, uncompressed, stream, ssh, remotecmd, insecure			clone: noupdate, updaterev, rev, branch, pull, uncompressed, stream, ssh, remotecmd, insecure
	commit: addremove, close-branch, amend, secret, edit, interactive, include, exclude, message, logfile, date, user, subrepos			commit: addremove, close-branch, amend, secret, edit, interactive, include, exclude, message, logfile, date, user, subrepos
	diff: rev, change, text, git, binary, nodates, noprefix, show-function, reverse, ignore-all-space, ignore-space-change, ignore-blank-lines, ignore-space-at-eol, unified, stat, root, include, exclude, subrepos			diff: rev, change, text, git, binary, nodates, noprefix, show-function, reverse, ignore-all-space, ignore-space-change, ignore-blank-lines, ignore-space-at-eol, unified, stat, root, include, exclude, subrepos
	export: output, switch-parent, rev, text, git, binary, nodates			export: output, switch-parent, rev, format, text, git, binary, nodates
	forget: include, exclude, dry-run			forget: include, exclude, dry-run
	init: ssh, remotecmd, insecure			init: ssh, remotecmd, insecure
	log: follow, follow-first, date, copies, keyword, rev, line-range, removed, only-merges, user, only-branch, branch, prune, patch, git, limit, no-merges, stat, graph, style, template, include, exclude			log: follow, follow-first, date, copies, keyword, rev, line-range, removed, only-merges, user, only-branch, branch, prune, patch, git, limit, no-merges, stat, graph, style, template, include, exclude
	merge: force, rev, preview, abort, tool			merge: force, rev, preview, abort, tool
	pull: update, force, rev, bookmark, branch, ssh, remotecmd, insecure			pull: update, force, rev, bookmark, branch, ssh, remotecmd, insecure
	push: force, rev, bookmark, branch, new-branch, pushvars, ssh, remotecmd, insecure			push: force, rev, bookmark, branch, new-branch, pushvars, ssh, remotecmd, insecure
	remove: after, force, subrepos, include, exclude, dry-run			remove: after, force, subrepos, include, exclude, dry-run
	serve: accesslog, daemon, daemon-postexec, errorlog, port, address, prefix, name, web-conf, webdir-conf, pid-file, stdio, cmdserver, templates, style, ipv6, certificate, subrepos			serve: accesslog, daemon, daemon-postexec, errorlog, port, address, prefix, name, web-conf, webdir-conf, pid-file, stdio, cmdserver, templates, style, ipv6, certificate, subrepos

tests/test-export.t

	[255]			[255]
	$ hg export 999			$ hg export 999
	abort: unknown revision '999'!			abort: unknown revision '999'!
	[255]			[255]
	$ hg export "not all()"			$ hg export "not all()"
	abort: export requires at least one changeset			abort: export requires at least one changeset
	[255]			[255]

				--format=hg works and is the default

				$ hg export > expected
				$ hg export --format hg > got
				$ diff got expected

				--format=cbor emits a CBOR patch

				$ hg export --format cbor > cbor
				$ f --sha256 --hexdump cbor
				cbor: sha256=825d5c5a23bb64d1aad17e39f1e151c933d519293f3d05eb591a7479453ea21a
				0000: 9f bf 47 76 65 72 73 69 6f 6e 01 44 75 73 65 72 \|..Gversion.Duser\|
				0010: 44 74 65 73 74 44 74 69 6d 65 fb 00 00 00 00 00 \|DtestDtime......\|
				0020: 00 00 00 48 74 69 6d 65 7a 6f 6e 65 00 46 62 72 \|...Htimezone.Fbr\|
				0030: 61 6e 63 68 47 64 65 66 61 75 6c 74 44 6e 6f 64 \|anchGdefaultDnod\|
				0040: 65 54 19 7e cd 81 a5 7f 76 0b 54 f3 4a 58 81 7a \|eT.~....v.T.JX.z\|
				0050: d5 b0 49 91 fa 47 47 70 61 72 65 6e 74 73 81 54 \|..I..GGparents.T\|
				0060: f3 ac ba fa c1 61 ec 68 f1 59 8a f3 8f 79 4f 28 \|.....a.h.Y...yO(\|
				0070: 84 7c a5 d3 4b 64 65 73 63 72 69 70 74 69 6f 6e \|.\|..Kdescription\|
				0080: 58 5b 20 21 22 23 24 25 26 28 2c 2d 2e 2f 30 31 \|X[ !"#$%&(,-./01\|
				0090: 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 \|23456789:;<=>?@A\|
				00a0: 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 \|BCDEFGHIJKLMNOPQ\|
				00b0: 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 \|RSTUVWXYZ[\]^_`a\|
				00c0: 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 \|bcdefghijklmnopq\|
				00d0: 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 4b 70 61 \|rstuvwxyz{\|}~Kpa\|
				00e0: 74 63 68 63 68 75 6e 6b 73 9f 58 7b 64 69 66 66 \|tchchunks.X{diff\|
				00f0: 20 2d 72 20 66 33 61 63 62 61 66 61 63 31 36 31 \| -r f3acbafac161\|
				0100: 20 2d 72 20 31 39 37 65 63 64 38 31 61 35 37 66 \| -r 197ecd81a57f\|
				0110: 20 66 6f 6f 0a 2d 2d 2d 20 61 2f 66 6f 6f 09 54 \| foo.--- a/foo.T\|
				0120: 68 75 20 4a 61 6e 20 30 31 20 30 30 3a 30 30 3a \|hu Jan 01 00:00:\|
				0130: 30 30 20 31 39 37 30 20 2b 30 30 30 30 0a 2b 2b \|00 1970 +0000.++\|
				0140: 2b 20 62 2f 66 6f 6f 09 54 68 75 20 4a 61 6e 20 \|+ b/foo.Thu Jan \|
				0150: 30 31 20 30 30 3a 30 30 3a 30 30 20 31 39 37 30 \|01 00:00:00 1970\|
				0160: 20 2b 30 30 30 30 0a 58 2f 40 40 20 2d 31 30 2c \| +0000.X/@@ -10,\|
				0170: 33 20 2b 31 30 2c 34 20 40 40 0a 20 66 6f 6f 2d \|3 +10,4 @@. foo-\|
				0180: 39 0a 20 66 6f 6f 2d 31 30 0a 20 66 6f 6f 2d 31 \|9. foo-10. foo-1\|
				0190: 31 0a 2b 6c 69 6e 65 0a ff ff ff \|1.+line....\|

				>>> from __future__ import print_function
				>>> from mercurial.thirdparty import cbor
				>>> from mercurial.utils import stringutil
				>>> with open('cbor', 'rb') as fh:
				... print(stringutil.pprint(cbor.load(fh)))
				[{b'branch': b'default', b'description': b' !"#$%&(,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{\|}~', b'node': b'\x19~\xcd\x81\xa5\x7fv\x0bT\xf3JX\x81z\xd5\xb0I\x91\xfaG', b'parents': [b'\xf3\xac\xba\xfa\xc1a\xech\xf1Y\x8a\xf3\x8fyO(\x84\|\xa5\xd3'], b'patchchunks': [b'diff -r f3acbafac161 -r 197ecd81a57f foo\n--- a/foo\tThu Jan 01 00:00:00 1970 +0000\n+++ b/foo\tThu Jan 01 00:00:00 1970 +0000\n', b'@@ -10,3 +10,4 @@\n foo-9\n foo-10\n foo-11\n+line\n'], b'time': 0.000000, b'timezone': 0, b'user': b'test', b'version': 1}]

	Check for color output			Check for color output
	$ cat <<EOF >> $HGRCPATH			$ cat <<EOF >> $HGRCPATH
	> [color]			> [color]
	> mode = ansi			> mode = ansi
	> [extensions]			> [extensions]
	> color =			> color =
	> EOF			> EOF

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	7815			Apr 6 2018, 6:10 PM	★	★