This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1952

bundlespec: move computing the bundle contentops in parsebundlespec
ClosedPublic

Authored by lothiraldan on Jan 31 2018, 11:25 AM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Commits
rHG6c7a6b04b274: bundlespec: move computing the bundle contentops in parsebundlespec
Summary

We will introduce a new bundlespec for stream bundle which will influence the
contentops.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

lothiraldan created this revision.Jan 31 2018, 11:25 AM
lothiraldan updated this revision to Diff 5151.Feb 2 2018, 4:30 AM
indygreg requested changes to this revision.Feb 20 2018, 11:48 PM
indygreg added a subscriber: indygreg.

I like where this is going.

It's worth noting that at some point this will reinvent the capabilities mechanism of bundle2. Over the wire protocol today, the client submits its bundle2 capabilities and the server emits parts based on the peer's advertised bundle2 feature support. If you squint, this looks a lot like content options. Have you given any consideration to merging the two concepts and having e.g. a bundlespec map to a pre-defined set of bundle2 capabilities?

mercurial/exchange.py
191

version here does not refer to the changegroup version but rather the bundlespec version.

IMO the changegroup version should be implied by the bundlespec version.

This revision now requires changes to proceed.Feb 20 2018, 11:48 PM
indygreg added a comment.Mar 19 2018, 4:26 PM

I took a bit of a deeper look and found a few more things.

Again, I like the direction of this patch. I think our goal for bundle2 generation should be to derive a set of options for its parts up front as quick as possible and pass that data structure into the lower-level bundle2 generation mechanism.

mercurial/exchange.py
49–51

I *really* like this and this is the direction we need to go. i.e. a bundlespec base version should imply a set of options for generation of the bundle.

89

This comment needs updated.

193

At the point we're returning yet another element, it might be worth converting the return into an attr-derived class. We're starting to use those a bit to define data structures that are too large for tuples but don't yet merit a normal class.

lothiraldan added a comment.Mar 26 2018, 4:24 PM
In D1952#38770, @indygreg wrote:

I like where this is going.
It's worth noting that at some point this will reinvent the capabilities mechanism of bundle2. Over the wire protocol today, the client submits its bundle2 capabilities and the server emits parts based on the peer's advertised bundle2 feature support. If you squint, this looks a lot like content options. Have you given any consideration to merging the two concepts and having e.g. a bundlespec map to a pre-defined set of bundle2 capabilities?

Both concepts seem intimately intricated but I was afraid of the complexity of the series that actually merge them and all the potential backward-incompatible implications. Do you have a specific design in mind we could iterate on?

I will update this series according to your feedbacks.

lothiraldan updated this revision to Diff 7374.Mar 30 2018, 8:55 AM
lothiraldan marked 4 inline comments as done.Mar 30 2018, 9:01 AM
indygreg accepted this revision.Mar 30 2018, 1:04 PM

Thanks for following up. This looks great!

This revision is now accepted and ready to land.Mar 30 2018, 1:04 PM
Closed by commit rHG6c7a6b04b274: bundlespec: move computing the bundle contentops in parsebundlespec (authored by lothiraldan). · Explain WhyMar 30 2018, 1:11 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/commands.py (7 lines)
Mmercurial/exchange.py (27 lines)

Diff 5151

mercurial/commands.py

if 'rev' in opts: if 'rev' in opts:
revstrings = opts['rev'] revstrings = opts['rev']
revs = scmutil.revrange(repo, revstrings) revs = scmutil.revrange(repo, revstrings)
if revstrings and not revs: if revstrings and not revs:
raise error.Abort(_('no commits to bundle')) raise error.Abort(_('no commits to bundle'))
bundletype = opts.get('type', 'bzip2').lower() bundletype = opts.get('type', 'bzip2').lower()
try: try:
bcompression, cgversion, params = exchange.parsebundlespec( bcompression, cgversion, params, contentopts = exchange.parsebundlespec(
repo, bundletype, strict=False) repo, bundletype, strict=False)
except error.UnsupportedBundleSpecification as e: except error.UnsupportedBundleSpecification as e:
raise error.Abort(str(e), raise error.Abort(str(e),
hint=_("see 'hg help bundlespec' for supported " hint=_("see 'hg help bundlespec' for supported "
"values for --type")) "values for --type"))
# Packed bundles are a pseudo bundle format for now. # Packed bundles are a pseudo bundle format for now.
if cgversion == 's1': if cgversion == 's1':
# This is a temporary hack to allow adjusting bundle compression # This is a temporary hack to allow adjusting bundle compression
# level without a) formalizing the bundlespec changes to declare it # level without a) formalizing the bundlespec changes to declare it
# b) introducing a command flag. # b) introducing a command flag.
compopts = {} compopts = {}
complevel = ui.configint('experimental', 'bundlecomplevel') complevel = ui.configint('experimental', 'bundlecomplevel')
if complevel is not None: if complevel is not None:
compopts['level'] = complevel compopts['level'] = complevel
# Allow overriding the bundling of obsmarker in phases through
contentopts = {'cg.version': cgversion, 'changegroup': True} # configuration while we don't have a bundle version that include them
if repo.ui.configbool('experimental', 'evolution.bundle-obsmarker'): if repo.ui.configbool('experimental', 'evolution.bundle-obsmarker'):
contentopts['obsolescence'] = True contentopts['obsolescence'] = True
if repo.ui.configbool('experimental', 'bundle-phases'): if repo.ui.configbool('experimental', 'bundle-phases'):
contentopts['phases'] = True contentopts['phases'] = True
bundle2.writenewbundle(ui, repo, 'bundle', fname, bversion, outgoing, bundle2.writenewbundle(ui, repo, 'bundle', fname, bversion, outgoing,
contentopts, compression=bcompression, contentopts, compression=bcompression,
compopts=compopts) compopts=compopts)
@command('cat', @command('cat',
[('o', 'output', '', [('o', 'output', '',
_('print output to file with formatted name'), _('FORMAT')), _('print output to file with formatted name'), _('FORMAT')),
('r', 'rev', '', _('print the given revision'), _('REV')), ('r', 'rev', '', _('print the given revision'), _('REV')),

mercurial/exchange.py

# Maps bundle version human names to changegroup versions. # Maps bundle version human names to changegroup versions.
_bundlespeccgversions = {'v1': '01', _bundlespeccgversions = {'v1': '01',
'v2': '02', 'v2': '02',
'packed1': 's1', 'packed1': 's1',
'bundle2': '02', #legacy 'bundle2': '02', #legacy
} }
# Maps bundle version with content opts to choose which part to bundle
_bundlespeccontentops = {
'v1': {
indygregUnsubmitted
Done

I *really* like this and this is the direction we need to go. i.e. a bundlespec base version should imply a set of options for generation of the bundle.

indygreg: I *really* like this and this is the direction we need to go. i.e. a bundlespec base version…
'changegroup': True,
'obsolescence': False,
'phases': False
},
'v2': {
'changegroup': True,
'obsolescence': False,
'phases': False
}
}
# Compression engines allowed in version 1. THIS SHOULD NEVER CHANGE. # Compression engines allowed in version 1. THIS SHOULD NEVER CHANGE.
_bundlespecv1compengines = {'gzip', 'bzip2', 'none'} _bundlespecv1compengines = {'gzip', 'bzip2', 'none'}
def parsebundlespec(repo, spec, strict=True, externalnames=False): def parsebundlespec(repo, spec, strict=True, externalnames=False):
"""Parse a bundle string specification into parts. """Parse a bundle string specification into parts.
Bundle specifications denote a well-defined bundle/exchange format. Bundle specifications denote a well-defined bundle/exchange format.
The content of a given specification should not change over time in The content of a given specification should not change over time in
pairs. pairs.
If ``strict`` is True (the default) <compression> is required. Otherwise, If ``strict`` is True (the default) <compression> is required. Otherwise,
it is optional. it is optional.
If ``externalnames`` is False (the default), the human-centric names will If ``externalnames`` is False (the default), the human-centric names will
be converted to their internal representation. be converted to their internal representation.
Returns a 3-tuple of (compression, version, parameters). Compression will Returns a 3-tuple of (compression, version, parameters). Compression will
indygregUnsubmitted
Done

This comment needs updated.

indygreg: This comment needs updated.
be ``None`` if not in strict mode and a compression isn't defined. be ``None`` if not in strict mode and a compression isn't defined.
An ``InvalidBundleSpecification`` is raised when the specification is An ``InvalidBundleSpecification`` is raised when the specification is
not syntactically well formed. not syntactically well formed.
An ``UnsupportedBundleSpecification`` is raised when the compression or An ``UnsupportedBundleSpecification`` is raised when the compression or
bundle type/version is not recognized. bundle type/version is not recognized.
if version == 'packed1' and 'requirements' in params: if version == 'packed1' and 'requirements' in params:
requirements = set(params['requirements'].split(',')) requirements = set(params['requirements'].split(','))
missingreqs = requirements - repo.supportedformats missingreqs = requirements - repo.supportedformats
if missingreqs: if missingreqs:
raise error.UnsupportedBundleSpecification( raise error.UnsupportedBundleSpecification(
_('missing support for repository features: %s') % _('missing support for repository features: %s') %
', '.join(sorted(missingreqs))) ', '.join(sorted(missingreqs)))
# Compute contentops based on the version
contentops = _bundlespeccontentops.get(version, {}).copy()
if not externalnames: if not externalnames:
engine = util.compengines.forbundlename(compression) engine = util.compengines.forbundlename(compression)
compression = engine.bundletype()[1] compression = engine.bundletype()[1]
version = _bundlespeccgversions[version] version = _bundlespeccgversions[version]
return compression, version, params
# Set the cg.version
contentops["cg.version"] = version
indygregUnsubmitted
Done

version here does not refer to the changegroup version but rather the bundlespec version.

IMO the changegroup version should be implied by the bundlespec version.

indygreg: `version` here does not refer to the changegroup version but rather the bundlespec version.
return compression, version, params, contentops
indygregUnsubmitted
Done

At the point we're returning yet another element, it might be worth converting the return into an attr-derived class. We're starting to use those a bit to define data structures that are too large for tuples but don't yet merit a normal class.

indygreg: At the point we're returning yet another element, it might be worth converting the return into…
def readbundle(ui, fh, fname, vfs=None): def readbundle(ui, fh, fname, vfs=None):
header = changegroup.readexactly(fh, 4) header = changegroup.readexactly(fh, 4)
alg = None alg = None
if not fname: if not fname:
fname = "stream" fname = "stream"
if not header.startswith('HG') and header.startswith('\0'): if not header.startswith('HG') and header.startswith('\0'):
value = urlreq.unquote(value) value = urlreq.unquote(value)
attrs[key] = value attrs[key] = value
# Parse BUNDLESPEC into components. This makes client-side # Parse BUNDLESPEC into components. This makes client-side
# preferences easier to specify since you can prefer a single # preferences easier to specify since you can prefer a single
# component of the BUNDLESPEC. # component of the BUNDLESPEC.
if key == 'BUNDLESPEC': if key == 'BUNDLESPEC':
try: try:
comp, version, params = parsebundlespec(repo, value, comp, version, params, _ = parsebundlespec(repo, value,
externalnames=True) externalnames=True)
attrs['COMPRESSION'] = comp attrs['COMPRESSION'] = comp
attrs['VERSION'] = version attrs['VERSION'] = version
except error.InvalidBundleSpecification: except error.InvalidBundleSpecification:
pass pass
except error.UnsupportedBundleSpecification: except error.UnsupportedBundleSpecification:
pass pass
There is no guarantee we'll be able to apply all returned entries because There is no guarantee we'll be able to apply all returned entries because
the metadata we use to filter on may be missing or wrong. the metadata we use to filter on may be missing or wrong.
""" """
newentries = [] newentries = []
for entry in entries: for entry in entries:
spec = entry.get('BUNDLESPEC') spec = entry.get('BUNDLESPEC')
if spec: if spec:
try: try:
comp, version, params = parsebundlespec(repo, spec, strict=True) comp, version, params, _ = parsebundlespec(repo, spec, strict=True)
# If a stream clone was requested, filter out non-streamclone # If a stream clone was requested, filter out non-streamclone
# entries. # entries.
if streamclonerequested and (comp != 'UN' or version != 's1'): if streamclonerequested and (comp != 'UN' or version != 's1'):
repo.ui.debug('filtering %s because not a stream clone\n' % repo.ui.debug('filtering %s because not a stream clone\n' %
entry['URL']) entry['URL'])
continue continue