This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4086

changegroup: declare shallow flag in constructor
ClosedPublic

Authored by indygreg on Aug 3 2018, 5:04 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGcdb9bc216771: changegroup: declare shallow flag in constructor
Summary

Thus begins the process of better formalizing ellipses and shallow
changegroup generation mode so it is tracked by cgpacker at
construction time instead of bolted on after the fact by a
wrapper function.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Aug 3 2018, 5:04 PM
Closed by commit rHGcdb9bc216771: changegroup: declare shallow flag in constructor (authored by indygreg). · Explain WhyAug 6 2018, 2:57 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/changegroup.py (38 lines)

Diff 9848

mercurial/changegroup.py

# 2 bytes of flags to apply to revision data. # 2 bytes of flags to apply to revision data.
flags = attr.ib() flags = attr.ib()
# Iterable of chunks holding raw delta data. # Iterable of chunks holding raw delta data.
deltachunks = attr.ib() deltachunks = attr.ib()
class cgpacker(object): class cgpacker(object):
def __init__(self, repo, filematcher, version, allowreorder, def __init__(self, repo, filematcher, version, allowreorder,
useprevdelta, builddeltaheader, manifestsend, useprevdelta, builddeltaheader, manifestsend,
sendtreemanifests, bundlecaps=None): sendtreemanifests, bundlecaps=None, shallow=False):
"""Given a source repo, construct a bundler. """Given a source repo, construct a bundler.
filematcher is a matcher that matches on files to include in the filematcher is a matcher that matches on files to include in the
changegroup. Used to facilitate sparse changegroups. changegroup. Used to facilitate sparse changegroups.
allowreorder controls whether reordering of revisions is allowed. allowreorder controls whether reordering of revisions is allowed.
This value is used when ``bundle.reorder`` is ``auto`` or isn't This value is used when ``bundle.reorder`` is ``auto`` or isn't
set. set.
useprevdelta controls whether revisions should always delta against useprevdelta controls whether revisions should always delta against
the previous revision in the changegroup. the previous revision in the changegroup.
builddeltaheader is a callable that constructs the header for a group builddeltaheader is a callable that constructs the header for a group
delta. delta.
manifestsend is a chunk to send after manifests have been fully emitted. manifestsend is a chunk to send after manifests have been fully emitted.
sendtreemanifests indicates whether tree manifests should be emitted. sendtreemanifests indicates whether tree manifests should be emitted.
bundlecaps is optional and can be used to specify the set of bundlecaps is optional and can be used to specify the set of
capabilities which can be used to build the bundle. While bundlecaps is capabilities which can be used to build the bundle. While bundlecaps is
unused in core Mercurial, extensions rely on this feature to communicate unused in core Mercurial, extensions rely on this feature to communicate
capabilities to customize the changegroup packer. capabilities to customize the changegroup packer.
shallow indicates whether shallow data might be sent. The packer may
need to pack file contents not introduced by the changes being packed.
""" """
assert filematcher assert filematcher
self._filematcher = filematcher self._filematcher = filematcher
self.version = version self.version = version
self._useprevdelta = useprevdelta self._useprevdelta = useprevdelta
self._builddeltaheader = builddeltaheader self._builddeltaheader = builddeltaheader
self._manifestsend = manifestsend self._manifestsend = manifestsend
self._sendtreemanifests = sendtreemanifests self._sendtreemanifests = sendtreemanifests
# Set of capabilities we can use to build the bundle. # Set of capabilities we can use to build the bundle.
if bundlecaps is None: if bundlecaps is None:
bundlecaps = set() bundlecaps = set()
self._bundlecaps = bundlecaps self._bundlecaps = bundlecaps
self._isshallow = shallow
# experimental config: bundle.reorder # experimental config: bundle.reorder
reorder = repo.ui.config('bundle', 'reorder') reorder = repo.ui.config('bundle', 'reorder')
if reorder == 'auto': if reorder == 'auto':
self._reorder = allowreorder self._reorder = allowreorder
else: else:
self._reorder = stringutil.parsebool(reorder) self._reorder = stringutil.parsebool(reorder)
# Set this narrow-specific dict so we have the lowest # Set this narrow-specific dict so we have the lowest
# manifest revnum to look up for this cl revnum. (Part of # manifest revnum to look up for this cl revnum. (Part of
# mapping changelog ellipsis parents to manifest ellipsis # mapping changelog ellipsis parents to manifest ellipsis
# parents) # parents)
self._next_clrev_to_localrev.setdefault(cl.rev(x), self._next_clrev_to_localrev.setdefault(cl.rev(x),
mfrevlog.rev(n)) mfrevlog.rev(n))
# We can't trust the changed files list in the changeset if the # We can't trust the changed files list in the changeset if the
# client requested a shallow clone. # client requested a shallow clone.
if self._is_shallow: if self._isshallow:
changedfiles.update(mfl[c[0]].read().keys()) changedfiles.update(mfl[c[0]].read().keys())
else: else:
changedfiles.update(c[3]) changedfiles.update(c[3])
else: else:
n = c[0] n = c[0]
# record the first changeset introducing this manifest version # record the first changeset introducing this manifest version
mfs.setdefault(n, x) mfs.setdefault(n, x)
'treemanifest' not in repo.requirements) 'treemanifest' not in repo.requirements)
for chunk in self.generatemanifests(commonrevs, clrevorder, for chunk in self.generatemanifests(commonrevs, clrevorder,
fastpathlinkrev, mfs, fnodes, source): fastpathlinkrev, mfs, fnodes, source):
yield chunk yield chunk
if ellipsesmode: if ellipsesmode:
mfdicts = None mfdicts = None
if self._is_shallow: if self._isshallow:
mfdicts = [(self._repo.manifestlog[n].read(), lr) mfdicts = [(self._repo.manifestlog[n].read(), lr)
for (n, lr) in mfs.iteritems()] for (n, lr) in mfs.iteritems()]
mfs.clear() mfs.clear()
clrevs = set(cl.rev(x) for x in clnodes) clrevs = set(cl.rev(x) for x in clnodes)
if not fastpathlinkrev: if not fastpathlinkrev:
def linknodes(unused, fname): def linknodes(unused, fname):
yield x yield x
self._verbosenote(_('%8.i (manifests)\n') % size) self._verbosenote(_('%8.i (manifests)\n') % size)
yield self._manifestsend yield self._manifestsend
# The 'source' parameter is useful for extensions # The 'source' parameter is useful for extensions
def generatefiles(self, changedfiles, linknodes, commonrevs, source): def generatefiles(self, changedfiles, linknodes, commonrevs, source):
changedfiles = list(filter(self._filematcher, changedfiles)) changedfiles = list(filter(self._filematcher, changedfiles))
if getattr(self, '_is_shallow', False): if self._isshallow:
# See comment in generate() for why this sadness is a thing. # See comment in generate() for why this sadness is a thing.
mfdicts = self._mfdicts mfdicts = self._mfdicts
del self._mfdicts del self._mfdicts
# In a shallow clone, the linknodes callback needs to also include # In a shallow clone, the linknodes callback needs to also include
# those file nodes that are in the manifests we sent but weren't # those file nodes that are in the manifests we sent but weren't
# introduced by those manifests. # introduced by those manifests.
commonctxs = [self._repo[c] for c in commonrevs] commonctxs = [self._repo[c] for c in commonrevs]
oldlinknodes = linknodes oldlinknodes = linknodes
p1node=p1n, p1node=p1n,
p2node=p2n, p2node=p2n,
basenode=nullid, basenode=nullid,
linknode=linknode, linknode=linknode,
flags=flags, flags=flags,
deltachunks=(diffheader, data), deltachunks=(diffheader, data),
) )
def _makecg1packer(repo, filematcher, bundlecaps): def _makecg1packer(repo, filematcher, bundlecaps, shallow=False):
builddeltaheader = lambda d: _CHANGEGROUPV1_DELTA_HEADER.pack( builddeltaheader = lambda d: _CHANGEGROUPV1_DELTA_HEADER.pack(
d.node, d.p1node, d.p2node, d.linknode) d.node, d.p1node, d.p2node, d.linknode)
return cgpacker(repo, filematcher, b'01', return cgpacker(repo, filematcher, b'01',
useprevdelta=True, useprevdelta=True,
allowreorder=None, allowreorder=None,
builddeltaheader=builddeltaheader, builddeltaheader=builddeltaheader,
manifestsend=b'', manifestsend=b'',
sendtreemanifests=False, sendtreemanifests=False,
bundlecaps=bundlecaps) bundlecaps=bundlecaps,
shallow=shallow)
def _makecg2packer(repo, filematcher, bundlecaps): def _makecg2packer(repo, filematcher, bundlecaps, shallow=False):
builddeltaheader = lambda d: _CHANGEGROUPV2_DELTA_HEADER.pack( builddeltaheader = lambda d: _CHANGEGROUPV2_DELTA_HEADER.pack(
d.node, d.p1node, d.p2node, d.basenode, d.linknode) d.node, d.p1node, d.p2node, d.basenode, d.linknode)
# Since generaldelta is directly supported by cg2, reordering # Since generaldelta is directly supported by cg2, reordering
# generally doesn't help, so we disable it by default (treating # generally doesn't help, so we disable it by default (treating
# bundle.reorder=auto just like bundle.reorder=False). # bundle.reorder=auto just like bundle.reorder=False).
return cgpacker(repo, filematcher, b'02', return cgpacker(repo, filematcher, b'02',
useprevdelta=False, useprevdelta=False,
allowreorder=False, allowreorder=False,
builddeltaheader=builddeltaheader, builddeltaheader=builddeltaheader,
manifestsend=b'', manifestsend=b'',
sendtreemanifests=False, sendtreemanifests=False,
bundlecaps=bundlecaps) bundlecaps=bundlecaps,
shallow=shallow)
def _makecg3packer(repo, filematcher, bundlecaps): def _makecg3packer(repo, filematcher, bundlecaps, shallow=False):
builddeltaheader = lambda d: _CHANGEGROUPV3_DELTA_HEADER.pack( builddeltaheader = lambda d: _CHANGEGROUPV3_DELTA_HEADER.pack(
d.node, d.p1node, d.p2node, d.basenode, d.linknode, d.flags) d.node, d.p1node, d.p2node, d.basenode, d.linknode, d.flags)
return cgpacker(repo, filematcher, b'03', return cgpacker(repo, filematcher, b'03',
useprevdelta=False, useprevdelta=False,
allowreorder=False, allowreorder=False,
builddeltaheader=builddeltaheader, builddeltaheader=builddeltaheader,
manifestsend=closechunk(), manifestsend=closechunk(),
sendtreemanifests=True, sendtreemanifests=True,
bundlecaps=bundlecaps) bundlecaps=bundlecaps,
shallow=shallow)
_packermap = {'01': (_makecg1packer, cg1unpacker), _packermap = {'01': (_makecg1packer, cg1unpacker),
# cg2 adds support for exchanging generaldelta # cg2 adds support for exchanging generaldelta
'02': (_makecg2packer, cg2unpacker), '02': (_makecg2packer, cg2unpacker),
# cg3 adds support for exchanging revlog flags and treemanifests # cg3 adds support for exchanging revlog flags and treemanifests
'03': (_makecg3packer, cg3unpacker), '03': (_makecg3packer, cg3unpacker),
} }
# will support. For example, all hg versions that support generaldelta also # will support. For example, all hg versions that support generaldelta also
# support changegroup 02. # support changegroup 02.
versions = supportedoutgoingversions(repo) versions = supportedoutgoingversions(repo)
if 'generaldelta' in repo.requirements: if 'generaldelta' in repo.requirements:
versions.discard('01') versions.discard('01')
assert versions assert versions
return min(versions) return min(versions)
def getbundler(version, repo, bundlecaps=None, filematcher=None): def getbundler(version, repo, bundlecaps=None, filematcher=None,
shallow=False):
assert version in supportedoutgoingversions(repo) assert version in supportedoutgoingversions(repo)
if filematcher is None: if filematcher is None:
filematcher = matchmod.alwaysmatcher(repo.root, '') filematcher = matchmod.alwaysmatcher(repo.root, '')
if version == '01' and not filematcher.always(): if version == '01' and not filematcher.always():
raise error.ProgrammingError('version 01 changegroups do not support ' raise error.ProgrammingError('version 01 changegroups do not support '
'sparse file matchers') 'sparse file matchers')
# Requested files could include files not in the local store. So # Requested files could include files not in the local store. So
# filter those out. # filter those out.
filematcher = matchmod.intersectmatchers(repo.narrowmatch(), filematcher = matchmod.intersectmatchers(repo.narrowmatch(),
filematcher) filematcher)
fn = _packermap[version][0] fn = _packermap[version][0]
return fn(repo, filematcher, bundlecaps) return fn(repo, filematcher, bundlecaps, shallow=shallow)
def getunbundler(version, fh, alg, extras=None): def getunbundler(version, fh, alg, extras=None):
return _packermap[version][1](fh, alg, extras=extras) return _packermap[version][1](fh, alg, extras=extras)
def _changegroupinfo(repo, nodes, source): def _changegroupinfo(repo, nodes, source):
if repo.ui.verbose or source == 'bundle': if repo.ui.verbose or source == 'bundle':
repo.ui.status(_("%d changesets found\n") % len(nodes)) repo.ui.status(_("%d changesets found\n") % len(nodes))
if repo.ui.debugflag: if repo.ui.debugflag:
raise error.Abort( raise error.Abort(
'ellipsis nodes require at least cg3 on client and server, ' 'ellipsis nodes require at least cg3 on client and server, '
'but negotiated version %s' % version) 'but negotiated version %s' % version)
# We wrap cg1packer.revchunk, using a side channel to pass # We wrap cg1packer.revchunk, using a side channel to pass
# relevant_nodes into that area. Then if linknode isn't in the # relevant_nodes into that area. Then if linknode isn't in the
# set, we know we have an ellipsis node and we should defer # set, we know we have an ellipsis node and we should defer
# sending that node's data. We override close() to detect # sending that node's data. We override close() to detect
# pending ellipsis nodes and flush them. # pending ellipsis nodes and flush them.
packer = getbundler(version, repo, filematcher=match) packer = getbundler(version, repo, filematcher=match,
shallow=depth is not None)
# Give the packer the list of nodes which should not be # Give the packer the list of nodes which should not be
# ellipsis nodes. We store this rather than the set of nodes # ellipsis nodes. We store this rather than the set of nodes
# that should be an ellipsis because for very large histories # that should be an ellipsis because for very large histories
# we expect this to be significantly smaller. # we expect this to be significantly smaller.
packer._full_nodes = relevant_nodes packer._full_nodes = relevant_nodes
# Maps ellipsis revs to their roots at the changelog level. # Maps ellipsis revs to their roots at the changelog level.
packer._precomputed_ellipsis = ellipsisroots packer._precomputed_ellipsis = ellipsisroots
# Maps CL revs to per-revlog revisions. Cleared in close() at # Maps CL revs to per-revlog revisions. Cleared in close() at
# the end of each group. # the end of each group.
packer._clrev_to_localrev = {} packer._clrev_to_localrev = {}
packer._next_clrev_to_localrev = {} packer._next_clrev_to_localrev = {}
# Maps changelog nodes to changelog revs. Filled in once # Maps changelog nodes to changelog revs. Filled in once
# during changelog stage and then left unmodified. # during changelog stage and then left unmodified.
packer._clnode_to_rev = {} packer._clnode_to_rev = {}
packer._changelog_done = False packer._changelog_done = False
# If true, informs the packer that it is serving shallow content and might
# need to pack file contents not introduced by the changes being packed.
packer._is_shallow = depth is not None
return packer.generate(common, visitnodes, False, source) return packer.generate(common, visitnodes, False, source)