This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4133

changegroup: factor changelog chunk generation into own function
ClosedPublic

Authored by indygreg on Aug 6 2018, 3:51 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGf7228c907ef4: changegroup: factor changelog chunk generation into own function
Summary

We have separate functions for generating manifests and filelogs.
Let's split changelog into its own function so things are consistent.

As part of this, we refactor the code slightly. Before, the
changelog linknode callback was updating state on variables
inherited via a closure. Since the closure is now separate from
generate(), we need to a way pass state between generate() and
_generatechangelog(). The return value of _generatechangelog()
is a 2-tuple where the first item is a dict containing accumulated
state. We then alias some of its members into the scope of
generate() to reduce code churn.

I will be converting other functions to a similar pattern in future
commits.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Aug 6 2018, 3:51 PM
Closed by commit rHGf7228c907ef4: changegroup: factor changelog chunk generation into own function (authored by indygreg). · Explain WhyAug 9 2018, 2:15 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/changegroup.py (166 lines)

Diff 10004

mercurial/changegroup.py

# TODO violates storage abstractions by assuming revlogs. # TODO violates storage abstractions by assuming revlogs.
dirlog = self._repo.manifestlog._revlog.dirlog(dir) dirlog = self._repo.manifestlog._revlog.dirlog(dir)
for chunk in self.group(mfnodes, dirlog, lookuplinknode, for chunk in self.group(mfnodes, dirlog, lookuplinknode,
units=_('manifests')): units=_('manifests')):
yield chunk yield chunk
def generate(self, commonrevs, clnodes, fastpathlinkrev, source): def generate(self, commonrevs, clnodes, fastpathlinkrev, source):
'''yield a sequence of changegroup chunks (strings)''' """Yield a sequence of changegroup byte chunks."""
repo = self._repo repo = self._repo
cl = repo.changelog cl = repo.changelog
clrevorder = {}
mfs = {} # needed manifests
fnodes = {} # needed file nodes
mfl = repo.manifestlog
# TODO violates storage abstraction.
mfrevlog = mfl._revlog
changedfiles = set()
# Callback for the changelog, used to collect changed files and
# manifest nodes.
# Returns the linkrev node (identity in the changelog case).
def lookupcl(x):
c = cl.read(x)
clrevorder[x] = len(clrevorder)
if self._ellipses:
# Only update mfs if x is going to be sent. Otherwise we
# end up with bogus linkrevs specified for manifests and
# we skip some manifest nodes that we should otherwise
# have sent.
if (x in self._fullnodes
or cl.rev(x) in self._precomputedellipsis):
n = c[0]
# Record the first changeset introducing this manifest
# version.
mfs.setdefault(n, x)
# Set this narrow-specific dict so we have the lowest
# manifest revnum to look up for this cl revnum. (Part of
# mapping changelog ellipsis parents to manifest ellipsis
# parents)
self._nextclrevtolocalrev.setdefault(cl.rev(x),
mfrevlog.rev(n))
# We can't trust the changed files list in the changeset if the
# client requested a shallow clone.
if self._isshallow:
changedfiles.update(mfl[c[0]].read().keys())
else:
changedfiles.update(c[3])
else:
n = c[0]
# record the first changeset introducing this manifest version
mfs.setdefault(n, x)
# Record a complete list of potentially-changed files in
# this manifest.
changedfiles.update(c[3])
return x
self._verbosenote(_('uncompressed size of bundle content:\n')) self._verbosenote(_('uncompressed size of bundle content:\n'))
size = 0 size = 0
for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets')):
clstate, chunks = self._generatechangelog(cl, clnodes)
for chunk in chunks:
size += len(chunk) size += len(chunk)
yield chunk yield chunk
self._verbosenote(_('%8.i (changelog)\n') % size) self._verbosenote(_('%8.i (changelog)\n') % size)
clrevorder = clstate['clrevorder']
mfs = clstate['mfs']
changedfiles = clstate['changedfiles']
# We need to make sure that the linkrev in the changegroup refers to # We need to make sure that the linkrev in the changegroup refers to
# the first changeset that introduced the manifest or file revision. # the first changeset that introduced the manifest or file revision.
# The fastpath is usually safer than the slowpath, because the filelogs # The fastpath is usually safer than the slowpath, because the filelogs
# are walked in revlog order. # are walked in revlog order.
# #
# When taking the slowpath with reorder=None and the manifest revlog # When taking the slowpath with reorder=None and the manifest revlog
# uses generaldelta, the manifest may be walked in the "wrong" order. # uses generaldelta, the manifest may be walked in the "wrong" order.
# Without 'clrevorder', we would get an incorrect linkrev (see fix in # Without 'clrevorder', we would get an incorrect linkrev (see fix in
# cc0ff93d0c0c). # cc0ff93d0c0c).
# #
# When taking the fastpath, we are only vulnerable to reordering # When taking the fastpath, we are only vulnerable to reordering
# of the changelog itself. The changelog never uses generaldelta, so # of the changelog itself. The changelog never uses generaldelta, so
# it is only reordered when reorder=True. To handle this case, we # it is only reordered when reorder=True. To handle this case, we
# simply take the slowpath, which already has the 'clrevorder' logic. # simply take the slowpath, which already has the 'clrevorder' logic.
# This was also fixed in cc0ff93d0c0c. # This was also fixed in cc0ff93d0c0c.
fastpathlinkrev = fastpathlinkrev and not self._reorder fastpathlinkrev = fastpathlinkrev and not self._reorder
# Treemanifests don't work correctly with fastpathlinkrev # Treemanifests don't work correctly with fastpathlinkrev
# either, because we don't discover which directory nodes to # either, because we don't discover which directory nodes to
# send along with files. This could probably be fixed. # send along with files. This could probably be fixed.
fastpathlinkrev = fastpathlinkrev and ( fastpathlinkrev = fastpathlinkrev and (
'treemanifest' not in repo.requirements) 'treemanifest' not in repo.requirements)
fnodes = {} # needed file nodes
for chunk in self.generatemanifests(commonrevs, clrevorder, for chunk in self.generatemanifests(commonrevs, clrevorder,
fastpathlinkrev, mfs, fnodes, source): fastpathlinkrev, mfs, fnodes, source):
yield chunk yield chunk
if self._ellipses: if self._ellipses:
mfdicts = None mfdicts = None
if self._isshallow: if self._isshallow:
mfdicts = [(self._repo.manifestlog[n].read(), lr) mfdicts = [(self._repo.manifestlog[n].read(), lr)
source): source):
yield chunk yield chunk
yield self._close() yield self._close()
if clnodes: if clnodes:
repo.hook('outgoing', node=hex(clnodes[0]), source=source) repo.hook('outgoing', node=hex(clnodes[0]), source=source)
def _generatechangelog(self, cl, nodes):
"""Generate data for changelog chunks.
Returns a 2-tuple of a dict containing state and an iterable of
byte chunks. The state will not be fully populated until the
chunk stream has been fully consumed.
"""
clrevorder = {}
mfs = {} # needed manifests
mfl = self._repo.manifestlog
# TODO violates storage abstraction.
mfrevlog = mfl._revlog
changedfiles = set()
# Callback for the changelog, used to collect changed files and
# manifest nodes.
# Returns the linkrev node (identity in the changelog case).
def lookupcl(x):
c = cl.read(x)
clrevorder[x] = len(clrevorder)
if self._ellipses:
# Only update mfs if x is going to be sent. Otherwise we
# end up with bogus linkrevs specified for manifests and
# we skip some manifest nodes that we should otherwise
# have sent.
if (x in self._fullnodes
or cl.rev(x) in self._precomputedellipsis):
n = c[0]
# Record the first changeset introducing this manifest
# version.
mfs.setdefault(n, x)
# Set this narrow-specific dict so we have the lowest
# manifest revnum to look up for this cl revnum. (Part of
# mapping changelog ellipsis parents to manifest ellipsis
# parents)
self._nextclrevtolocalrev.setdefault(cl.rev(x),
mfrevlog.rev(n))
# We can't trust the changed files list in the changeset if the
# client requested a shallow clone.
if self._isshallow:
changedfiles.update(mfl[c[0]].read().keys())
else:
changedfiles.update(c[3])
else:
n = c[0]
# record the first changeset introducing this manifest version
mfs.setdefault(n, x)
# Record a complete list of potentially-changed files in
# this manifest.
changedfiles.update(c[3])
return x
state = {
'clrevorder': clrevorder,
'mfs': mfs,
'changedfiles': changedfiles,
}
gen = self.group(nodes, cl, lookupcl, units=_('changesets'))
return state, gen
def generatemanifests(self, commonrevs, clrevorder, fastpathlinkrev, mfs, def generatemanifests(self, commonrevs, clrevorder, fastpathlinkrev, mfs,
fnodes, source): fnodes, source):
"""Returns an iterator of changegroup chunks containing manifests. """Returns an iterator of changegroup chunks containing manifests.
`source` is unused here, but is used by extensions like remotefilelog to `source` is unused here, but is used by extensions like remotefilelog to
change what is sent based in pulls vs pushes, etc. change what is sent based in pulls vs pushes, etc.
""" """
repo = self._repo repo = self._repo