This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4137

changegroup: factor changelogdone into an argument
ClosedPublic

Authored by indygreg on Aug 6 2018, 3:51 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG4a202bccafcf: changegroup: factor changelogdone into an argument
Summary

The variable was basically tracking whether the current operation
is being performed against the changelog or something else. So
let's just pass such a flag to everything that needs to access it.

I'm still not a huge fan of building changelog awareness into
low-level functions like revision delta generation. But passing
an argument is strictly better than state on the packer instance.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Aug 6 2018, 3:51 PM
Closed by commit rHG4a202bccafcf: changegroup: factor changelogdone into an argument (authored by indygreg). · Explain WhyAug 9 2018, 2:15 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/changegroup.py (38 lines)

Diff 10008

mercurial/changegroup.py

self._repo = repo self._repo = repo
if self._repo.ui.verbose and not self._repo.ui.debugflag: if self._repo.ui.verbose and not self._repo.ui.debugflag:
self._verbosenote = self._repo.ui.note self._verbosenote = self._repo.ui.note
else: else:
self._verbosenote = lambda s: None self._verbosenote = lambda s: None
# TODO the functionality keyed off of this should probably be
# controlled via arguments to group() that influence behavior.
self._changelogdone = False
# Maps CL revs to per-revlog revisions. Cleared in close() at # Maps CL revs to per-revlog revisions. Cleared in close() at
# the end of each group. # the end of each group.
self._clrevtolocalrev = {} self._clrevtolocalrev = {}
self._nextclrevtolocalrev = {} self._nextclrevtolocalrev = {}
# Maps changelog nodes to changelog revs. Filled in once # Maps changelog nodes to changelog revs. Filled in once
# during changelog stage and then left unmodified. # during changelog stage and then left unmodified.
self._clnodetorev = {} self._clnodetorev = {}
def _close(self): def _close(self):
# Ellipses serving mode. # Ellipses serving mode.
self._clrevtolocalrev.clear() self._clrevtolocalrev.clear()
if self._nextclrevtolocalrev: if self._nextclrevtolocalrev:
self.clrevtolocalrev = self._nextclrevtolocalrev self.clrevtolocalrev = self._nextclrevtolocalrev
self._nextclrevtolocalrev.clear() self._nextclrevtolocalrev.clear()
return closechunk() return closechunk()
def _fileheader(self, fname): def _fileheader(self, fname):
return chunkheader(len(fname)) + fname return chunkheader(len(fname)) + fname
# Extracted both for clarity and for overriding in extensions. # Extracted both for clarity and for overriding in extensions.
def _sortgroup(self, store, nodelist, lookup): def _sortgroup(self, store, ischangelog, nodelist, lookup):
"""Sort nodes for change group and turn them into revnums.""" """Sort nodes for change group and turn them into revnums."""
# Ellipses serving mode. # Ellipses serving mode.
# #
# In a perfect world, we'd generate better ellipsis-ified graphs # In a perfect world, we'd generate better ellipsis-ified graphs
# for non-changelog revlogs. In practice, we haven't started doing # for non-changelog revlogs. In practice, we haven't started doing
# that yet, so the resulting DAGs for the manifestlog and filelogs # that yet, so the resulting DAGs for the manifestlog and filelogs
# are actually full of bogus parentage on all the ellipsis # are actually full of bogus parentage on all the ellipsis
# nodes. This has the side effect that, while the contents are # nodes. This has the side effect that, while the contents are
# correct, the individual DAGs might be completely out of whack in # correct, the individual DAGs might be completely out of whack in
# a case like 882681bc3166 and its ancestors (back about 10 # a case like 882681bc3166 and its ancestors (back about 10
# revisions or so) in the main hg repo. # revisions or so) in the main hg repo.
# #
# The one invariant we *know* holds is that the new (potentially # The one invariant we *know* holds is that the new (potentially
# bogus) DAG shape will be valid if we order the nodes in the # bogus) DAG shape will be valid if we order the nodes in the
# order that they're introduced in dramatis personae by the # order that they're introduced in dramatis personae by the
# changelog, so what we do is we sort the non-changelog histories # changelog, so what we do is we sort the non-changelog histories
# by the order in which they are used by the changelog. # by the order in which they are used by the changelog.
if self._ellipses and self._changelogdone: if self._ellipses and not ischangelog:
key = lambda n: self._clnodetorev[lookup(n)] key = lambda n: self._clnodetorev[lookup(n)]
return [store.rev(n) for n in sorted(nodelist, key=key)] return [store.rev(n) for n in sorted(nodelist, key=key)]
# for generaldelta revlogs, we linearize the revs; this will both be # for generaldelta revlogs, we linearize the revs; this will both be
# much quicker and generate a much smaller bundle # much quicker and generate a much smaller bundle
if (store._generaldelta and self._reorder is None) or self._reorder: if (store._generaldelta and self._reorder is None) or self._reorder:
dag = dagutil.revlogdag(store) dag = dagutil.revlogdag(store)
return dag.linearize(set(store.rev(n) for n in nodelist)) return dag.linearize(set(store.rev(n) for n in nodelist))
else: else:
return sorted([store.rev(n) for n in nodelist]) return sorted([store.rev(n) for n in nodelist])
def group(self, nodelist, store, lookup, units=None): def group(self, nodelist, store, ischangelog, lookup, units=None):
"""Calculate a delta group, yielding a sequence of changegroup chunks """Calculate a delta group, yielding a sequence of changegroup chunks
(strings). (strings).
Given a list of changeset revs, return a set of deltas and Given a list of changeset revs, return a set of deltas and
metadata corresponding to nodes. The first delta is metadata corresponding to nodes. The first delta is
first parent(nodelist[0]) -> nodelist[0], the receiver is first parent(nodelist[0]) -> nodelist[0], the receiver is
guaranteed to have this parent as it has all history before guaranteed to have this parent as it has all history before
these changesets. In the case firstparent is nullrev the these changesets. In the case firstparent is nullrev the
changegroup starts with a full revision. changegroup starts with a full revision.
If units is not None, progress detail will be generated, units specifies If units is not None, progress detail will be generated, units specifies
the type of revlog that is touched (changelog, manifest, etc.). the type of revlog that is touched (changelog, manifest, etc.).
""" """
# if we don't have any revisions touched by these changesets, bail # if we don't have any revisions touched by these changesets, bail
if len(nodelist) == 0: if len(nodelist) == 0:
yield self._close() yield self._close()
return return
revs = self._sortgroup(store, nodelist, lookup) revs = self._sortgroup(store, ischangelog, nodelist, lookup)
# add the parent of the first rev # add the parent of the first rev
p = store.parentrevs(revs[0])[0] p = store.parentrevs(revs[0])[0]
revs.insert(0, p) revs.insert(0, p)
# build deltas # build deltas
progress = None progress = None
if units is not None: if units is not None:
progress = self._repo.ui.makeprogress(_('bundling'), unit=units, progress = self._repo.ui.makeprogress(_('bundling'), unit=units,
total=(len(revs) - 1)) total=(len(revs) - 1))
for r in pycompat.xrange(len(revs) - 1): for r in pycompat.xrange(len(revs) - 1):
if progress: if progress:
progress.update(r + 1) progress.update(r + 1)
prev, curr = revs[r], revs[r + 1] prev, curr = revs[r], revs[r + 1]
linknode = lookup(store.node(curr)) linknode = lookup(store.node(curr))
for c in self._revchunk(store, curr, prev, linknode): for c in self._revchunk(store, ischangelog, curr, prev, linknode):
yield c yield c
if progress: if progress:
progress.complete() progress.complete()
yield self._close() yield self._close()
# filter any nodes that claim to be part of the known set # filter any nodes that claim to be part of the known set
def _prune(self, store, missing, commonrevs): def _prune(self, store, missing, commonrevs):
""" """
if dir: if dir:
assert self.version == b'03' assert self.version == b'03'
yield self._fileheader(dir) yield self._fileheader(dir)
# TODO violates storage abstractions by assuming revlogs. # TODO violates storage abstractions by assuming revlogs.
dirlog = self._repo.manifestlog._revlog.dirlog(dir) dirlog = self._repo.manifestlog._revlog.dirlog(dir)
for chunk in self.group(mfnodes, dirlog, lookuplinknode, for chunk in self.group(mfnodes, dirlog, False, lookuplinknode,
units=_('manifests')): units=_('manifests')):
yield chunk yield chunk
def generate(self, commonrevs, clnodes, fastpathlinkrev, source): def generate(self, commonrevs, clnodes, fastpathlinkrev, source):
"""Yield a sequence of changegroup byte chunks.""" """Yield a sequence of changegroup byte chunks."""
repo = self._repo repo = self._repo
cl = repo.changelog cl = repo.changelog
self._verbosenote(_('uncompressed size of bundle content:\n')) self._verbosenote(_('uncompressed size of bundle content:\n'))
size = 0 size = 0
clstate, chunks = self._generatechangelog(cl, clnodes) clstate, chunks = self._generatechangelog(cl, clnodes)
for chunk in chunks: for chunk in chunks:
size += len(chunk) size += len(chunk)
yield chunk yield chunk
self._verbosenote(_('%8.i (changelog)\n') % size) self._verbosenote(_('%8.i (changelog)\n') % size)
self._changelogdone = True
clrevorder = clstate['clrevorder'] clrevorder = clstate['clrevorder']
mfs = clstate['mfs'] mfs = clstate['mfs']
changedfiles = clstate['changedfiles'] changedfiles = clstate['changedfiles']
# We need to make sure that the linkrev in the changegroup refers to # We need to make sure that the linkrev in the changegroup refers to
# the first changeset that introduced the manifest or file revision. # the first changeset that introduced the manifest or file revision.
# The fastpath is usually safer than the slowpath, because the filelogs # The fastpath is usually safer than the slowpath, because the filelogs
# are walked in revlog order. # are walked in revlog order.
return x return x
state = { state = {
'clrevorder': clrevorder, 'clrevorder': clrevorder,
'mfs': mfs, 'mfs': mfs,
'changedfiles': changedfiles, 'changedfiles': changedfiles,
} }
gen = self.group(nodes, cl, lookupcl, units=_('changesets')) gen = self.group(nodes, cl, True, lookupcl, units=_('changesets'))
return state, gen return state, gen
def generatemanifests(self, commonrevs, clrevorder, fastpathlinkrev, mfs, def generatemanifests(self, commonrevs, clrevorder, fastpathlinkrev, mfs,
fnodes, source): fnodes, source):
"""Returns an iterator of changegroup chunks containing manifests. """Returns an iterator of changegroup chunks containing manifests.
`source` is unused here, but is used by extensions like remotefilelog to `source` is unused here, but is used by extensions like remotefilelog to
return linkrevnodes[x] return linkrevnodes[x]
filenodes = self._prune(filerevlog, linkrevnodes, commonrevs) filenodes = self._prune(filerevlog, linkrevnodes, commonrevs)
if filenodes: if filenodes:
progress.update(i + 1, item=fname) progress.update(i + 1, item=fname)
h = self._fileheader(fname) h = self._fileheader(fname)
size = len(h) size = len(h)
yield h yield h
for chunk in self.group(filenodes, filerevlog, lookupfilelog): for chunk in self.group(filenodes, filerevlog, False,
lookupfilelog):
size += len(chunk) size += len(chunk)
yield chunk yield chunk
self._verbosenote(_('%8.i %s\n') % (size, fname)) self._verbosenote(_('%8.i %s\n') % (size, fname))
progress.complete() progress.complete()
def _revchunk(self, store, rev, prev, linknode): def _revchunk(self, store, ischangelog, rev, prev, linknode):
if self._ellipses: if self._ellipses:
fn = self._revisiondeltanarrow fn = self._revisiondeltanarrow
else: else:
fn = self._revisiondeltanormal fn = self._revisiondeltanormal
delta = fn(store, rev, prev, linknode) delta = fn(store, ischangelog, rev, prev, linknode)
if not delta: if not delta:
return return
meta = self._builddeltaheader(delta) meta = self._builddeltaheader(delta)
l = len(meta) + sum(len(x) for x in delta.deltachunks) l = len(meta) + sum(len(x) for x in delta.deltachunks)
yield chunkheader(l) yield chunkheader(l)
yield meta yield meta
for x in delta.deltachunks: for x in delta.deltachunks:
yield x yield x
def _revisiondeltanormal(self, store, rev, prev, linknode): def _revisiondeltanormal(self, store, ischangelog, rev, prev, linknode):
node = store.node(rev) node = store.node(rev)
p1, p2 = store.parentrevs(rev) p1, p2 = store.parentrevs(rev)
base = self._deltaparentfn(store, rev, p1, p2, prev) base = self._deltaparentfn(store, rev, p1, p2, prev)
prefix = '' prefix = ''
if store.iscensored(base) or store.iscensored(rev): if store.iscensored(base) or store.iscensored(rev):
try: try:
delta = store.revision(node, raw=True) delta = store.revision(node, raw=True)
p1node=p1n, p1node=p1n,
p2node=p2n, p2node=p2n,
basenode=store.node(base), basenode=store.node(base),
linknode=linknode, linknode=linknode,
flags=store.flags(rev), flags=store.flags(rev),
deltachunks=(prefix, delta), deltachunks=(prefix, delta),
) )
def _revisiondeltanarrow(self, store, rev, prev, linknode): def _revisiondeltanarrow(self, store, ischangelog, rev, prev, linknode):
# build up some mapping information that's useful later. See # build up some mapping information that's useful later. See
# the local() nested function below. # the local() nested function below.
if not self._changelogdone: if ischangelog:
self._clnodetorev[linknode] = rev self._clnodetorev[linknode] = rev
linkrev = rev linkrev = rev
self._clrevtolocalrev[linkrev] = rev self._clrevtolocalrev[linkrev] = rev
else: else:
linkrev = self._clnodetorev[linknode] linkrev = self._clnodetorev[linknode]
self._clrevtolocalrev[linkrev] = rev self._clrevtolocalrev[linkrev] = rev
# This is a node to send in full, because the changeset it # This is a node to send in full, because the changeset it
# corresponds to was a full changeset. # corresponds to was a full changeset.
if linknode in self._fullnodes: if linknode in self._fullnodes:
return self._revisiondeltanormal(store, rev, prev, linknode) return self._revisiondeltanormal(store, ischangelog, rev, prev,
linknode)
# At this point, a node can either be one we should skip or an # At this point, a node can either be one we should skip or an
# ellipsis. If it's not an ellipsis, bail immediately. # ellipsis. If it's not an ellipsis, bail immediately.
if linkrev not in self._precomputedellipsis: if linkrev not in self._precomputedellipsis:
return return
linkparents = self._precomputedellipsis[linkrev] linkparents = self._precomputedellipsis[linkrev]
def local(clrev): def local(clrev):
"""Turn a changelog revnum into a local revnum. """Turn a changelog revnum into a local revnum.
The ellipsis dag is stored as revnums on the changelog, The ellipsis dag is stored as revnums on the changelog,
but when we're producing ellipsis entries for but when we're producing ellipsis entries for
non-changelog revlogs, we need to turn those numbers into non-changelog revlogs, we need to turn those numbers into
something local. This does that for us, and during the something local. This does that for us, and during the
changelog sending phase will also expand the stored changelog sending phase will also expand the stored
mappings as needed. mappings as needed.
""" """
if clrev == nullrev: if clrev == nullrev:
return nullrev return nullrev
if not self._changelogdone: if ischangelog:
# If we're doing the changelog, it's possible that we # If we're doing the changelog, it's possible that we
# have a parent that is already on the client, and we # have a parent that is already on the client, and we
# need to store some extra mapping information so that # need to store some extra mapping information so that
# our contained ellipsis nodes will be able to resolve # our contained ellipsis nodes will be able to resolve
# their parents. # their parents.
if clrev not in self._clrevtolocalrev: if clrev not in self._clrevtolocalrev:
clnode = store.node(clrev) clnode = store.node(clrev)
self._clnodetorev[clnode] = clrev self._clnodetorev[clnode] = clrev