This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4725

changegroup: port to emitrevisions() (issue5976)
ClosedPublic

Authored by indygreg on Sep 24 2018, 1:15 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG31b7e8e7132e: changegroup: port to emitrevisions() (issue5976)
Summary

We now have a unified API for emitting revision data from a storage
backend. It handles sorting nodes and the complicated delta versus
revision decisions for us.

This commit ports changegroup to that API.

There should be no behavior changes for changegroups not using
ellipsis. And lack of test changes seems to confirm that.

There are some changes for ellipsis mode, however.

Before, when sending an ellipsis revision, we would always send a
fulltext revision (as opposed to a delta). There was a TODO tracking
this open item.

One of the things the emitrevisions() API does for us is figure out
whether we can safely emit a delta. So, it is now possible for
ellipsis revisions to be sent as deltas! (It does this by not
assuming parent/ancestor revisions are available and tracking which
revisions have been sent out.)

Because we eliminated the list of revision delta request objects,
performance has improved substantially:

$ hg perfchangegroupchangelog
before: ! wall 24.348077 comb 24.330000 user 24.140000 sys 0.190000 (best of 3)
after: ! wall 18.245911 comb 18.240000 user 18.100000 sys 0.140000 (best of 3)

That's a lot of overhead for creating a few hundred thousand Python
objects!

This is still a little slower than 4.7. Probably due to 23d582ca
introducing a type for the revision/delta results. There is
potentially room to optimize. But at some point we need to abstract
storage in order to support alternate storage backends. Unfortunately
that means using a Python data structure to represent results. And
unfortunately there is overhead with every new Python object created.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Revision Contents
Changeset List

PathPackages
Mmercurial/changegroup.py (158 lines)

Diff 11333

mercurial/changegroup.py

short, short,
) )
from .thirdparty import ( from .thirdparty import (
attr, attr,
) )
from . import ( from . import (
dagop,
error, error,
match as matchmod, match as matchmod,
mdiff, mdiff,
phases, phases,
pycompat, pycompat,
repository, repository,
revlog,
util, util,
) )
from .utils import ( from .utils import (
interfaceutil, interfaceutil,
) )
_CHANGEGROUPV1_DELTA_HEADER = struct.Struct("20s20s20s20s") _CHANGEGROUPV1_DELTA_HEADER = struct.Struct("20s20s20s20s")
meta = headerfn(delta) meta = headerfn(delta)
yield chunkheader(len(meta) + len(prefix) + len(data)) yield chunkheader(len(meta) + len(prefix) + len(data))
yield meta yield meta
if prefix: if prefix:
yield prefix yield prefix
yield data yield data
def _sortnodesnormal(store, nodes):
"""Sort nodes for changegroup generation and turn into revnums."""
# for generaldelta revlogs, we linearize the revs; this will both be
# much quicker and generate a much smaller bundle
if store._generaldelta:
revs = set(store.rev(n) for n in nodes)
return dagop.linearize(revs, store.parentrevs)
else:
return sorted([store.rev(n) for n in nodes])
def _sortnodesellipsis(store, nodes, cl, lookup): def _sortnodesellipsis(store, nodes, cl, lookup):
"""Sort nodes for changegroup generation and turn into revnums.""" """Sort nodes for changegroup generation."""
# Ellipses serving mode. # Ellipses serving mode.
# #
# In a perfect world, we'd generate better ellipsis-ified graphs # In a perfect world, we'd generate better ellipsis-ified graphs
# for non-changelog revlogs. In practice, we haven't started doing # for non-changelog revlogs. In practice, we haven't started doing
# that yet, so the resulting DAGs for the manifestlog and filelogs # that yet, so the resulting DAGs for the manifestlog and filelogs
# are actually full of bogus parentage on all the ellipsis # are actually full of bogus parentage on all the ellipsis
# nodes. This has the side effect that, while the contents are # nodes. This has the side effect that, while the contents are
# correct, the individual DAGs might be completely out of whack in # correct, the individual DAGs might be completely out of whack in
# a case like 882681bc3166 and its ancestors (back about 10 # a case like 882681bc3166 and its ancestors (back about 10
# revisions or so) in the main hg repo. # revisions or so) in the main hg repo.
# #
# The one invariant we *know* holds is that the new (potentially # The one invariant we *know* holds is that the new (potentially
# bogus) DAG shape will be valid if we order the nodes in the # bogus) DAG shape will be valid if we order the nodes in the
# order that they're introduced in dramatis personae by the # order that they're introduced in dramatis personae by the
# changelog, so what we do is we sort the non-changelog histories # changelog, so what we do is we sort the non-changelog histories
# by the order in which they are used by the changelog. # by the order in which they are used by the changelog.
key = lambda n: cl.rev(lookup(n)) key = lambda n: cl.rev(lookup(n))
return [store.rev(n) for n in sorted(nodes, key=key)] return sorted(nodes, key=key)
def _makenarrowdeltarequest(cl, store, ischangelog, rev, node, linkrev, def _resolvenarrowrevisioninfo(cl, store, ischangelog, rev, linkrev,
linknode, clrevtolocalrev, fullclnodes, linknode, clrevtolocalrev, fullclnodes,
precomputedellipsis): precomputedellipsis):
linkparents = precomputedellipsis[linkrev] linkparents = precomputedellipsis[linkrev]
def local(clrev): def local(clrev):
"""Turn a changelog revnum into a local revnum. """Turn a changelog revnum into a local revnum.
The ellipsis dag is stored as revnums on the changelog, The ellipsis dag is stored as revnums on the changelog,
but when we're producing ellipsis entries for but when we're producing ellipsis entries for
non-changelog revlogs, we need to turn those numbers into non-changelog revlogs, we need to turn those numbers into
something local. This does that for us, and during the something local. This does that for us, and during the
elif len(linkparents) == 1: elif len(linkparents) == 1:
p1, = sorted(local(p) for p in linkparents) p1, = sorted(local(p) for p in linkparents)
p2 = nullrev p2 = nullrev
else: else:
p1, p2 = sorted(local(p) for p in linkparents) p1, p2 = sorted(local(p) for p in linkparents)
p1node, p2node = store.node(p1), store.node(p2) p1node, p2node = store.node(p1), store.node(p2)
# TODO: try and actually send deltas for ellipsis data blocks return p1node, p2node, linknode
return revisiondeltarequest(
node=node,
p1node=p1node,
p2node=p2node,
linknode=linknode,
basenode=nullid,
ellipsis=True,
)
def deltagroup(repo, store, nodes, ischangelog, lookup, forcedeltaparentprev, def deltagroup(repo, store, nodes, ischangelog, lookup, forcedeltaparentprev,
topic=None, topic=None,
ellipses=False, clrevtolocalrev=None, fullclnodes=None, ellipses=False, clrevtolocalrev=None, fullclnodes=None,
precomputedellipsis=None): precomputedellipsis=None):
"""Calculate deltas for a set of revisions. """Calculate deltas for a set of revisions.
Is a generator of ``revisiondelta`` instances. Is a generator of ``revisiondelta`` instances.
If topic is not None, progress detail will be generated using this If topic is not None, progress detail will be generated using this
topic name (e.g. changesets, manifests, etc). topic name (e.g. changesets, manifests, etc).
""" """
if not nodes: if not nodes:
return return
# We perform two passes over the revisions whose data we will emit.
#
# In the first pass, we obtain information about the deltas that will
# be generated. This involves computing linknodes and adjusting the
# request to take shallow fetching into account. The end result of
# this pass is a list of "request" objects stating which deltas
# to obtain.
#
# The second pass is simply resolving the requested deltas.
cl = repo.changelog cl = repo.changelog
if ischangelog: if ischangelog:
# Changelog doesn't benefit from reordering revisions. So send # `hg log` shows changesets in storage order. To preserve order
# out revisions in store order. # across clones, send out changesets in storage order.
# TODO the API would be cleaner if this were controlled by the nodesorder = 'storage'
# store producing the deltas.
revs = sorted(cl.rev(n) for n in nodes)
elif ellipses: elif ellipses:
revs = _sortnodesellipsis(store, nodes, cl, lookup) nodes = _sortnodesellipsis(store, nodes, cl, lookup)
nodesorder = 'nodes'
else: else:
revs = _sortnodesnormal(store, nodes) nodesorder = None
# In the first pass, collect info about the deltas we'll be
# generating.
requests = []
# Add the parent of the first rev. # Perform ellipses filtering and revision massaging. We do this before
revs.insert(0, store.parentrevs(revs[0])[0]) # emitrevisions() because a) filtering out revisions creates less work
# for emitrevisions() b) dropping revisions would break emitrevisions()'s
for i in pycompat.xrange(len(revs) - 1): # assumptions about delta choices and we would possibly send a delta
prev = revs[i] # referencing a missing base revision.
curr = revs[i + 1] #
# Also, calling lookup() has side-effects with regards to populating
# data structures. If we don't call lookup() for each node or if we call
# lookup() after the first pass through each node, things can break -
# possibly intermittently depending on the python hash seed! For that
# reason, we store a mapping of all linknodes during the initial node
# pass rather than use lookup() on the output side.
if ellipses:
filtered = []
adjustedparents = {}
linknodes = {}
node = store.node(curr) for node in nodes:
rev = store.rev(node)
linknode = lookup(node) linknode = lookup(node)
p1node, p2node = store.parents(node)
if ellipses:
linkrev = cl.rev(linknode) linkrev = cl.rev(linknode)
clrevtolocalrev[linkrev] = curr clrevtolocalrev[linkrev] = rev
# This is a node to send in full, because the changeset it # If linknode is in fullclnodes, it means the corresponding
# corresponds to was a full changeset. # changeset was a full changeset and is being sent unaltered.
if linknode in fullclnodes: if linknode in fullclnodes:
requests.append(revisiondeltarequest( linknodes[node] = linknode
node=node,
p1node=p1node,
p2node=p2node,
linknode=linknode,
basenode=None,
))
# If the corresponding changeset wasn't in the set computed
# as relevant to us, it should be dropped outright.
elif linkrev not in precomputedellipsis: elif linkrev not in precomputedellipsis:
pass continue
else: else:
requests.append(_makenarrowdeltarequest( # We could probably do this later and avoid the dict
cl, store, ischangelog, curr, node, linkrev, linknode, # holding state. But it likely doesn't matter.
clrevtolocalrev, fullclnodes, p1node, p2node, linknode = _resolvenarrowrevisioninfo(
precomputedellipsis)) cl, store, ischangelog, rev, linkrev, linknode,
else: clrevtolocalrev, fullclnodes, precomputedellipsis)
requests.append(revisiondeltarequest(
node=node, adjustedparents[node] = (p1node, p2node)
p1node=p1node, linknodes[node] = linknode
p2node=p2node,
linknode=linknode, filtered.append(node)
basenode=store.node(prev) if forcedeltaparentprev else None,
)) nodes = filtered
# We expect the first pass to be fast, so we only engage the progress # We expect the first pass to be fast, so we only engage the progress
# meter for constructing the revision deltas. # meter for constructing the revision deltas.
progress = None progress = None
if topic is not None: if topic is not None:
progress = repo.ui.makeprogress(topic, unit=_('chunks'), progress = repo.ui.makeprogress(topic, unit=_('chunks'),
total=len(requests)) total=len(nodes))
for i, delta in enumerate(store.emitrevisiondeltas(requests)): revisions = store.emitrevisions(
nodes,
nodesorder=nodesorder,
revisiondata=True,
assumehaveparentrevisions=not ellipses,
deltaprevious=forcedeltaparentprev)
for i, revision in enumerate(revisions):
if progress: if progress:
progress.update(i + 1) progress.update(i + 1)
yield delta if ellipses:
linknode = linknodes[revision.node]
if revision.node in adjustedparents:
p1node, p2node = adjustedparents[revision.node]
revision.p1node = p1node
revision.p2node = p2node
revision.flags |= revlog.REVIDX_ELLIPSIS
else:
linknode = lookup(revision.node)
revision.linknode = linknode
yield revision
if progress: if progress:
progress.complete() progress.complete()
class cgpacker(object): class cgpacker(object):
def __init__(self, repo, filematcher, version, def __init__(self, repo, filematcher, version,
builddeltaheader, manifestsend, builddeltaheader, manifestsend,
forcedeltaparentprev=False, forcedeltaparentprev=False,