( )⚙ D5733 changegroup: don't try to build changelog chunks if not required

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5733

changegroup: don't try to build changelog chunks if not required
ClosedPublic

Authored by pulkit on Jan 29 2019, 8:07 AM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGfa7d61f9c512: changegroup: don't try to build changelog chunks if not required
Summary

When we extend a narrow clone without ellipsis, we don't download changelog
information because that's already present with the client. However we still try
to build that chunk stream. Building that chunk stream involves calling a lookup
function and store.emitrevisions() API. The lookup function is called len(cl)
number of times.

On large repositories, where len(cl) is in millions, calling that lookup
function is not a good idea. Also it's not required to use the
store.emitrevisons() API because we already have nodes present which we can use.

This patch short-circuits state building logic if we are processing a
non-ellipsis case and changelog is not required.

This saves up ~20 seconds on our internal repo for a single extend call.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

pulkit created this revision.Jan 29 2019, 8:07 AM
Closed by commit rHGfa7d61f9c512: changegroup: don't try to build changelog chunks if not required (authored by pulkit). · Explain WhyJan 29 2019, 3:16 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/changegroup.py (31 lines)

Diff 13546

mercurial/changegroup.py

""" """
repo = self._repo repo = self._repo
cl = repo.changelog cl = repo.changelog
self._verbosenote(_('uncompressed size of bundle content:\n')) self._verbosenote(_('uncompressed size of bundle content:\n'))
size = 0 size = 0
clstate, deltas = self._generatechangelog(cl, clnodes) clstate, deltas = self._generatechangelog(cl, clnodes,
generate=changelog)
for delta in deltas: for delta in deltas:
if changelog:
for chunk in _revisiondeltatochunks(delta, for chunk in _revisiondeltatochunks(delta,
self._builddeltaheader): self._builddeltaheader):
size += len(chunk) size += len(chunk)
yield chunk yield chunk
close = closechunk() close = closechunk()
size += len(close) size += len(close)
yield closechunk() yield closechunk()
self._verbosenote(_('%8.i (changelog)\n') % size) self._verbosenote(_('%8.i (changelog)\n') % size)
clrevorder = clstate['clrevorder'] clrevorder = clstate['clrevorder']
self._verbosenote(_('%8.i %s\n') % (size, path)) self._verbosenote(_('%8.i %s\n') % (size, path))
yield closechunk() yield closechunk()
if clnodes: if clnodes:
repo.hook('outgoing', node=hex(clnodes[0]), source=source) repo.hook('outgoing', node=hex(clnodes[0]), source=source)
def _generatechangelog(self, cl, nodes): def _generatechangelog(self, cl, nodes, generate=True):
"""Generate data for changelog chunks. """Generate data for changelog chunks.
Returns a 2-tuple of a dict containing state and an iterable of Returns a 2-tuple of a dict containing state and an iterable of
byte chunks. The state will not be fully populated until the byte chunks. The state will not be fully populated until the
chunk stream has been fully consumed. chunk stream has been fully consumed.
if generate is False, the state will be fully populated and no chunk
stream will be yielded
""" """
clrevorder = {} clrevorder = {}
manifests = {} manifests = {}
mfl = self._repo.manifestlog mfl = self._repo.manifestlog
changedfiles = set() changedfiles = set()
clrevtomanifestrev = {} clrevtomanifestrev = {}
state = { state = {
'clrevorder': clrevorder, 'clrevorder': clrevorder,
'manifests': manifests, 'manifests': manifests,
'changedfiles': changedfiles, 'changedfiles': changedfiles,
'clrevtomanifestrev': clrevtomanifestrev, 'clrevtomanifestrev': clrevtomanifestrev,
} }
if not (generate or self._ellipses):
# sort the nodes in storage order
nodes = sorted(nodes, key=cl.rev)
for node in nodes:
c = cl.changelogrevision(node)
clrevorder[node] = len(clrevorder)
# record the first changeset introducing this manifest version
manifests.setdefault(c.manifest, node)
# Record a complete list of potentially-changed files in
# this manifest.
changedfiles.update(c.files)
return state, ()
# Callback for the changelog, used to collect changed files and # Callback for the changelog, used to collect changed files and
# manifest nodes. # manifest nodes.
# Returns the linkrev node (identity in the changelog case). # Returns the linkrev node (identity in the changelog case).
def lookupcl(x): def lookupcl(x):
c = cl.changelogrevision(x) c = cl.changelogrevision(x)
clrevorder[x] = len(clrevorder) clrevorder[x] = len(clrevorder)
if self._ellipses: if self._ellipses: