This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4859

exchangev2: add progress bar around manifest scanning
ClosedPublic

Authored by indygreg on Oct 3 2018, 12:51 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG7a347d362a45: exchangev2: add progress bar around manifest scanning
Summary

This can take a long time on large repositories. Let's add a progress
bar so we don't have long periods where it isn't obvious what is
going on.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Oct 3 2018, 12:51 PM
martinvonz added a subscriber: martinvonz.Oct 3 2018, 2:42 PM
martinvonz added inline comments.
mercurial/exchangev2.py
344

should call progress.complete() here to remove the progress bar (or use it as a context manager)

indygreg added inline comments.Oct 3 2018, 2:56 PM
mercurial/exchangev2.py
344

Yes we should. I'll send an updated patch in a few minutes.

indygreg updated this revision to Diff 11651.Oct 3 2018, 2:58 PM
indygreg updated this revision to Diff 11655.Oct 3 2018, 4:54 PM
Closed by commit rHG7a347d362a45: exchangev2: add progress bar around manifest scanning (authored by indygreg). · Explain WhyOct 3 2018, 8:07 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/exchangev2.py (28 lines)

Diff 11666

mercurial/exchangev2.py

"""Determine what file nodes are relevant given a set of manifest nodes. """Determine what file nodes are relevant given a set of manifest nodes.
Returns a dict mapping file paths to dicts of file node to first manifest Returns a dict mapping file paths to dicts of file node to first manifest
node. node.
""" """
ml = repo.manifestlog ml = repo.manifestlog
fnodes = collections.defaultdict(dict) fnodes = collections.defaultdict(dict)
progress = repo.ui.makeprogress(
_('scanning manifests'), total=len(manifestnodes))
with progress:
for manifestnode in manifestnodes: for manifestnode in manifestnodes:
m = ml.get(b'', manifestnode) m = ml.get(b'', manifestnode)
# TODO this will pull in unwanted nodes because it takes the storage # TODO this will pull in unwanted nodes because it takes the storage
# delta into consideration. What we really want is something that takes # delta into consideration. What we really want is something that
# the delta between the manifest's parents. And ideally we would # takes the delta between the manifest's parents. And ideally we
# ignore file nodes that are known locally. For now, ignore both # would ignore file nodes that are known locally. For now, ignore
# these limitations. This will result in incremental fetches requesting # both these limitations. This will result in incremental fetches
# data we already have. So this is far from ideal. # requesting data we already have. So this is far from ideal.
md = m.readfast() md = m.readfast()
for path, fnode in md.items(): for path, fnode in md.items():
fnodes[path].setdefault(fnode, manifestnode) fnodes[path].setdefault(fnode, manifestnode)
progress.increment()
return fnodes return fnodes
martinvonzUnsubmitted
Not Done

should call progress.complete() here to remove the progress bar (or use it as a context manager)

martinvonz: should call `progress.complete()` here to remove the progress bar (or use it as a context…
indygregAuthorUnsubmitted
Not Done

Yes we should. I'll send an updated patch in a few minutes.

indygreg: Yes we should. I'll send an updated patch in a few minutes.
def _fetchfiles(repo, tr, remote, fnodes, linkrevs): def _fetchfiles(repo, tr, remote, fnodes, linkrevs):
def iterrevisions(objs, progress): def iterrevisions(objs, progress):
for filerevision in objs: for filerevision in objs:
node = filerevision[b'node'] node = filerevision[b'node']
extrafields = {} extrafields = {}
for field, size in filerevision.get(b'fieldsfollowing', []): for field, size in filerevision.get(b'fieldsfollowing', []):