This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8723

fix: prefetch file contents
ClosedPublic

Authored by rdamazio on Jul 10 2020, 2:45 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG797ef6f8295e: fix: prefetch file contents
rHG263cf0f60b70: fix: prefetch file contents
Summary

This prevents the worker subprocesses from contacting the server individually,
which is either inefficient, or leads to problems if the connection is shared
among them.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

rdamazio created this revision.Jul 10 2020, 2:45 AM
pulkit accepted this revision.Jul 10 2020, 6:16 AM
This revision is now accepted and ready to land.Jul 10 2020, 6:16 AM
rdamazio added a commit: rHG263cf0f60b70: fix: prefetch file contents.Jul 10 2020, 6:47 AM
Closed by commit rHG263cf0f60b70: fix: prefetch file contents (authored by rdamazio). · Explain Why
This revision was automatically updated to reflect the committed changes.
rdamazio updated this revision to Diff 21846.Jul 10 2020, 9:20 PM

Revision Contents
Changeset List

PathPackages
Mhgext/fix.py (23 lines)

Diff 21832

hgext/fix.py

revstofix = getrevstofix(ui, repo, opts) revstofix = getrevstofix(ui, repo, opts)
basectxs = getbasectxs(repo, opts, revstofix) basectxs = getbasectxs(repo, opts, revstofix)
workqueue, numitems = getworkqueue( workqueue, numitems = getworkqueue(
ui, repo, pats, opts, revstofix, basectxs ui, repo, pats, opts, revstofix, basectxs
) )
basepaths = getbasepaths(repo, opts, workqueue, basectxs) basepaths = getbasepaths(repo, opts, workqueue, basectxs)
fixers = getfixers(ui) fixers = getfixers(ui)
# Rather than letting each worker independently fetch the files
# (which also would add complications for shared/keepalive
# connections), prefetch them all first.
_prefetchfiles(repo, workqueue, basepaths)
# There are no data dependencies between the workers fixing each file # There are no data dependencies between the workers fixing each file
# revision, so we can use all available parallelism. # revision, so we can use all available parallelism.
def getfixes(items): def getfixes(items):
for rev, path in items: for rev, path in items:
ctx = repo[rev] ctx = repo[rev]
olddata = ctx[path].data() olddata = ctx[path].data()
metadata, newdata = fixfile( metadata, newdata = fixfile(
ui, repo, opts, fixers, ctx, path, basepaths, basectxs[rev] ui, repo, opts, fixers, ctx, path, basepaths, basectxs[rev]
ctx = repo[rev] ctx = repo[rev]
for pctx in ctx.parents(): for pctx in ctx.parents():
if pctx.rev() in basectxs: if pctx.rev() in basectxs:
basectxs[rev].update(basectxs[pctx.rev()]) basectxs[rev].update(basectxs[pctx.rev()])
else: else:
basectxs[rev].add(pctx) basectxs[rev].add(pctx)
return basectxs return basectxs
def _prefetchfiles(repo, workqueue, basepaths):
toprefetch = set()
# Prefetch the files that will be fixed.
for rev, path in workqueue:
if rev == wdirrev:
continue
toprefetch.add((rev, path))
# Prefetch the base contents for lineranges().
for (baserev, fixrev, path), basepath in basepaths.items():
toprefetch.add((baserev, basepath))
if toprefetch:
scmutil.prefetchfiles(repo, [
(rev, scmutil.matchfiles(repo, [path])) for rev, path in toprefetch
])
def fixfile(ui, repo, opts, fixers, fixctx, path, basepaths, basectxs): def fixfile(ui, repo, opts, fixers, fixctx, path, basepaths, basectxs):
"""Run any configured fixers that should affect the file in this context """Run any configured fixers that should affect the file in this context
Returns the file content that results from applying the fixers in some order Returns the file content that results from applying the fixers in some order
starting with the file's content in the fixctx. Fixers that support line starting with the file's content in the fixctx. Fixers that support line
ranges will affect lines that have changed relative to any of the basectxs ranges will affect lines that have changed relative to any of the basectxs
(i.e. they will only avoid lines that are common to all basectxs). (i.e. they will only avoid lines that are common to all basectxs).