This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8721

scmutil: allowing different files to be prefetched per revision
ClosedPublic

Authored by rdamazio on Jul 10 2020, 2:45 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHGa56ba57c837d: scmutil: allowing different files to be prefetched per revision
rHG2fac78d5ae87: scmutil: allowing different files to be prefetched per revision
Summary

The old API takes a list of revision separate from the file matcher, and thus
provides no way to fetch different sets of files from each revision. In
preparation for adding one such usage, I'm changing the API to take a list of
(revision, file matcher) tuples instead.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

rdamazio created this revision.Jul 10 2020, 2:45 AM
pulkit accepted this revision.Jul 10 2020, 5:56 AM
This revision is now accepted and ready to land.Jul 10 2020, 5:56 AM
martinvonz added a subscriber: martinvonz.Jul 10 2020, 10:53 AM

Should we have an entry in the release notes about this or have we given up on documenting API changes? I'm fine with either, as long as we have some policy.

mjacob added a subscriber: mjacob.Jul 10 2020, 8:10 PM
mjacob added inline comments.
mercurial/context.py
2543–2549

test-check-format.t complains about this.

rdamazio added a comment.Jul 10 2020, 8:27 PM
In D8721#130159, @martinvonz wrote:

Should we have an entry in the release notes about this or have we given up on documenting API changes? I'm fine with either, as long as we have some policy.

I'm happy to send another change for the release notes, let me know what's decided on policy.

mercurial/context.py
2543–2549

Hmmm apparently I'm missing something in my setup, I ran all tests, but it output this:

Skipped test-check-format.t: missing feature: the black formatter for python

So sorry about that. How do you want me to fix? Send another commit on top?

rdamazio added inline comments.Jul 10 2020, 8:38 PM
mercurial/context.py
2543–2549

Apparently the issue was that I had black 19.3b0, and the tests require 19.10b0.

mjacob added inline comments.Jul 10 2020, 8:48 PM
mercurial/context.py
2543–2549

I’m not sure what the policy for this is. What happened recently in a similar situation is that another patch was sent and folded into the changeset creating the test failure (see https://phab.mercurial-scm.org/D8680#129685). I don’t know if that’s the preferred way or it was done because the patch for the fix was already sent.

rdamazio updated this revision to Diff 21844.Jul 10 2020, 9:20 PM
rdamazio marked an inline comment as done.Jul 10 2020, 9:20 PM
rdamazio added inline comments.
mercurial/context.py
2543–2549

Well, phabricator let me just upload the amended commits after applying the formatting. Hope this works.

rdamazio marked 2 inline comments as done.Jul 10 2020, 9:21 PM

(also notice the formatting changed D8721, D8722 and D8723)

mjacob added inline comments.Jul 10 2020, 9:35 PM
hgext/remotefilelog/__init__.py
1124

Sorry, I didn’t realize before that this patch created another test failure:

--- /home/manu/vcs/hg/tests/test-check-pyflakes.t
+++ /home/manu/vcs/hg/tests/test-check-pyflakes.t.err
@@ -23,4 +23,5 @@
   mercurial/hgweb/server.py:*:* undefined name 'reload' (glob) (?)
   mercurial/util.py:*:* undefined name 'file' (glob) (?)
   mercurial/encoding.py:*:* undefined name 'localstr' (glob) (?)
+  hgext/remotefilelog/__init__.py:1124:18 import 'match' from line 136 shadowed by loop variable

I don’t know why it complains about this but not the cases where the import is shadowed by parameters.

rdamazio added inline comments.Jul 10 2020, 9:55 PM
hgext/remotefilelog/__init__.py
1124

Ugh:

Skipped test-check-pyflakes.t: missing feature: Pyflakes python linter

To be fair, the full list is this:

Skipped test-casecollision-merge.t: missing feature: case insensitive file system
Skipped test-casefolding.t: missing feature: case insensitive file system
Skipped test-check-clang-format.t: missing feature: clang-format C code formatter
Skipped test-check-format.t: missing feature: the black formatter for python
Skipped test-check-jshint.t: missing feature: JSHint static code analysis tool
Skipped test-check-pyflakes.t: missing feature: Pyflakes python linter
Skipped test-check-pylint.t: missing feature: Pylint python linter
Skipped test-check-rust-format.t: missing feature: rustfmt tool
Skipped test-chg.t: missing feature: running with chg
Skipped test-convert-baz.t: missing feature: GNU Arch baz client
Skipped test-convert-bzr-114.t: missing feature: Canonical's Bazaar client >= 1.14
Skipped test-convert-bzr-directories.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr-ghosts.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr-merges.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr-treeroot.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr.t: missing feature: Canonical's Bazaar client
Skipped test-convert-darcs.t: missing feature: darcs client
Skipped test-convert-hg-svn.t: missing feature: subversion python bindings
Skipped test-convert-mtn.t: missing feature: monotone client (>= 1.0)
Skipped test-convert-p4-filetypes.t: missing feature: Perforce server and client
Skipped test-convert-p4.t: missing feature: Perforce server and client
Skipped test-convert-svn-branches.t: missing feature: subversion python bindings
Skipped test-convert-svn-encoding.t: missing feature: subversion python bindings
Skipped test-convert-svn-move.t: missing feature: subversion python bindings
Skipped test-convert-svn-source.t: missing feature: subversion python bindings
Skipped test-convert-svn-startrev.t: missing feature: subversion python bindings
Skipped test-convert-svn-tags.t: missing feature: subversion python bindings
Skipped test-convert-tla.t: missing feature: GNU Arch tla client
Skipped test-debian-packages.t: missing feature: debian build dependencies (run dpkg-checkbuilddeps in contrib/)
Skipped test-docker-packaging.t: missing feature: docker support
Skipped test-fix-clang-format.t: missing feature: clang-format C code formatter
Skipped test-git-interop.t: missing feature: pygit2 Python library
Skipped test-lfs-test-server.t#git-server: missing feature: git-lfs test server
Skipped test-mac-packages.t: missing feature: OS X packaging tools
Skipped test-no-symlinks.t: system supports symbolic links
Skipped test-phabricator.t: missing feature: vcr http mocking library
Skipped test-releasenotes-formatting.t: missing feature: Fuzzy string matching library
Skipped test-releasenotes-merging.t: missing feature: Fuzzy string matching library
Skipped test-releasenotes-parsing.t: missing feature: Fuzzy string matching library
Skipped test-remotefilelog-bundle2-legacy.t: skipped
Skipped test-rhg.t: missing feature: Using the Rust extensions
Skipped test-sparse-fsmonitor.t: skipped
Skipped test-sparse-revlog.t: missing artifact, run "/<redacted>/hg/tests/artifacts/scripts/generate-churning-bundle.py"
Skipped test-sqlitestore.t: missing feature: sqlite3 module is available
Skipped test-verify-repo-operations.py: missing feature: allow slow tests (use --allow-slow-tests)
Skipped test-wireproto-exchangev2-shallow.t: missing feature: sqlite3 module is available

Where's the documentation on setting this up correctly with all the required tools? I don't see anything in https://www.mercurial-scm.org/wiki/ContributingChanges .

I'll install and fix the pyflakes issue.

rdamazio marked an inline comment as done.Jul 10 2020, 10:08 PM
rdamazio updated this revision to Diff 21847.
rdamazio added inline comments.Jul 10 2020, 10:08 PM
hgext/remotefilelog/__init__.py
1124

There were other offenses of this one in the same file, so renamed the import.

mjacob added inline comments.Jul 10 2020, 10:14 PM
hgext/remotefilelog/__init__.py
1124

I don’t think that everyone has all the required tools installed. Even the buildbot didn’t catch the black issue (but it catched the pyflakes issue). Recently I found that hgext.convert.subversion was not working on Python 3 at all and nobody realized. More and more builders will be migrated to use Python 3. I think that would be a good chance to install most of the requirements.

Unless you work on C code, JS code or Rust code, I think that installing all Python-specific tools for tests/test-check-* is sufficient for most cases. The other failures will hopefully be catched by some buildbot even after it was queued. :)

rdamazio added inline comments.Jul 10 2020, 10:37 PM
hgext/remotefilelog/__init__.py
1124

Makes sense.

Thanks for the review, let me know if you need me to do anything else before you can queue this.

mjacob added inline comments.Jul 10 2020, 11:49 PM
hgext/remotefilelog/__init__.py
1124

I can’t queue, but @pulkit or others can do it. The changes to the three revisions look good, although it would probably be a bit better for looking at the history later if the import rename was in a separate changeset.

pulkit added inline comments.Jul 11 2020, 4:44 AM
hgext/remotefilelog/__init__.py
1124

Replaced the existing patches with the new uploaded ones. Seems like I was also missing black and pyflakes on the machine I was testing.

Revision Contents
Changeset List

Diff 21847

hgext/lfs/wrapper.py

self.options[k] = v self.options[k] = v
# also copy lfs blobstores. note: this can run before reposetup, so lfs # also copy lfs blobstores. note: this can run before reposetup, so lfs
# blobstore attributes are not always ready at this time. # blobstore attributes are not always ready at this time.
for name in [b'lfslocalblobstore', b'lfsremoteblobstore']: for name in [b'lfslocalblobstore', b'lfsremoteblobstore']:
if util.safehasattr(othervfs, name): if util.safehasattr(othervfs, name):
setattr(self, name, getattr(othervfs, name)) setattr(self, name, getattr(othervfs, name))
def _prefetchfiles(repo, revs, match): def _prefetchfiles(repo, revmatches):
"""Ensure that required LFS blobs are present, fetching them as a group if """Ensure that required LFS blobs are present, fetching them as a group if
needed.""" needed."""
if not util.safehasattr(repo.svfs, b'lfslocalblobstore'): if not util.safehasattr(repo.svfs, b'lfslocalblobstore'):
return return
pointers = [] pointers = []
oids = set() oids = set()
localstore = repo.svfs.lfslocalblobstore localstore = repo.svfs.lfslocalblobstore
for rev in revs: for rev, match in revmatches:
ctx = repo[rev] ctx = repo[rev]
for f in ctx.walk(match): for f in ctx.walk(match):
p = pointerfromctx(ctx, f) p = pointerfromctx(ctx, f)
if p and p.oid() not in oids and not localstore.has(p.oid()): if p and p.oid() not in oids and not localstore.has(p.oid()):
p.filename = f p.filename = f
pointers.append(p) pointers.append(p)
oids.add(p.oid()) oids.add(p.oid())

hgext/remotefilelog/__init__.py

copies, copies,
debugcommands as hgdebugcommands, debugcommands as hgdebugcommands,
dispatch, dispatch,
error, error,
exchange, exchange,
extensions, extensions,
hg, hg,
localrepo, localrepo,
match, match as matchmod,
merge, merge,
node as nodemod, node as nodemod,
patch, patch,
pycompat, pycompat,
registrar, registrar,
repair, repair,
repoview, repoview,
revset, revset,
a slower, more accurate result, use ``file()``. a slower, more accurate result, use ``file()``.
""" """
if not isenabled(repo): if not isenabled(repo):
return orig(repo, subset, x) return orig(repo, subset, x)
# i18n: "filelog" is a keyword # i18n: "filelog" is a keyword
pat = revset.getstring(x, _(b"filelog requires a pattern")) pat = revset.getstring(x, _(b"filelog requires a pattern"))
m = match.match( m = matchmod.match(
repo.root, repo.getcwd(), [pat], default=b'relpath', ctx=repo[None] repo.root, repo.getcwd(), [pat], default=b'relpath', ctx=repo[None]
) )
s = set() s = set()
if not match.patkind(pat): if not matchmod.patkind(pat):
# slow # slow
for r in subset: for r in subset:
ctx = repo[r] ctx = repo[r]
cfiles = ctx.files() cfiles = ctx.files()
for f in m.files(): for f in m.files():
if f in cfiles: if f in cfiles:
s.add(ctx.rev()) s.add(ctx.rev())
break break
if util.safehasattr(remote, b'_callstream'): if util.safehasattr(remote, b'_callstream'):
remote._localrepo = repo remote._localrepo = repo
elif util.safehasattr(remote, b'getbundle'): elif util.safehasattr(remote, b'getbundle'):
extensions.wrapfunction(remote, b'getbundle', localgetbundle) extensions.wrapfunction(remote, b'getbundle', localgetbundle)
return orig(repo, remote, *args, **kwargs) return orig(repo, remote, *args, **kwargs)
def _fileprefetchhook(repo, revs, match): def _fileprefetchhook(repo, revmatches):
if isenabled(repo): if isenabled(repo):
allfiles = [] allfiles = []
for rev in revs: for rev, match in revmatches:
mjacobUnsubmitted
Done

Sorry, I didn’t realize before that this patch created another test failure:

--- /home/manu/vcs/hg/tests/test-check-pyflakes.t
+++ /home/manu/vcs/hg/tests/test-check-pyflakes.t.err
@@ -23,4 +23,5 @@
   mercurial/hgweb/server.py:*:* undefined name 'reload' (glob) (?)
   mercurial/util.py:*:* undefined name 'file' (glob) (?)
   mercurial/encoding.py:*:* undefined name 'localstr' (glob) (?)
+  hgext/remotefilelog/__init__.py:1124:18 import 'match' from line 136 shadowed by loop variable

I don’t know why it complains about this but not the cases where the import is shadowed by parameters.

mjacob: Sorry, I didn’t realize before that this patch created another test failure: ```…
rdamazioAuthorUnsubmitted
Done

Ugh:

Skipped test-check-pyflakes.t: missing feature: Pyflakes python linter

To be fair, the full list is this:

Skipped test-casecollision-merge.t: missing feature: case insensitive file system
Skipped test-casefolding.t: missing feature: case insensitive file system
Skipped test-check-clang-format.t: missing feature: clang-format C code formatter
Skipped test-check-format.t: missing feature: the black formatter for python
Skipped test-check-jshint.t: missing feature: JSHint static code analysis tool
Skipped test-check-pyflakes.t: missing feature: Pyflakes python linter
Skipped test-check-pylint.t: missing feature: Pylint python linter
Skipped test-check-rust-format.t: missing feature: rustfmt tool
Skipped test-chg.t: missing feature: running with chg
Skipped test-convert-baz.t: missing feature: GNU Arch baz client
Skipped test-convert-bzr-114.t: missing feature: Canonical's Bazaar client >= 1.14
Skipped test-convert-bzr-directories.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr-ghosts.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr-merges.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr-treeroot.t: missing feature: Canonical's Bazaar client
Skipped test-convert-bzr.t: missing feature: Canonical's Bazaar client
Skipped test-convert-darcs.t: missing feature: darcs client
Skipped test-convert-hg-svn.t: missing feature: subversion python bindings
Skipped test-convert-mtn.t: missing feature: monotone client (>= 1.0)
Skipped test-convert-p4-filetypes.t: missing feature: Perforce server and client
Skipped test-convert-p4.t: missing feature: Perforce server and client
Skipped test-convert-svn-branches.t: missing feature: subversion python bindings
Skipped test-convert-svn-encoding.t: missing feature: subversion python bindings
Skipped test-convert-svn-move.t: missing feature: subversion python bindings
Skipped test-convert-svn-source.t: missing feature: subversion python bindings
Skipped test-convert-svn-startrev.t: missing feature: subversion python bindings
Skipped test-convert-svn-tags.t: missing feature: subversion python bindings
Skipped test-convert-tla.t: missing feature: GNU Arch tla client
Skipped test-debian-packages.t: missing feature: debian build dependencies (run dpkg-checkbuilddeps in contrib/)
Skipped test-docker-packaging.t: missing feature: docker support
Skipped test-fix-clang-format.t: missing feature: clang-format C code formatter
Skipped test-git-interop.t: missing feature: pygit2 Python library
Skipped test-lfs-test-server.t#git-server: missing feature: git-lfs test server
Skipped test-mac-packages.t: missing feature: OS X packaging tools
Skipped test-no-symlinks.t: system supports symbolic links
Skipped test-phabricator.t: missing feature: vcr http mocking library
Skipped test-releasenotes-formatting.t: missing feature: Fuzzy string matching library
Skipped test-releasenotes-merging.t: missing feature: Fuzzy string matching library
Skipped test-releasenotes-parsing.t: missing feature: Fuzzy string matching library
Skipped test-remotefilelog-bundle2-legacy.t: skipped
Skipped test-rhg.t: missing feature: Using the Rust extensions
Skipped test-sparse-fsmonitor.t: skipped
Skipped test-sparse-revlog.t: missing artifact, run "/<redacted>/hg/tests/artifacts/scripts/generate-churning-bundle.py"
Skipped test-sqlitestore.t: missing feature: sqlite3 module is available
Skipped test-verify-repo-operations.py: missing feature: allow slow tests (use --allow-slow-tests)
Skipped test-wireproto-exchangev2-shallow.t: missing feature: sqlite3 module is available

Where's the documentation on setting this up correctly with all the required tools? I don't see anything in https://www.mercurial-scm.org/wiki/ContributingChanges .

I'll install and fix the pyflakes issue.

rdamazio: Ugh: ``` Skipped test-check-pyflakes.t: missing feature: Pyflakes python linter ``` To be…
mjacobUnsubmitted
Done

I don’t think that everyone has all the required tools installed. Even the buildbot didn’t catch the black issue (but it catched the pyflakes issue). Recently I found that hgext.convert.subversion was not working on Python 3 at all and nobody realized. More and more builders will be migrated to use Python 3. I think that would be a good chance to install most of the requirements.

Unless you work on C code, JS code or Rust code, I think that installing all Python-specific tools for tests/test-check-* is sufficient for most cases. The other failures will hopefully be catched by some buildbot even after it was queued. :)

mjacob: I don’t think that everyone has all the required tools installed. Even the buildbot didn’t…
rdamazioAuthorUnsubmitted
Done

Makes sense.

Thanks for the review, let me know if you need me to do anything else before you can queue this.

rdamazio: Makes sense. Thanks for the review, let me know if you need me to do anything else before you…
mjacobUnsubmitted
Not Done

I can’t queue, but @pulkit or others can do it. The changes to the three revisions look good, although it would probably be a bit better for looking at the history later if the import rename was in a separate changeset.

mjacob: I can’t queue, but @pulkit or others can do it. The changes to the three revisions look good…
pulkitUnsubmitted
Not Done

Replaced the existing patches with the new uploaded ones. Seems like I was also missing black and pyflakes on the machine I was testing.

pulkit: Replaced the existing patches with the new uploaded ones. Seems like I was also missing black…
rdamazioAuthorUnsubmitted
Done

There were other offenses of this one in the same file, so renamed the import.

rdamazio: There were other offenses of this one in the same file, so renamed the import.
if rev == nodemod.wdirrev or rev is None: if rev == nodemod.wdirrev or rev is None:
continue continue
ctx = repo[rev] ctx = repo[rev]
mf = ctx.manifest() mf = ctx.manifest()
sparsematch = repo.maybesparsematch(ctx.rev()) sparsematch = repo.maybesparsematch(ctx.rev())
for path in ctx.walk(match): for path in ctx.walk(match):
if (not sparsematch or sparsematch(path)) and path in mf: if (not sparsematch or sparsematch(path)) and path in mf:
allfiles.append((path, hex(mf[path]))) allfiles.append((path, hex(mf[path])))

mercurial/archival.py

if match(name): if match(name):
write(name, 0o644, False, lambda: buildmetadata(ctx)) write(name, 0o644, False, lambda: buildmetadata(ctx))
files = list(ctx.manifest().walk(match)) files = list(ctx.manifest().walk(match))
total = len(files) total = len(files)
if total: if total:
files.sort() files.sort()
scmutil.prefetchfiles( scmutil.prefetchfiles(
repo, [ctx.rev()], scmutil.matchfiles(repo, files) repo, [(ctx.rev(), scmutil.matchfiles(repo, files))]
) )
progress = repo.ui.makeprogress( progress = repo.ui.makeprogress(
_(b'archiving'), unit=_(b'files'), total=total _(b'archiving'), unit=_(b'files'), total=total
) )
progress.update(0) progress.update(0)
for f in files: for f in files:
ff = ctx.flags(f) ff = ctx.flags(f)
write(f, b'x' in ff and 0o755 or 0o644, b'l' in ff, ctx[f].data) write(f, b'x' in ff and 0o755 or 0o644, b'l' in ff, ctx[f].data)

mercurial/cmdutil.py

def _prefetchchangedfiles(repo, revs, match): def _prefetchchangedfiles(repo, revs, match):
allfiles = set() allfiles = set()
for rev in revs: for rev in revs:
for file in repo[rev].files(): for file in repo[rev].files():
if not match or match(file): if not match or match(file):
allfiles.add(file) allfiles.add(file)
scmutil.prefetchfiles(repo, revs, scmutil.matchfiles(repo, allfiles)) match = scmutil.matchfiles(repo, allfiles)
revmatches = [(rev, match) for rev in revs]
scmutil.prefetchfiles(repo, revmatches)
def export( def export(
repo, repo,
revs, revs,
basefm, basefm,
fntemplate=b'hg-%h.patch', fntemplate=b'hg-%h.patch',
switch_parent=False, switch_parent=False,
# for performance to avoid the cost of parsing the manifest. # for performance to avoid the cost of parsing the manifest.
if len(matcher.files()) == 1 and not matcher.anypats(): if len(matcher.files()) == 1 and not matcher.anypats():
file = matcher.files()[0] file = matcher.files()[0]
mfl = repo.manifestlog mfl = repo.manifestlog
mfnode = ctx.manifestnode() mfnode = ctx.manifestnode()
try: try:
if mfnode and mfl[mfnode].find(file)[0]: if mfnode and mfl[mfnode].find(file)[0]:
if _catfmtneedsdata(basefm): if _catfmtneedsdata(basefm):
scmutil.prefetchfiles(repo, [ctx.rev()], matcher) scmutil.prefetchfiles(repo, [(ctx.rev(), matcher)])
write(file) write(file)
return 0 return 0
except KeyError: except KeyError:
pass pass
if _catfmtneedsdata(basefm): if _catfmtneedsdata(basefm):
scmutil.prefetchfiles(repo, [ctx.rev()], matcher) scmutil.prefetchfiles(repo, [(ctx.rev(), matcher)])
for abs in ctx.walk(matcher): for abs in ctx.walk(matcher):
write(abs) write(abs)
err = 0 err = 0
uipathfn = scmutil.getuipathfn(repo, legacyrelativevalue=True) uipathfn = scmutil.getuipathfn(repo, legacyrelativevalue=True)
for subpath in sorted(ctx.substate): for subpath in sorted(ctx.substate):
sub = ctx.sub(subpath) sub = ctx.sub(subpath)
elif exact: elif exact:
ui.warn(msg % uipathfn(abs)) ui.warn(msg % uipathfn(abs))
break break
if not opts.get(b'dry_run'): if not opts.get(b'dry_run'):
needdata = (b'revert', b'add', b'undelete') needdata = (b'revert', b'add', b'undelete')
oplist = [actions[name][0] for name in needdata] oplist = [actions[name][0] for name in needdata]
prefetch = scmutil.prefetchfiles prefetch = scmutil.prefetchfiles
matchfiles = scmutil.matchfiles matchfiles = scmutil.matchfiles(
repo, [f for sublist in oplist for f in sublist]
)
prefetch( prefetch(
repo, repo, [(ctx.rev(), matchfiles)],
[ctx.rev()],
matchfiles(repo, [f for sublist in oplist for f in sublist]),
) )
match = scmutil.match(repo[None], pats) match = scmutil.match(repo[None], pats)
_performrevert( _performrevert(
repo, repo,
parents, parents,
ctx, ctx,
names, names,
uipathfn, uipathfn,

mercurial/context.py

This can occur during the merge process, e.g. by passing --tool :local This can occur during the merge process, e.g. by passing --tool :local
to resolve a conflict. to resolve a conflict.
""" """
keys = [] keys = []
# This won't be perfect, but can help performance significantly when # This won't be perfect, but can help performance significantly when
# using things like remotefilelog. # using things like remotefilelog.
scmutil.prefetchfiles( scmutil.prefetchfiles(
self.repo(), self.repo(),
[self.p1().rev()], [
(
self.p1().rev(),
scmutil.matchfiles(self.repo(), self._cache.keys()), scmutil.matchfiles(self.repo(), self._cache.keys()),
) )
],
)
mjacobUnsubmitted
Done

test-check-format.t complains about this.

mjacob: `test-check-format.t` complains about this.
rdamazioAuthorUnsubmitted
Done

Hmmm apparently I'm missing something in my setup, I ran all tests, but it output this:

Skipped test-check-format.t: missing feature: the black formatter for python

So sorry about that. How do you want me to fix? Send another commit on top?

rdamazio: Hmmm apparently I'm missing something in my setup, I ran all tests, but it output this…
rdamazioAuthorUnsubmitted
Done

Apparently the issue was that I had black 19.3b0, and the tests require 19.10b0.

rdamazio: Apparently the issue was that I had black 19.3b0, and the tests require 19.10b0.
mjacobUnsubmitted
Done

I’m not sure what the policy for this is. What happened recently in a similar situation is that another patch was sent and folded into the changeset creating the test failure (see https://phab.mercurial-scm.org/D8680#129685). I don’t know if that’s the preferred way or it was done because the patch for the fix was already sent.

mjacob: I’m not sure what the policy for this is. What happened recently in a similar situation is that…
rdamazioAuthorUnsubmitted
Done

Well, phabricator let me just upload the amended commits after applying the formatting. Hope this works.

rdamazio: Well, phabricator let me just upload the amended commits after applying the formatting. Hope…
for path in self._cache.keys(): for path in self._cache.keys():
cache = self._cache[path] cache = self._cache[path]
try: try:
underlying = self._wrappedctx[path] underlying = self._wrappedctx[path]
if ( if (
underlying.data() == cache[b'data'] underlying.data() == cache[b'data']
and underlying.flags() == cache[b'flags'] and underlying.flags() == cache[b'flags']

mercurial/merge.py

mergestatemod.ACTION_LOCAL_DIR_RENAME_GET, mergestatemod.ACTION_LOCAL_DIR_RENAME_GET,
mergestatemod.ACTION_MERGE, mergestatemod.ACTION_MERGE,
) )
] ]
prefetch = scmutil.prefetchfiles prefetch = scmutil.prefetchfiles
matchfiles = scmutil.matchfiles matchfiles = scmutil.matchfiles
prefetch( prefetch(
repo, repo,
[ctx.rev()], [
matchfiles(repo, [f for sublist in oplist for f, args, msg in sublist]), (
ctx.rev(),
matchfiles(
repo, [f for sublist in oplist for f, args, msg in sublist]
),
)
],
) )
@attr.s(frozen=True) @attr.s(frozen=True)
class updateresult(object): class updateresult(object):
updatedcount = attr.ib() updatedcount = attr.ib()
mergedcount = attr.ib() mergedcount = attr.ib()
removedcount = attr.ib() removedcount = attr.ib()

mercurial/patch.py

if src not in ctx1: if src not in ctx1:
# Files merged in during a merge and then copied/renamed are # Files merged in during a merge and then copied/renamed are
# reported as copies. We want to show them in the diff as additions. # reported as copies. We want to show them in the diff as additions.
del copy[dst] del copy[dst]
prefetchmatch = scmutil.matchfiles( prefetchmatch = scmutil.matchfiles(
repo, list(modifiedset | addedset | removedset) repo, list(modifiedset | addedset | removedset)
) )
scmutil.prefetchfiles(repo, [ctx1.rev(), ctx2.rev()], prefetchmatch) revmatches = [
(ctx1.rev(), prefetchmatch),
(ctx2.rev(), prefetchmatch),
]
scmutil.prefetchfiles(repo, revmatches)
def difffn(opts, losedata): def difffn(opts, losedata):
return trydiff( return trydiff(
repo, repo,
revs, revs,
ctx1, ctx1,
ctx2, ctx2,
modified, modified,

mercurial/scmutil.py

] ]
_reportnewcssource = [ _reportnewcssource = [
b'pull', b'pull',
b'unbundle', b'unbundle',
] ]
def prefetchfiles(repo, revs, match): def prefetchfiles(repo, revmatches):
"""Invokes the registered file prefetch functions, allowing extensions to """Invokes the registered file prefetch functions, allowing extensions to
ensure the corresponding files are available locally, before the command ensure the corresponding files are available locally, before the command
uses them.""" uses them.
if match:
Args:
revmatches: a list of (revision, match) tuples to indicate the files to
fetch at each revision. If any of the match elements is None, it matches
all files.
"""
def _matcher(m):
if m:
assert isinstance(m, matchmod.basematcher)
# The command itself will complain about files that don't exist, so # The command itself will complain about files that don't exist, so
# don't duplicate the message. # don't duplicate the message.
match = matchmod.badmatch(match, lambda fn, msg: None) return matchmod.badmatch(m, lambda fn, msg: None)
else: else:
match = matchall(repo) return matchall(repo)
revbadmatches = [(rev, _matcher(match)) for (rev, match) in revmatches]
fileprefetchhooks(repo, revs, match) fileprefetchhooks(repo, revbadmatches)
# a list of (repo, revs, match) prefetch functions # a list of (repo, revs, match) prefetch functions
fileprefetchhooks = util.hooks() fileprefetchhooks = util.hooks()
# A marker that tells the evolve extension to suppress its own reporting # A marker that tells the evolve extension to suppress its own reporting
_reportstroubledchangesets = True _reportstroubledchangesets = True

mercurial/subrepo.py

def archive(self, archiver, prefix, match=None, decode=True): def archive(self, archiver, prefix, match=None, decode=True):
self._get(self._state + (b'hg',)) self._get(self._state + (b'hg',))
files = self.files() files = self.files()
if match: if match:
files = [f for f in files if match(f)] files = [f for f in files if match(f)]
rev = self._state[1] rev = self._state[1]
ctx = self._repo[rev] ctx = self._repo[rev]
scmutil.prefetchfiles( scmutil.prefetchfiles(
self._repo, [ctx.rev()], scmutil.matchfiles(self._repo, files) self._repo, [(ctx.rev(), scmutil.matchfiles(self._repo, files))]
) )
total = abstractsubrepo.archive(self, archiver, prefix, match) total = abstractsubrepo.archive(self, archiver, prefix, match)
for subpath in ctx.substate: for subpath in ctx.substate:
s = subrepo(ctx, subpath, True) s = subrepo(ctx, subpath, True)
submatch = matchmod.subdirmatcher(subpath, match) submatch = matchmod.subdirmatcher(subpath, match)
subprefix = prefix + subpath + b'/' subprefix = prefix + subpath + b'/'
total += s.archive(archiver, subprefix, submatch, decode) total += s.archive(archiver, subprefix, submatch, decode)
return total return total