This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7954

hgdemandimport: apply lazy module loading to sys.meta_path finders
ClosedPublic

Authored by indygreg on Jan 21 2020, 2:48 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHGf81c17ec303c: hgdemandimport: apply lazy module loading to sys.meta_path finders
Summary

Python's sys.meta_path finders are the primary objects whose job it
is to find a module at import time. When import is called, Python
iterates objects in this list and calls o.find_spec(...) to find
a ModuleSpec (or None if the module couldn't be found by that
finder). If no meta path finder can find a module, import fails.

One of the default meta path finders is PathFinder. Its job is to
import modules from the filesystem and is probably the most important
importer. This finder looks at sys.path and sys.path_hooks to do
its job.

The ModuleSpec returned by MetaPathImporter.find_spec() has a
loader attribute, which defines the concrete module loader to use.
sys.path_hooks is a hook point for teaching PathFinder to
instantiate custom loader types.

Previously, we injected a custom sys.path_hook that told PathFinder
to wrap the default loaders with a loader that creates a module object
that is lazy.

This approach worked. But its main limitation was that it only applied
to the PathFinder meta path importer. There are other meta path
importers that are registered. And in the case of PyOxidizer loading
modules from memory, PathFinder doesn't come into play since
PyOxidizer's own meta path importer was handling all imports.

This commit changes our approach to lazy module loading by proxying
all meta path importers. Specifically, we overload the find_spec()
method to swap in a wrapped loader on the ModuleSpec before it
is returned. The end result of this is all meta path importers should
be lazy.

As much as I would have loved to utilize .class manipulation to
achieve this, some meta path importers are implemented in C/Rust
in such a way that they cannot be monkeypatched. This is why we
use getattribute to define a proxy.

Also, this change could theoretically open us up to regressions in
meta path importers whose loader is creating module objects which
can't be monkeypatched. But I'm not aware of any of these in the
wild. So I think we'll be safe.

According to hyperfine, this change yields a decent startup time win of
5-6ms:

Benchmark #1: ~/.pyenv/versions/3.6.10/bin/python ./hg version
  Time (mean ± σ):      86.8 ms ±   0.5 ms    [User: 78.0 ms, System: 8.7 ms]
  Range (min … max):    86.0 ms …  89.1 ms    50 runs

  Time (mean ± σ):      81.1 ms ±   2.7 ms    [User: 74.5 ms, System: 6.5 ms]
  Range (min … max):    77.8 ms …  90.5 ms    50 runs

Benchmark #2: ~/.pyenv/versions/3.7.6/bin/python ./hg version
  Time (mean ± σ):      78.9 ms ±   0.6 ms    [User: 70.2 ms, System: 8.7 ms]
  Range (min … max):    78.1 ms …  81.2 ms    50 runs

  Time (mean ± σ):      73.4 ms ±   0.6 ms    [User: 65.3 ms, System: 8.0 ms]
  Range (min … max):    72.4 ms …  75.7 ms    50 runs

Benchmark #3: ~/.pyenv/versions/3.8.1/bin/python ./hg version
  Time (mean ± σ):      78.1 ms ±   0.6 ms    [User: 70.2 ms, System: 7.9 ms]
  Range (min … max):    77.4 ms …  80.9 ms    50 runs

  Time (mean ± σ):      72.1 ms ±   0.4 ms    [User: 64.4 ms, System: 7.6 ms]
  Range (min … max):    71.4 ms …  74.1 ms    50 runs

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Jan 21 2020, 2:48 AM
indygreg edited the summary of this revision. (Show Details)Jan 21 2020, 2:51 AM
pulkit accepted this revision.Jan 21 2020, 11:02 AM
This revision is now accepted and ready to land.Jan 21 2020, 11:02 AM

Revision Contents
Changeset List

Diff 19477

hgdemandimport/demandimportpy3.py

practice, we only expect builtins to be loaded before demandimport is practice, we only expect builtins to be loaded before demandimport is
enabled. enabled.
""" """
# This line is unnecessary, but it satisfies test-check-py3-compat.t. # This line is unnecessary, but it satisfies test-check-py3-compat.t.
from __future__ import absolute_import from __future__ import absolute_import
import contextlib import contextlib
import importlib.abc
import importlib.machinery
import importlib.util import importlib.util
import sys import sys
from . import tracing from . import tracing
_deactivated = False _deactivated = False
# Python 3.5's LazyLoader doesn't work for some reason. # Python 3.5's LazyLoader doesn't work for some reason.
"""Make the module load lazily.""" """Make the module load lazily."""
with tracing.log('demandimport %s', module): with tracing.log('demandimport %s', module):
if _deactivated or module.__name__ in ignores: if _deactivated or module.__name__ in ignores:
self.loader.exec_module(module) self.loader.exec_module(module)
else: else:
super().exec_module(module) super().exec_module(module)
_extensions_loader = _lazyloaderex.factory( class LazyFinder(object):
importlib.machinery.ExtensionFileLoader """A wrapper around a ``MetaPathFinder`` that makes loaders lazy.
)
_bytecode_loader = _lazyloaderex.factory(
importlib.machinery.SourcelessFileLoader
)
_source_loader = _lazyloaderex.factory(importlib.machinery.SourceFileLoader)
``sys.meta_path`` finders have their ``find_spec()`` called to locate a
module. This returns a ``ModuleSpec`` if found or ``None``. The
``ModuleSpec`` has a ``loader`` attribute, which is called to actually
load a module.
def _makefinder(path): Our class wraps an existing finder and overloads its ``find_spec()`` to
return importlib.machinery.FileFinder( replace the ``loader`` with our lazy loader proxy.
path,
# This is the order in which loaders are passed in in core Python. We have to use __getattribute__ to proxy the instance because some meta
(_extensions_loader, importlib.machinery.EXTENSION_SUFFIXES), path finders don't support monkeypatching.
(_source_loader, importlib.machinery.SOURCE_SUFFIXES), """
(_bytecode_loader, importlib.machinery.BYTECODE_SUFFIXES),
) __slots__ = ("_finder",)
def __init__(self, finder):
object.__setattr__(self, "_finder", finder)
def __repr__(self):
return "<LazyFinder for %r>" % object.__getattribute__(self, "_finder")
# __bool__ is canonical Python 3. But check-code insists on __nonzero__ being
# defined via `def`.
def __nonzero__(self):
return bool(object.__getattribute__(self, "_finder"))
__bool__ = __nonzero__
def __getattribute__(self, name):
if name in ("_finder", "find_spec"):
return object.__getattribute__(self, name)
return getattr(object.__getattribute__(self, "_finder"), name)
def __delattr__(self, name):
return delattr(object.__getattribute__(self, "_finder"))
def __setattr__(self, name, value):
return setattr(object.__getattribute__(self, "_finder"), name, value)
def find_spec(self, *args, **kwargs):
finder = object.__getattribute__(self, "_finder")
spec = finder.find_spec(*args, **kwargs)
# Lazy loader requires exec_module().
if (
spec is not None
and spec.loader is not None
and getattr(spec.loader, "exec_module")
):
spec.loader = _lazyloaderex(spec.loader)
return spec
ignores = set() ignores = set()
def init(ignoreset): def init(ignoreset):
global ignores global ignores
ignores = ignoreset ignores = ignoreset
def isenabled(): def isenabled():
return _makefinder in sys.path_hooks and not _deactivated return not _deactivated and any(
isinstance(finder, LazyFinder) for finder in sys.meta_path
)
def disable(): def disable():
try: new_finders = []
while True: for finder in sys.meta_path:
sys.path_hooks.remove(_makefinder) new_finders.append(
except ValueError: finder._finder if isinstance(finder, LazyFinder) else finder
pass )
sys.meta_path[:] = new_finders
def enable(): def enable():
if not _supported: if not _supported:
return return
sys.path_hooks.insert(0, _makefinder) new_finders = []
for finder in sys.meta_path:
new_finders.append(
LazyFinder(finder) if not isinstance(finder, LazyFinder) else finder
)
sys.meta_path[:] = new_finders
@contextlib.contextmanager @contextlib.contextmanager
def deactivated(): def deactivated():
# This implementation is a bit different from Python 2's. Python 3 # This implementation is a bit different from Python 2's. Python 3
# maintains a per-package finder cache in sys.path_importer_cache (see # maintains a per-package finder cache in sys.path_importer_cache (see
# PEP 302). This means that we can't just call disable + enable. # PEP 302). This means that we can't just call disable + enable.
# If we do that, in situations like: # If we do that, in situations like:

tests/test-demandimport.py

procutil procutil
) )
assert f(procutil.system) == '<function system at 0x?>', f(procutil.system) assert f(procutil.system) == '<function system at 0x?>', f(procutil.system)
assert 'mercurial.hgweb' not in sys.modules assert 'mercurial.hgweb' not in sys.modules
from mercurial import hgweb from mercurial import hgweb
if ispy3: if ispy3:
assert not isinstance(hgweb, _LazyModule) assert isinstance(hgweb, _LazyModule)
assert f(hgweb) == "<module 'mercurial.hgweb' from '?'>", f(hgweb) assert f(hgweb) == "<module 'mercurial.hgweb' from '?'>", f(hgweb)
assert isinstance(hgweb.hgweb_mod, _LazyModule) assert isinstance(hgweb.hgweb_mod, _LazyModule)
assert ( assert (
f(hgweb.hgweb_mod) == "<module 'mercurial.hgweb.hgweb_mod' from '?'>" f(hgweb.hgweb_mod) == "<module 'mercurial.hgweb.hgweb_mod' from '?'>"
), f(hgweb.hgweb_mod) ), f(hgweb.hgweb_mod)
else: else:
assert f(hgweb) == "<unloaded module 'hgweb'>", f(hgweb) assert f(hgweb) == "<unloaded module 'hgweb'>", f(hgweb)
assert f(hgweb.hgweb_mod) == "<unloaded module 'hgweb_mod'>", f( assert f(hgweb.hgweb_mod) == "<unloaded module 'hgweb_mod'>", f(
re.stderr re.stderr
) )
assert f(re) == "<proxied module 'sys'>", f(re) assert f(re) == "<proxied module 'sys'>", f(re)
assert 'telnetlib' not in sys.modules assert 'telnetlib' not in sys.modules
import telnetlib import telnetlib
if ispy3: if ispy3:
assert not isinstance(telnetlib, _LazyModule) assert isinstance(telnetlib, _LazyModule)
assert f(telnetlib) == "<module 'telnetlib' from '?'>" assert f(telnetlib) == "<module 'telnetlib' from '?'>"
else: else:
assert f(telnetlib) == "<unloaded module 'telnetlib'>", f(telnetlib) assert f(telnetlib) == "<unloaded module 'telnetlib'>", f(telnetlib)
try: try:
from telnetlib import unknownattr from telnetlib import unknownattr
assert False, ( assert False, (