From 6609cadbe828a63bad6b8bec37c7e2c0aff2bac0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Sun, 1 May 2022 13:08:19 -0600 Subject: [PATCH 01/18] Added first draft for Lazy Imports PEP --- pep-9999.rst | 214 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 pep-9999.rst diff --git a/pep-9999.rst b/pep-9999.rst new file mode 100644 index 00000000000..546a046c673 --- /dev/null +++ b/pep-9999.rst @@ -0,0 +1,214 @@ +PEP: 9999 +Title: Lazy Imports +Author: Germán Méndez Bravo +Sponsor: +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 29-Apr-2022 + +Abstract +======== + +This PEP proposes a feature to make defer the load and execution of imported +objects, providing a robust solution to the often used paradigm of making +things lazy in Python. Making imports lazy, besides reducing the number of +loaded modules at run time improving startup (and even overall execution) +speed, also has the pretty great side effect of reducing the risk of hitting +circular imports. + +Motivation +========== + +Tightly coupled design or highly entwined modules, make hard to stop the +"Import Domino Effect" when importing anything in a codebase; this is something +we frequently see in big and complex codebases that have evolved over time, +where many of the imported modules over the files are only imported at module +level for readability, maintainability and/or simplicity. The unfortunate +result of this kind of state is that an increasing number of loaded modules +during startup can significantly increase the time a system needs to start up, +and some times we are spending a large amount of time loading modules that may +or may not be used during the execution of the system. + +In the past, experience has shown us that trying to manually solve these +problems by making things lazy (using inner imports or carefully tailoring +solutions to lazify expensive subsystems) leads to easily breakable and hard to +maintain efforts. What’s more, in attempts to solve these problems or improve +design, refactoring of these highly evolved and highly entwined systems is +difficult because even changes in imports order can lead to hard to debug and +difficult to solve import cycles. + +Rationale and Goals +=================== + +Fast iteration cycle is one of the reasons that Python is loved by developers. +However, in large and complex codebases, we’ve found that Python encounters a +serious usability problem, where just starting an application may take a long +time (minutes in some cases!), due to spending a lot of time executing imports +and module-level code at import time. + +The goal of this PEP is to provide a feature for deferring loading and +executing of imported modules from the time of import to the time of first use +which: + +* provides a mostly transparent and robust mechanism to achieve lazy + importing of modules; + +* implements an option flag to enable to mechanism, with minimal risk of + breaking backwards compatibility; + +* helps reducing the likelihood of bumping into import cycles when refactoring + codebases; and, + +* has none or negligible performance impact on the existing code or the code + that will be using the new mechanism. + +Specification +============= + +The mechanism for enabling Lazy Imports is optional and can be globally enabled +by passing a flag (``-X lazyimportsall``) to the python runtime. + +When enabled, the loading and execution of all (and only) top level import +modules shall be deferred until the imported name is used; this could happen +immediately, i.e. in the very next line after the import statement, or much +later, e.g. while using the name inside a function being calling by some other +code at some later time. + +For these top level import modules, there are two exceptions which will make +them eager (not deferred): modules inside ``try``/``except``/``finally`` +blocks; and star imports. + +All other imports throughout the modules remain eager. + +Backwards Compatibility +======================= + +This proposal preserves 100% backwards compatibility when the feature is +disabled. + +When enabled, the mechanics of the import system is modified in a way +such that deferred imports could produce currently unexpected results and +behaviors in existing codebases. The problems that are expected to be more +common when we enabling Lazy Imports mechanism are related to: + +Import Side Effects +------------------- + +Those using this feature should be aware that import side effects that are +otherwise usually produced by the execution of imported modules during the +execution of import statements could and will be deferred at least until the +imported objects are used and even indefinitely delayed for modules imported +inside imported modules. + +These import side effects may include: + +* code executing any logic when being imported; +* relying on submodules being set as attributes in the parent modules. + +Dynamic Paths +------------- + +There could be issues related to dynamic Python paths; particularly, adding +(and then removing after the import) paths from ``sys.path``. + +.. code-block py + sys.path.insert(0, "/path/to/foo/module") + import foo + del sys.path[0] + foo.Bar() + +Deferred Errors +--------------- + +All the errors are deferred from import time to first-use time (including +``ModuleNotFoundError``), which might complicate debugging. Accessing an object +in the middle of an statement could trigger an import and produce `ImportError` +or any number of other exceptions resulting from the resolution of the deferred +object, while loading and executing the related imported module. + +Security Implications +===================== + +Deferred execution of code could produce security concerns if process owner, +path, ``sys.path``, or other sensitive environment or contextual states change +between the time the ``import`` statement is executed and the time where the +imported object is used. + +Performance Impact +================== + +The reference implementation has shown that the feature alone has negligible +performance impact on the existing code or the code that will be using the new +mechanism, including C extensions. Some times we can see a degraded performance +under certain circumstances, but some times also improved performance as well, +due to the reduced number of modules needed to be loaded. + +How to Teach This +================= + +Some best practices to deal with some of the issues that could arise and to +better take advantage of Lazy Imports are: + +* Avoid relying in import side effects. Often times used for the Registry + Pattern, where systems expect registration happens implicitly during the + importing of modules. Registration should have to be refactored for be + explicitly called, triggering a discovery process, if needed. + +* Always import each module being used, don't rely on module objects having + attributes to child submodules; i.e.: do ``import foo.bar; foo.bar.Baz``, + not ``import foo; foo.bar.Baz``. The latter only works (unreliably) because + the attribute bar in the module ``foo`` is added as an import side effect of + ``foo.bar`` being imported somewhere else. With Lazy Imports this may not + always happen on time. + +* Avoid using star imports, as those are always eager. + +* When possible, do not import whole submodules. Import specific names instead; + i.e.: do ``from foo.bar import Baz``, not ``import foo.bar`` and then + ``foo.bar.Baz``. If you import submodules (such as ``foo.qux`` and + ``foo.fred``), with Lazy Imports enabled, when you access the parent module's + name (``foo`` in this case), that will trigger loading all of the sibling + submodules of the parent module (``foo.bar``, ``foo.qux`` and ``foo.fred``), + not only the one being accessed, because the parent module ``foo`` is the + actual deferred object name. + +* Don't use inner imports, unless absolutely necessary. Circular imports should + no longer be a big issue with Lazy Imports enabled, so there’s no need to add + complexity or more opcodes in a potentially hot path. + +* Always use ``from __future__ import annotations`` when possible. This way, + modules that are imported only for typing purposes will never be loaded under + Lazy Imports. + +* Use string type annotations for ``typing.TypeVar()`` and ``typing.NewType()``. + The reason is Python doesn't have postponed evaluation of types being used in + these helper classes. + +* Wrap type aliases inside a ``TYPE_CHECKING`` conditional block (only type + aliases, there is no particular need to do type-only imports inside this + block). The reason is Python doesn't support postponed evaluation of types + for type aliases. + +Reference Implementation +======================== + +The current reference implementation is available as part of Cinder [1]_ as a +runtime-level feature that provides a transparent and robust mechanism for lazy +loading imports. Reference implementation is in use within Meta Platforms and +has proven to achieve improvements in startup time (and total runtime for some +applications) in the range of 40%-70%, as well as significant reduction in +memory footprint (up to 40%), thanks to not needing to resolve some of the +deferred imports that end up being unused in the common flow. + +References +========== + +.. [1] Reference implementation + (https://github.com/facebookincubator/cinder) + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 65c4c7c888053189919ee8cee7852bdf01c84559 Mon Sep 17 00:00:00 2001 From: Carl Meyer Date: Mon, 2 May 2022 15:29:16 -0600 Subject: [PATCH 02/18] Suggested changes to lazy imports PEP. --- .github/CODEOWNERS | 1 + pep-9999.rst | 360 ++++++++++++++++++++++++++++++--------------- 2 files changed, 240 insertions(+), 121 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 27acae546d8..f51158ab64f 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -657,3 +657,4 @@ pep-8016.rst @njsmith @dstufft pep-8100.rst @njsmith # pep-8101.rst # pep-8102.rst +pep-9999.rst @warsaw diff --git a/pep-9999.rst b/pep-9999.rst index 546a046c673..a59ea1f8163 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -1,85 +1,154 @@ PEP: 9999 Title: Lazy Imports -Author: Germán Méndez Bravo -Sponsor: +Author: Germán Méndez Bravo , Carl Meyer +Sponsor: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Apr-2022 +Python-Version: 3.12 +Discussions-To: https://discuss.python.org Abstract ======== -This PEP proposes a feature to make defer the load and execution of imported -objects, providing a robust solution to the often used paradigm of making -things lazy in Python. Making imports lazy, besides reducing the number of -loaded modules at run time improving startup (and even overall execution) -speed, also has the pretty great side effect of reducing the risk of hitting -circular imports. +This PEP proposes an opt-in experimental feature to transparently defer the +execution of imported modules until the moment when an imported object is used. +Since Python programs commonly import many more modules than a single +invocation of the program is likely to use in practice, lazy imports can +greatly reduce the overall number of modules loaded, improving startup time and +memory usage. Lazy imports also mostly eliminate the risk of import cycles. + Motivation ========== -Tightly coupled design or highly entwined modules, make hard to stop the -"Import Domino Effect" when importing anything in a codebase; this is something -we frequently see in big and complex codebases that have evolved over time, -where many of the imported modules over the files are only imported at module -level for readability, maintainability and/or simplicity. The unfortunate -result of this kind of state is that an increasing number of loaded modules -during startup can significantly increase the time a system needs to start up, -and some times we are spending a large amount of time loading modules that may -or may not be used during the execution of the system. - -In the past, experience has shown us that trying to manually solve these -problems by making things lazy (using inner imports or carefully tailoring -solutions to lazify expensive subsystems) leads to easily breakable and hard to -maintain efforts. What’s more, in attempts to solve these problems or improve -design, refactoring of these highly evolved and highly entwined systems is -difficult because even changes in imports order can lead to hard to debug and -difficult to solve import cycles. - -Rationale and Goals -=================== - -Fast iteration cycle is one of the reasons that Python is loved by developers. -However, in large and complex codebases, we’ve found that Python encounters a -serious usability problem, where just starting an application may take a long -time (minutes in some cases!), due to spending a lot of time executing imports -and module-level code at import time. - -The goal of this PEP is to provide a feature for deferring loading and -executing of imported modules from the time of import to the time of first use -which: - -* provides a mostly transparent and robust mechanism to achieve lazy - importing of modules; - -* implements an option flag to enable to mechanism, with minimal risk of - breaking backwards compatibility; - -* helps reducing the likelihood of bumping into import cycles when refactoring - codebases; and, - -* has none or negligible performance impact on the existing code or the code - that will be using the new mechanism. +Common Python code style prefers imports at module level, so they don't have +to be repeated within each scope the imported object is used in, and to avoid +the inefficiency of repeated execution of the import opcodes at runtime. This +means that importing the main module of a program typically results in an +immediate cascade of imports of most or all of the modules that may ever be +needed by the program. + +Consider the example of a Python command line program with a number of +subcommands. Each subcommand may perform different tasks, requiring the import +of different dependencies. But a given invocation of the program will only +execute a single subcommand, or possibly none (i.e. if just ``--help`` usage +info is requested.) Top-level eager imports in such a program will result in +the import of many modules that will never be used at all; the time spent +(possibly compiling and) executing these modules is pure waste. + +Some large Python CLIs, in an effort to improve startup time, make some imports +lazy by manually placing imports inline into functions to delay imports of +expensive subsystems. This manual approach is labor-intensive and fragile; one +misplaced import or refactor can easily undo painstaking optimization work. + +Existing import-hook-based solutions such as demandimport [1] are limited in +that only certain styles of import can be made truly lazy (imports such as +``from foo import a, b`` will still eagerly import the module ``foo``) and they +impose additional runtime overhead on every module attribute access. + +This PEP proposes a more comprehensive solution for lazy imports that does not +impose detectable overhead in real-world use. The implementation in this PEP +has already demonstrated startup time wins up to 70% and memory-use wins up to +40% on real-world Python CLIs. + +Lazy imports also eliminate most import cycles. With eager imports, "false +cycles" can easily occur which are fixed by simply moving an import to the +bottom of a module or inline into a function, or switching from ``from foo +import bar`` to ``import foo``. With lazy imports, these "cycles" just work. +The only cycles which will remain are those where two modules actually each use +a name from the other at module level; these "true" cycles are only fixable by +refactoring the classes or functions involved. + + +Rationale +========= + +The aim of this feature is to make imports transparently lazy. "Lazy" means +that the import of a module (execution of the module body and addition of the +module object to ``sys.modules``) should not occur until the module (or a name +imported from it) is actually referenced during execution. "Transparent" means +that besides the delayed import (and necessarily observable effects of that, +such as delayed import side effects and changes to ``sys.modules``), there is +no other observable change in behavior: the imported object is present in the +module namespace as normal and is transparently loaded whenever first used: its +status as a "lazy imported object" is not directly observable. + +The requirement that the imported object be present in the module namespace as +usual, even before the import has actually occurred, means that we need some +kind of "lazy object" placeholder to represent the not-yet-imported object. +The transparency requirement dictates that this placeholder must never be +visible to Python code; any reference to it must trigger the import and replace +it with the real imported object. + +Given the possibility that Python (or C extension) code may pull objects +directly out of a module ``__dict__``, the only way to reliably prevent +accidental leakage of lazy objects is to have the dictionary itself be +responsible to ensure resolution of lazy objects on lookup. + +To avoid a performance penalty on the vast majority of dictionaries which never +contain any lazy object, we install a specialized lookup function +(``lookdict_unicode_lazy``) for module namespace dictionaries when they first +gain a lazy-object value. This lookup function checks the looked-up value and +resolves it before returning it, if it is a lazy object. + +This implementation comprehensively prevents leakage of lazy objects, ensuring +they are always resolved to the real imported object before anyone can get hold +of them for any use, while avoiding any noticeable performance impact on +dictionaries in general. + Specification ============= -The mechanism for enabling Lazy Imports is optional and can be globally enabled -by passing a flag (``-X lazyimportsall``) to the python runtime. +Lazy imports are opt-in, and globally enabled via a new ``-L`` flag to the +python interpreter, or a ``PYTHONLAZYIMPORTS`` environment variable. + +When enabled, the loading and execution of all (and only) top level imports is +deferred until the imported name is used. This could happen immediately (e.g. +on the very next line after the import statement) or much later (e.g. while +using the name inside a function being called by some other code at some later +time.) + +For these top level imports, there are two exceptions which will make them +eager (not lazy): imports inside ``try``/``except``/``finally`` or ``with`` +blocks, and star imports (``from foo import *``.) Imports inside +exception-handling blocks remain eager so that any exceptions arising from the +import can be handled. Star imports must remain eager since performing the +import is the only way to know which names should be added to the namespace. -When enabled, the loading and execution of all (and only) top level import -modules shall be deferred until the imported name is used; this could happen -immediately, i.e. in the very next line after the import statement, or much -later, e.g. while using the name inside a function being calling by some other -code at some later time. +Imports inside class definitions or inside functions/methods are not "top +level" and are never lazy. -For these top level import modules, there are two exceptions which will make -them eager (not deferred): modules inside ``try``/``except``/``finally`` -blocks; and star imports. +Dynamic imports using ``__import__`` or ``importlib.import_module`` are also +never lazy. + + +Per-module opt out +------------------ + +Due to the backwards compatibility issues mentioned below, it may be necessary +to force some imports to be eager. In first-party code, this can be easily +accomplished via any module-level reference to the name, e.g. even re-assigning +the name to itself will trigger the import: + +.. code-block py + import foo + + # ensure 'foo' is eagerly imported + foo = foo + +The more difficult case can occur if an import in third-party code that can't +easily be modified must be forced to be eager. For this purpose, we propose to +add an API to ``importlib`` that can be called early in the process to specify +a list of module names within which all imports will be eager: + +.. code-block py + from importlib import set_eager_imports + + set_eager_imports(["one.mod", "another"]) -All other imports throughout the modules remain eager. Backwards Compatibility ======================= @@ -87,30 +156,29 @@ Backwards Compatibility This proposal preserves 100% backwards compatibility when the feature is disabled. -When enabled, the mechanics of the import system is modified in a way -such that deferred imports could produce currently unexpected results and -behaviors in existing codebases. The problems that are expected to be more -common when we enabling Lazy Imports mechanism are related to: +When enabled, lazy imports could produce currently unexpected results and +behaviors in existing codebases. The problems that we may see when enabling +lazy imports in an existing codebase are related to: + Import Side Effects ------------------- -Those using this feature should be aware that import side effects that are -otherwise usually produced by the execution of imported modules during the -execution of import statements could and will be deferred at least until the -imported objects are used and even indefinitely delayed for modules imported -inside imported modules. +Import side effects that would otherwise be produced by the execution of +imported modules during the execution of import statements will be deferred at +least until the imported objects are used. These import side effects may include: -* code executing any logic when being imported; -* relying on submodules being set as attributes in the parent modules. +* code executing any side-effecting logic during import; +* relying on imported submodules being set as attributes in the parent module. + Dynamic Paths ------------- -There could be issues related to dynamic Python paths; particularly, adding -(and then removing after the import) paths from ``sys.path``. +There could be issues related to dynamic Python import paths; particularly, +adding (and then removing after the import) paths from ``sys.path``. .. code-block py sys.path.insert(0, "/path/to/foo/module") @@ -118,14 +186,20 @@ There could be issues related to dynamic Python paths; particularly, adding del sys.path[0] foo.Bar() -Deferred Errors ---------------- +In this case, with lazy imports enabled, the import of ``foo`` will not +actually occur while the addition to ``sys.path`` is present. + + +Deferred Exceptions +------------------- + +All exceptions arising from import (including ``ModuleNotFoundError``) are +deferred from import time to first-use time, which might complicate debugging. +Accessing an object in the middle of any code could trigger a deferred import +and produce ``ImportError`` or any other exception resulting from the +resolution of the deferred object, while loading and executing the related +imported module. -All the errors are deferred from import time to first-use time (including -``ModuleNotFoundError``), which might complicate debugging. Accessing an object -in the middle of an statement could trigger an import and produce `ImportError` -or any number of other exceptions resulting from the resolution of the deferred -object, while loading and executing the related imported module. Security Implications ===================== @@ -135,77 +209,121 @@ path, ``sys.path``, or other sensitive environment or contextual states change between the time the ``import`` statement is executed and the time where the imported object is used. + Performance Impact ================== -The reference implementation has shown that the feature alone has negligible -performance impact on the existing code or the code that will be using the new -mechanism, including C extensions. Some times we can see a degraded performance -under certain circumstances, but some times also improved performance as well, -due to the reduced number of modules needed to be loaded. +The reference implementation has shown that the feature has negligible +performance impact on existing real-world codebases (Instagram Server and other +several CLI programs at Meta), while providing substantial improvements to +startup time and memory usage. + +The reference implementation shows small performance regressions in a few +pyperformance benchmarks, but improvements in others. (TODO update with +detailed data from 3.11 port of implementation.) + How to Teach This ================= +In most cases, lazy imports should just work transparently and no teaching of +the feature should be necessary. + +The implementation will ensure that errors resulting from a deferred import +have metadata attached pointing the user to the original import statement, to +ease debuggability of errors from lazy imports. + Some best practices to deal with some of the issues that could arise and to -better take advantage of Lazy Imports are: +better take advantage of lazy imports are: -* Avoid relying in import side effects. Often times used for the Registry - Pattern, where systems expect registration happens implicitly during the - importing of modules. Registration should have to be refactored for be - explicitly called, triggering a discovery process, if needed. +* Avoid relying on import side effects. Perhaps the most common reliance on + import side effects is the registry pattern, where population of some + external registry happens implicitly during the importing of modules, often + via decorators. Instead, the registry should be built via an explicit call + that perhaps does a discovery process to find decorated functions or classes. -* Always import each module being used, don't rely on module objects having - attributes to child submodules; i.e.: do ``import foo.bar; foo.bar.Baz``, - not ``import foo; foo.bar.Baz``. The latter only works (unreliably) because - the attribute bar in the module ``foo`` is added as an import side effect of - ``foo.bar`` being imported somewhere else. With Lazy Imports this may not - always happen on time. +* Always import needed submodules explicitly, don't rely on some other import + to ensure a module has its submodules as attributes. That is, do ``import + foo.bar; foo.bar.Baz``, not ``import foo; foo.bar.Baz``. The latter only + works (unreliably) because the attribute ``foo.bar`` is added as a side + effect of ``foo.bar`` being imported somewhere else. With lazy imports this + may not always happen on time. * Avoid using star imports, as those are always eager. * When possible, do not import whole submodules. Import specific names instead; i.e.: do ``from foo.bar import Baz``, not ``import foo.bar`` and then ``foo.bar.Baz``. If you import submodules (such as ``foo.qux`` and - ``foo.fred``), with Lazy Imports enabled, when you access the parent module's + ``foo.fred``), with lazy imports enabled, when you access the parent module's name (``foo`` in this case), that will trigger loading all of the sibling submodules of the parent module (``foo.bar``, ``foo.qux`` and ``foo.fred``), not only the one being accessed, because the parent module ``foo`` is the actual deferred object name. -* Don't use inner imports, unless absolutely necessary. Circular imports should - no longer be a big issue with Lazy Imports enabled, so there’s no need to add +* Don't use inline imports, unless absolutely necessary. Import cycles should + no longer be a problem with lazy imports enabled, so there’s no need to add complexity or more opcodes in a potentially hot path. -* Always use ``from __future__ import annotations`` when possible. This way, - modules that are imported only for typing purposes will never be loaded under - Lazy Imports. - -* Use string type annotations for ``typing.TypeVar()`` and ``typing.NewType()``. - The reason is Python doesn't have postponed evaluation of types being used in - these helper classes. - -* Wrap type aliases inside a ``TYPE_CHECKING`` conditional block (only type - aliases, there is no particular need to do type-only imports inside this - block). The reason is Python doesn't support postponed evaluation of types - for type aliases. Reference Implementation ======================== -The current reference implementation is available as part of Cinder [1]_ as a -runtime-level feature that provides a transparent and robust mechanism for lazy -loading imports. Reference implementation is in use within Meta Platforms and -has proven to achieve improvements in startup time (and total runtime for some -applications) in the range of 40%-70%, as well as significant reduction in -memory footprint (up to 40%), thanks to not needing to resolve some of the -deferred imports that end up being unused in the common flow. +The current reference implementation is available as part of Cinder [2]_. +Reference implementation is in use within Meta Platforms and has proven to +achieve improvements in startup time (and total runtime for some applications) +in the range of 40%-70%, as well as significant reduction in memory footprint +(up to 40%), thanks to not needing to execute imports that end up being unused +in the common flow. + + +Rejected Ideas +============== + +Explicit syntax for lazy imports +-------------------------------- + +If the primary objective of lazy imports were solely to work around import +cycles and forward references, an explicitly-marked syntax for particular +targeted imports to be lazy would make a lot of sense. But in practice it would +be very hard to get robust startup time or memory use benefits from this +approach, since it would require converting most imports within your code base +(and in third-party dependencies) to use the lazy import syntax. + +It would be possible to aim for a "shallow" laziness where only the top-level +imports of subsystems from the main module are made explicitly lazy, but then +imports within the subsystems are all eager. This is extremely fragile, though +-- it only takes one mis-placed import to undo the carefully constructed +shallow laziness. Globally enabling lazy imports, on the other hand, provides +in-depth robust laziness where you always pay only for the imports you use. + + +Half-lazy imports +----------------- + +It would be possible to eagerly run the import loader to the point of finding +the module source, but then defer the actual execution of the module and +creation of the module object. The advantage of this would be that certain +classes of import errors (e.g. a simple typo in the module name) would be +caught eagerly instead of being deferred to the use of an imported name. + +The disadvantage would be that the startup time benefits of lazy imports would +be significantly reduced, since unused imports would still require a filesystem +``stat()`` call, at least. It would also introduce a possibly non-obvious split +between _which_ import errors are raised eagerly and which are delayed, when +lazy imports are enabled. + +This idea is rejected for now on the basis that in practice, confusion about +import typos has not been an observed problem with the reference +implementation. Generally delayed imports are not delayed forever, and errors +show up soon enough to be caught and fixed (unless the import is truly unused.) + References ========== -.. [1] Reference implementation - (https://github.com/facebookincubator/cinder) +.. [1] demandimport (https://github.com/bwesterb/py-demandimport/) +.. [2] Reference implementation (https://github.com/facebookincubator/cinder) + Copyright ========= From d6248cb181a4d24841c273e495e845b96ddaf1c7 Mon Sep 17 00:00:00 2001 From: Carl Meyer Date: Mon, 2 May 2022 15:56:35 -0600 Subject: [PATCH 03/18] Add another example of forcing eager imports. --- pep-9999.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index a59ea1f8163..beee0672260 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -139,6 +139,16 @@ the name to itself will trigger the import: # ensure 'foo' is eagerly imported foo = foo +Another option that scales better to making multiple imports lazy is to place +them inside a ``try/finally``: + +.. code-block py + try: # force these imports to be eager + import foo + import bar + finally: + pass + The more difficult case can occur if an import in third-party code that can't easily be modified must be forced to be eager. For this purpose, we propose to add an API to ``importlib`` that can be called early in the process to specify @@ -154,7 +164,7 @@ Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility when the feature is -disabled. +disabled, which is the default. When enabled, lazy imports could produce currently unexpected results and behaviors in existing codebases. The problems that we may see when enabling From db9cf680fa2727f0a1d4c32facbd881787304975 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:17:31 -0700 Subject: [PATCH 04/18] Update pep-9999.rst Co-authored-by: Jelle Zijlstra --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index beee0672260..4452c0f99a7 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -34,7 +34,7 @@ Consider the example of a Python command line program with a number of subcommands. Each subcommand may perform different tasks, requiring the import of different dependencies. But a given invocation of the program will only execute a single subcommand, or possibly none (i.e. if just ``--help`` usage -info is requested.) Top-level eager imports in such a program will result in +info is requested). Top-level eager imports in such a program will result in the import of many modules that will never be used at all; the time spent (possibly compiling and) executing these modules is pure waste. From 2ff03bb81dfde9c7b0aea7a8b40f31a8e23b9f71 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:18:37 -0700 Subject: [PATCH 05/18] Update Carl's email --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 4452c0f99a7..6d66bceaacb 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -1,6 +1,6 @@ PEP: 9999 Title: Lazy Imports -Author: Germán Méndez Bravo , Carl Meyer +Author: Germán Méndez Bravo , Carl Meyer Sponsor: Barry Warsaw Status: Draft Type: Standards Track From 93add80d5e68ce86d2583772962534f237616643 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:21:04 -0700 Subject: [PATCH 06/18] Update pep-9999.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- pep-9999.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 6d66bceaacb..6db930a15ce 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -7,7 +7,6 @@ Type: Standards Track Content-Type: text/x-rst Created: 29-Apr-2022 Python-Version: 3.12 -Discussions-To: https://discuss.python.org Abstract ======== From 170be4df91d29a5aa815724296c5f6006106aaa0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:21:12 -0700 Subject: [PATCH 07/18] Update pep-9999.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 6db930a15ce..3ccd84efb9d 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -132,7 +132,7 @@ to force some imports to be eager. In first-party code, this can be easily accomplished via any module-level reference to the name, e.g. even re-assigning the name to itself will trigger the import: -.. code-block py +.. code-block python import foo # ensure 'foo' is eagerly imported From a05c5eb4be601b17d69896337be56a57898b1247 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:21:17 -0700 Subject: [PATCH 08/18] Update pep-9999.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 3ccd84efb9d..a12e9f49165 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -162,7 +162,7 @@ a list of module names within which all imports will be eager: Backwards Compatibility ======================= -This proposal preserves 100% backwards compatibility when the feature is +This proposal preserves full backwards compatibility when the feature is disabled, which is the default. When enabled, lazy imports could produce currently unexpected results and From 05345dece0acc774e1f909760b721667df81add1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:24:07 -0700 Subject: [PATCH 09/18] Update pep-9999.rst Co-authored-by: Carl Meyer --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index a12e9f49165..0e468ea486e 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -42,7 +42,7 @@ lazy by manually placing imports inline into functions to delay imports of expensive subsystems. This manual approach is labor-intensive and fragile; one misplaced import or refactor can easily undo painstaking optimization work. -Existing import-hook-based solutions such as demandimport [1] are limited in +Existing import-hook-based solutions such as demandimport [1]_ are limited in that only certain styles of import can be made truly lazy (imports such as ``from foo import a, b`` will still eagerly import the module ``foo``) and they impose additional runtime overhead on every module attribute access. From 318a70da6c1cbcce305c6acf5fe9fba585d5eb96 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:37:34 -0700 Subject: [PATCH 10/18] Update and rename pep-9999.rst to pep-0690.rst --- pep-9999.rst => pep-0690.rst | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) rename pep-9999.rst => pep-0690.rst (95%) diff --git a/pep-9999.rst b/pep-0690.rst similarity index 95% rename from pep-9999.rst rename to pep-0690.rst index 0e468ea486e..1ff465cd2bc 100644 --- a/pep-9999.rst +++ b/pep-0690.rst @@ -1,4 +1,4 @@ -PEP: 9999 +PEP: 690 Title: Lazy Imports Author: Germán Méndez Bravo , Carl Meyer Sponsor: Barry Warsaw @@ -22,12 +22,13 @@ memory usage. Lazy imports also mostly eliminate the risk of import cycles. Motivation ========== -Common Python code style prefers imports at module level, so they don't have -to be repeated within each scope the imported object is used in, and to avoid -the inefficiency of repeated execution of the import opcodes at runtime. This -means that importing the main module of a program typically results in an -immediate cascade of imports of most or all of the modules that may ever be -needed by the program. +Common Python code style, as suggested in the imports section in :pep:`8`, +prefers imports at module level, so they don't have to be repeated within +each scope the imported object is used in, and to avoid the inefficiency +of repeated execution of the import opcodes at runtime. This means that +importing the main module of a program typically results in an immediate +cascade of imports of most or all of the modules that may ever be needed +by the program. Consider the example of a Python command line program with a number of subcommands. Each subcommand may perform different tasks, requiring the import @@ -50,7 +51,7 @@ impose additional runtime overhead on every module attribute access. This PEP proposes a more comprehensive solution for lazy imports that does not impose detectable overhead in real-world use. The implementation in this PEP has already demonstrated startup time wins up to 70% and memory-use wins up to -40% on real-world Python CLIs. +40% on real-world Python CLIs [2]_. Lazy imports also eliminate most import cycles. With eager imports, "false cycles" can easily occur which are fixed by simply moving an import to the @@ -277,7 +278,7 @@ better take advantage of lazy imports are: Reference Implementation ======================== -The current reference implementation is available as part of Cinder [2]_. +The current reference implementation is available as part of Cinder [3]_. Reference implementation is in use within Meta Platforms and has proven to achieve improvements in startup time (and total runtime for some applications) in the range of 40%-70%, as well as significant reduction in memory footprint @@ -331,7 +332,8 @@ References ========== .. [1] demandimport (https://github.com/bwesterb/py-demandimport/) -.. [2] Reference implementation (https://github.com/facebookincubator/cinder) +.. [2] Lazy Imports https://github.com/facebookincubator/cinder/blob/cinder/3.8/CinderDoc/lazy_imports.rst +.. [3] Reference implementation (https://github.com/facebookincubator/cinder) Copyright From b0c1c70853ad1b6714a9c86fdcbd799ac5914be6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:38:52 -0700 Subject: [PATCH 11/18] Update .github/CODEOWNERS Co-authored-by: Carl Meyer --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f51158ab64f..63303b34fc1 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -657,4 +657,4 @@ pep-8016.rst @njsmith @dstufft pep-8100.rst @njsmith # pep-8101.rst # pep-8102.rst -pep-9999.rst @warsaw +pep-0690.rst @warsaw From 3148bb14d6dee053ebcdd53fdba6488eff85378f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 15:42:51 -0700 Subject: [PATCH 12/18] Update pep-0690.rst Co-authored-by: Carl Meyer --- pep-0690.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0690.rst b/pep-0690.rst index 1ff465cd2bc..419a9cc797d 100644 --- a/pep-0690.rst +++ b/pep-0690.rst @@ -139,7 +139,7 @@ the name to itself will trigger the import: # ensure 'foo' is eagerly imported foo = foo -Another option that scales better to making multiple imports lazy is to place +Another option that scales better to making multiple imports eager is to place them inside a ``try/finally``: .. code-block py From 51f41223f69a5fff02a634c69448821c7596044b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 16:20:42 -0700 Subject: [PATCH 13/18] Update pep-0690.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- pep-0690.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0690.rst b/pep-0690.rst index 419a9cc797d..f35baec376b 100644 --- a/pep-0690.rst +++ b/pep-0690.rst @@ -190,7 +190,7 @@ Dynamic Paths There could be issues related to dynamic Python import paths; particularly, adding (and then removing after the import) paths from ``sys.path``. -.. code-block py +.. code-block python:: sys.path.insert(0, "/path/to/foo/module") import foo del sys.path[0] From e27cf261eedd635bd22a95119892f422d7dbbd70 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 17:31:48 -0600 Subject: [PATCH 14/18] Added myself to AUTHOR_OVERRIDES --- AUTHOR_OVERRIDES.csv | 1 + 1 file changed, 1 insertion(+) diff --git a/AUTHOR_OVERRIDES.csv b/AUTHOR_OVERRIDES.csv index 4d58942d445..72289408f75 100644 --- a/AUTHOR_OVERRIDES.csv +++ b/AUTHOR_OVERRIDES.csv @@ -9,3 +9,4 @@ Just van Rossum,"van Rossum, Just (JvR)",JvR Martin v. Löwis,"von Löwis, Martin",von Löwis Nathaniel Smith,"Smith, Nathaniel J.",Smith P.J. Eby,"Eby, Phillip J.",Eby +Germán Méndez Bravo,"Méndez Bravo, Germán",Méndez Bravo From 76f2a9b2e0a15be59b304a6362105f29bc13c4dc Mon Sep 17 00:00:00 2001 From: Carl Meyer Date: Mon, 2 May 2022 17:45:24 -0600 Subject: [PATCH 15/18] More updates to lazy imports --- pep-0690.rst | 56 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 21 deletions(-) diff --git a/pep-0690.rst b/pep-0690.rst index f35baec376b..518f9258166 100644 --- a/pep-0690.rst +++ b/pep-0690.rst @@ -11,24 +11,23 @@ Python-Version: 3.12 Abstract ======== -This PEP proposes an opt-in experimental feature to transparently defer the -execution of imported modules until the moment when an imported object is used. -Since Python programs commonly import many more modules than a single -invocation of the program is likely to use in practice, lazy imports can -greatly reduce the overall number of modules loaded, improving startup time and -memory usage. Lazy imports also mostly eliminate the risk of import cycles. +This PEP proposes a feature to transparently defer the execution of imported +modules until the moment when an imported object is used. Since Python +programs commonly import many more modules than a single invocation of the +program is likely to use in practice, lazy imports can greatly reduce the +overall number of modules loaded, improving startup time and memory usage. Lazy +imports also mostly eliminate the risk of import cycles. Motivation ========== Common Python code style, as suggested in the imports section in :pep:`8`, -prefers imports at module level, so they don't have to be repeated within -each scope the imported object is used in, and to avoid the inefficiency -of repeated execution of the import opcodes at runtime. This means that -importing the main module of a program typically results in an immediate -cascade of imports of most or all of the modules that may ever be needed -by the program. +prefers imports at module level, so they don't have to be repeated within each +scope the imported object is used in, and to avoid the inefficiency of repeated +execution of the import system at runtime. This means that importing the main +module of a program typically results in an immediate cascade of imports of +most or all of the modules that may ever be needed by the program. Consider the example of a Python command line program with a number of subcommands. Each subcommand may perform different tasks, requiring the import @@ -73,7 +72,8 @@ that besides the delayed import (and necessarily observable effects of that, such as delayed import side effects and changes to ``sys.modules``), there is no other observable change in behavior: the imported object is present in the module namespace as normal and is transparently loaded whenever first used: its -status as a "lazy imported object" is not directly observable. +status as a "lazy imported object" is not directly observable from Python or +from C extension code. The requirement that the imported object be present in the module namespace as usual, even before the import has actually occurred, means that we need some @@ -88,10 +88,11 @@ accidental leakage of lazy objects is to have the dictionary itself be responsible to ensure resolution of lazy objects on lookup. To avoid a performance penalty on the vast majority of dictionaries which never -contain any lazy object, we install a specialized lookup function +contain any lazy objects, we install a specialized lookup function (``lookdict_unicode_lazy``) for module namespace dictionaries when they first -gain a lazy-object value. This lookup function checks the looked-up value and -resolves it before returning it, if it is a lazy object. +gain a lazy-object value. When this lookup function finds that the key +references a lazy object, it resolves the lazy object immediately before +returning it. This implementation comprehensively prevents leakage of lazy objects, ensuring they are always resolved to the real imported object before anyone can get hold @@ -103,7 +104,7 @@ Specification ============= Lazy imports are opt-in, and globally enabled via a new ``-L`` flag to the -python interpreter, or a ``PYTHONLAZYIMPORTS`` environment variable. +Python interpreter, or a ``PYTHONLAZYIMPORTS`` environment variable. When enabled, the loading and execution of all (and only) top level imports is deferred until the imported name is used. This could happen immediately (e.g. @@ -114,15 +115,16 @@ time.) For these top level imports, there are two exceptions which will make them eager (not lazy): imports inside ``try``/``except``/``finally`` or ``with`` blocks, and star imports (``from foo import *``.) Imports inside -exception-handling blocks remain eager so that any exceptions arising from the -import can be handled. Star imports must remain eager since performing the +exception-handling blocks (this includes ``with`` blocks, since those can also +"catch" and handle exceptions) remain eager so that any exceptions arising from +the import can be handled. Star imports must remain eager since performing the import is the only way to know which names should be added to the namespace. Imports inside class definitions or inside functions/methods are not "top level" and are never lazy. -Dynamic imports using ``__import__`` or ``importlib.import_module`` are also -never lazy. +Dynamic imports using ``__import__()`` or ``importlib.import_module()`` are +also never lazy. Per-module opt out @@ -328,6 +330,18 @@ implementation. Generally delayed imports are not delayed forever, and errors show up soon enough to be caught and fixed (unless the import is truly unused.) +Lazy dynamic imports +-------------------- + +It would be possible to add a ``lazy=True`` or similar option to ``__import__`` +and/or ``importlib.import_module()``, to enable them to perform lazy imports. +That idea is rejected in this PEP for lack of a clear use case. Dynamic imports +are already far outside the :pep:`8` code style recommendations for imports, +and can easily be made precisely as lazy as desired by placing them at the +desired point in the code flow. These aren't commonly used at module top level, +which is where lazy imports applies. + + References ========== From 26a7abb120b5f6f270797077fa836037326a3470 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Mon, 2 May 2022 18:18:38 -0600 Subject: [PATCH 16/18] Resolved comments --- .github/CODEOWNERS | 2 +- pep-0690.rst | 55 ++++++++++++++++++++-------------------------- 2 files changed, 25 insertions(+), 32 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 63303b34fc1..bb98413d806 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -570,6 +570,7 @@ pep-0686.rst @methane pep-0687.rst @encukou pep-0688.rst @jellezijlstra pep-0689.rst @encukou +pep-0690.rst @warsaw # ... # pep-0754.txt # ... @@ -657,4 +658,3 @@ pep-8016.rst @njsmith @dstufft pep-8100.rst @njsmith # pep-8101.rst # pep-8102.rst -pep-0690.rst @warsaw diff --git a/pep-0690.rst b/pep-0690.rst index 518f9258166..545ec043239 100644 --- a/pep-0690.rst +++ b/pep-0690.rst @@ -8,6 +8,7 @@ Content-Type: text/x-rst Created: 29-Apr-2022 Python-Version: 3.12 + Abstract ======== @@ -22,12 +23,12 @@ imports also mostly eliminate the risk of import cycles. Motivation ========== -Common Python code style, as suggested in the imports section in :pep:`8`, -prefers imports at module level, so they don't have to be repeated within each -scope the imported object is used in, and to avoid the inefficiency of repeated -execution of the import system at runtime. This means that importing the main -module of a program typically results in an immediate cascade of imports of -most or all of the modules that may ever be needed by the program. +Common Python code style :pep:`prefers <8#imports>` imports at module +level, so they don't have to be repeated within each scope the imported object +is used in, and to avoid the inefficiency of repeated execution of the import +system at runtime. This means that importing the main module of a program +typically results in an immediate cascade of imports of most or all of the +modules that may ever be needed by the program. Consider the example of a Python command line program with a number of subcommands. Each subcommand may perform different tasks, requiring the import @@ -37,20 +38,23 @@ info is requested). Top-level eager imports in such a program will result in the import of many modules that will never be used at all; the time spent (possibly compiling and) executing these modules is pure waste. -Some large Python CLIs, in an effort to improve startup time, make some imports +In an effort to improve startup time, some large Python CLIs tools make imports lazy by manually placing imports inline into functions to delay imports of expensive subsystems. This manual approach is labor-intensive and fragile; one misplaced import or refactor can easily undo painstaking optimization work. -Existing import-hook-based solutions such as demandimport [1]_ are limited in -that only certain styles of import can be made truly lazy (imports such as +Existing import-hook-based solutions such as `demandimport +`_ are limited in are limited +in that only certain styles of import can be made truly lazy (imports such as ``from foo import a, b`` will still eagerly import the module ``foo``) and they impose additional runtime overhead on every module attribute access. This PEP proposes a more comprehensive solution for lazy imports that does not impose detectable overhead in real-world use. The implementation in this PEP -has already demonstrated startup time wins up to 70% and memory-use wins up to -40% on real-world Python CLIs [2]_. +has already `demonstrated +`_ +startup time improvements up to 70% and memory-use reductions up to +40% on real-world Python CLIs. Lazy imports also eliminate most import cycles. With eager imports, "false cycles" can easily occur which are fixed by simply moving an import to the @@ -85,7 +89,7 @@ it with the real imported object. Given the possibility that Python (or C extension) code may pull objects directly out of a module ``__dict__``, the only way to reliably prevent accidental leakage of lazy objects is to have the dictionary itself be -responsible to ensure resolution of lazy objects on lookup. +responsible to ensure resolution of lazy objects on lookup. To avoid a performance penalty on the vast majority of dictionaries which never contain any lazy objects, we install a specialized lookup function @@ -133,18 +137,16 @@ Per-module opt out Due to the backwards compatibility issues mentioned below, it may be necessary to force some imports to be eager. In first-party code, this can be easily accomplished via any module-level reference to the name, e.g. even re-assigning -the name to itself will trigger the import: +the name to itself will trigger the import:: -.. code-block python import foo - + # ensure 'foo' is eagerly imported foo = foo Another option that scales better to making multiple imports eager is to place -them inside a ``try/finally``: +them inside a ``try/finally``:: -.. code-block py try: # force these imports to be eager import foo import bar @@ -154,9 +156,8 @@ them inside a ``try/finally``: The more difficult case can occur if an import in third-party code that can't easily be modified must be forced to be eager. For this purpose, we propose to add an API to ``importlib`` that can be called early in the process to specify -a list of module names within which all imports will be eager: +a list of module names within which all imports will be eager:: -.. code-block py from importlib import set_eager_imports set_eager_imports(["one.mod", "another"]) @@ -190,9 +191,8 @@ Dynamic Paths ------------- There could be issues related to dynamic Python import paths; particularly, -adding (and then removing after the import) paths from ``sys.path``. +adding (and then removing after the import) paths from ``sys.path``:: -.. code-block python:: sys.path.insert(0, "/path/to/foo/module") import foo del sys.path[0] @@ -280,7 +280,8 @@ better take advantage of lazy imports are: Reference Implementation ======================== -The current reference implementation is available as part of Cinder [3]_. +The current reference implementation is available as part of +`Cinder `_. Reference implementation is in use within Meta Platforms and has proven to achieve improvements in startup time (and total runtime for some applications) in the range of 40%-70%, as well as significant reduction in memory footprint @@ -321,7 +322,7 @@ caught eagerly instead of being deferred to the use of an imported name. The disadvantage would be that the startup time benefits of lazy imports would be significantly reduced, since unused imports would still require a filesystem ``stat()`` call, at least. It would also introduce a possibly non-obvious split -between _which_ import errors are raised eagerly and which are delayed, when +between *which* import errors are raised eagerly and which are delayed, when lazy imports are enabled. This idea is rejected for now on the basis that in practice, confusion about @@ -342,14 +343,6 @@ desired point in the code flow. These aren't commonly used at module top level, which is where lazy imports applies. -References -========== - -.. [1] demandimport (https://github.com/bwesterb/py-demandimport/) -.. [2] Lazy Imports https://github.com/facebookincubator/cinder/blob/cinder/3.8/CinderDoc/lazy_imports.rst -.. [3] Reference implementation (https://github.com/facebookincubator/cinder) - - Copyright ========= From d9dc1dc91c1234d0d571ea01742fda152004e39c Mon Sep 17 00:00:00 2001 From: Carl Meyer Date: Tue, 3 May 2022 12:01:42 -0600 Subject: [PATCH 17/18] More updates to lazy imports PEP. --- pep-0690.rst | 136 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 99 insertions(+), 37 deletions(-) diff --git a/pep-0690.rst b/pep-0690.rst index 545ec043239..c6bb9a90615 100644 --- a/pep-0690.rst +++ b/pep-0690.rst @@ -44,10 +44,12 @@ expensive subsystems. This manual approach is labor-intensive and fragile; one misplaced import or refactor can easily undo painstaking optimization work. Existing import-hook-based solutions such as `demandimport -`_ are limited in are limited -in that only certain styles of import can be made truly lazy (imports such as -``from foo import a, b`` will still eagerly import the module ``foo``) and they -impose additional runtime overhead on every module attribute access. +`_ or `importlib.util.LazyLoader +`_ +are limited in that only certain styles of import can be made truly lazy +(imports such as ``from foo import a, b`` will still eagerly import the module +``foo``) and they impose additional runtime overhead on every module attribute +access. This PEP proposes a more comprehensive solution for lazy imports that does not impose detectable overhead in real-world use. The implementation in this PEP @@ -98,9 +100,13 @@ gain a lazy-object value. When this lookup function finds that the key references a lazy object, it resolves the lazy object immediately before returning it. +Some operations on dictionaries (e.g. iterating all values) don't go through +the lookup function; in these cases we have to add a check if the lookup +function is ``lookdict_unicode_lazy`` and if so, resolve all lazy values first. + This implementation comprehensively prevents leakage of lazy objects, ensuring they are always resolved to the real imported object before anyone can get hold -of them for any use, while avoiding any noticeable performance impact on +of them for any use, while avoiding any significant performance impact on dictionaries in general. @@ -131,21 +137,29 @@ Dynamic imports using ``__import__()`` or ``importlib.import_module()`` are also never lazy. +Debuggability +------------- + +The implementation will ensure that exceptions resulting from a deferred import +have metadata attached pointing the user to the original import statement, to +ease debuggability of errors from lazy imports. + +Additionally, debug logging from ``python -v`` will include logging when an +import statement has been encountered but execution of the import will be +deferred. + +Python's ``-X importtime`` feature for profiling import costs adapts naturally +to lazy imports; the profiled time is the time spent actually importing. + + Per-module opt out ------------------ Due to the backwards compatibility issues mentioned below, it may be necessary -to force some imports to be eager. In first-party code, this can be easily -accomplished via any module-level reference to the name, e.g. even re-assigning -the name to itself will trigger the import:: - - import foo - - # ensure 'foo' is eagerly imported - foo = foo +to force some imports to be eager. -Another option that scales better to making multiple imports eager is to place -them inside a ``try/finally``:: +In first-party code, since imports inside a ``try`` or ``with`` block are never +lazy, this can be easily accomplished:: try: # force these imports to be eager import foo @@ -153,6 +167,20 @@ them inside a ``try/finally``:: finally: pass +This PEP proposes to add a new ``importlib.eager_imports()`` context manager, +so the above technique can be less verbose and doesn't require comments to +clarify its intent:: + + with eager_imports(): + import foo + import bar + +Since imports within context managers are always eager, the ``eager_imports()`` +context manager can just be an alias to a null context manager. The context +manager does not force all imports to be recursively eager: ``foo`` and ``bar`` +will be imported eagerly, but imports within those modules will still follow +the usual laziness rules. + The more difficult case can occur if an import in third-party code that can't easily be modified must be forced to be eager. For this purpose, we propose to add an API to ``importlib`` that can be called early in the process to specify @@ -162,6 +190,9 @@ a list of module names within which all imports will be eager:: set_eager_imports(["one.mod", "another"]) +The effect of this is also shallow: all imports within ``one.mod`` will be +eager, but not imports in all modules imported by ``one.mod``. + Backwards Compatibility ======================= @@ -169,9 +200,14 @@ Backwards Compatibility This proposal preserves full backwards compatibility when the feature is disabled, which is the default. -When enabled, lazy imports could produce currently unexpected results and -behaviors in existing codebases. The problems that we may see when enabling -lazy imports in an existing codebase are related to: +Even when enabled, most code will continue to work normally without any +observable change (other than improved startup time and memory usage.) +Namespace packages are not effected: they work just as they do currently, +except lazily. + +In some existing code, lazy imports could produce currently unexpected results +and behaviors. The problems that we may see when enabling lazy imports in an +existing codebase are related to: Import Side Effects @@ -186,6 +222,16 @@ These import side effects may include: * code executing any side-effecting logic during import; * relying on imported submodules being set as attributes in the parent module. +A relevant and typical affected case is the `click +`_ library for building Python command-line +interfaces. If e.g. ``cli = click.group()`` is defined in ``main.py``, and +``sub.py`` imports ``cli`` from ``main`` and adds subcommands to it via +decorator (``@cli.command(...)``), but the actual ``cli()`` call is in +``main.py``, then lazy imports may prevent the subcommands from being +registered, since in this case Click is depending on side effects of the import +of ``sub.py``. In this case the fix is to ensure the import of ``sub.py`` is +eager, e.g. by using the ``importlib.eager_imports()`` context manager. + Dynamic Paths ------------- @@ -206,11 +252,12 @@ Deferred Exceptions ------------------- All exceptions arising from import (including ``ModuleNotFoundError``) are -deferred from import time to first-use time, which might complicate debugging. +deferred from import time to first-use time, which could complicate debugging. Accessing an object in the middle of any code could trigger a deferred import and produce ``ImportError`` or any other exception resulting from the resolution of the deferred object, while loading and executing the related -imported module. +imported module. The implementation will provide debugging assistance in +lazy-import-triggered tracebacks to mitigate this issue. Security Implications @@ -238,12 +285,9 @@ detailed data from 3.11 port of implementation.) How to Teach This ================= -In most cases, lazy imports should just work transparently and no teaching of -the feature should be necessary. - -The implementation will ensure that errors resulting from a deferred import -have metadata attached pointing the user to the original import statement, to -ease debuggability of errors from lazy imports. +Since the feature is opt-in, beginners should not encounter it by default. +Documentation of the ``-L`` flag and ``PYTHONLAZYIMPORTS`` environment variable +can clarify the behavior of lazy imports. Some best practices to deal with some of the issues that could arise and to better take advantage of lazy imports are: @@ -272,10 +316,6 @@ better take advantage of lazy imports are: not only the one being accessed, because the parent module ``foo`` is the actual deferred object name. -* Don't use inline imports, unless absolutely necessary. Import cycles should - no longer be a problem with lazy imports enabled, so there’s no need to add - complexity or more opcodes in a potentially hot path. - Reference Implementation ======================== @@ -334,13 +374,35 @@ show up soon enough to be caught and fixed (unless the import is truly unused.) Lazy dynamic imports -------------------- -It would be possible to add a ``lazy=True`` or similar option to ``__import__`` -and/or ``importlib.import_module()``, to enable them to perform lazy imports. -That idea is rejected in this PEP for lack of a clear use case. Dynamic imports -are already far outside the :pep:`8` code style recommendations for imports, -and can easily be made precisely as lazy as desired by placing them at the -desired point in the code flow. These aren't commonly used at module top level, -which is where lazy imports applies. +It would be possible to add a ``lazy=True`` or similar option to +``__import__()`` and/or ``importlib.import_module()``, to enable them to +perform lazy imports. That idea is rejected in this PEP for lack of a clear +use case. Dynamic imports are already far outside the :pep:`8` code style +recommendations for imports, and can easily be made precisely as lazy as +desired by placing them at the desired point in the code flow. These aren't +commonly used at module top level, which is where lazy imports applies. + + +Deep eager-imports override +--------------------------- + +The proposed ``importlib.eager_imports()`` context manager and +``importlib.set_eager_imports()`` override both have shallow effects: they only +force eagerness for the location where they are applied, not transitively. It +would be possible (although not simple) to provide a deep/transitive version of +one or both. That idea is rejected in this PEP because experience with the +reference implementation has not shown it to be necessary, and because it +prevents local reasoning about laziness of imports. + +A deep override can lead to confusing behavior because the +transitively-imported modules may be imported from multiple locations, some of +which use the "deep eager override" and some of which don't. Thus those modules +may still be imported lazily initially, if they are first imported from a +location that doesn't have the override. + +With deep overrides it is not possible to locally reason about whether a given +import will be lazy or eager. With the behavior specified in this PEP, such +local reasoning is possible. Copyright From 5429b14de852137d7a9ba1b561edc95d3270b3d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Germ=C3=A1n=20M=C3=A9ndez=20Bravo?= Date: Tue, 3 May 2022 11:19:23 -0700 Subject: [PATCH 18/18] Update pep-0690.rst Co-authored-by: Jelle Zijlstra --- pep-0690.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0690.rst b/pep-0690.rst index c6bb9a90615..bf57d9fc741 100644 --- a/pep-0690.rst +++ b/pep-0690.rst @@ -202,7 +202,7 @@ disabled, which is the default. Even when enabled, most code will continue to work normally without any observable change (other than improved startup time and memory usage.) -Namespace packages are not effected: they work just as they do currently, +Namespace packages are not affected: they work just as they do currently, except lazily. In some existing code, lazy imports could produce currently unexpected results