Optimise import qiskit with lazy imports

[jakelishman]

Summary

• Speed up import qiskit significantly - 1.17s to 0.57s (2x faster) for a Python environment with almost all the optional dependencies, and 0.78s to 0.52s (33% faster) for a Python environment with only Terra and its requirements.
• Add a new module qiskit.utils.optionals, with many lazy-evaluation HAS_X variables (e.g. HAS_MATPLOTLIB, HAS_AER), which can be used for lazy testing of optional dependencies, and unified handling of errors if they are not present.

Details and comments

These two commits significantly speed up import qiskit by deferring many imports to their call sites. In particular, no component of Scipy is loaded, nor is pkg_resources (which is surprisingly heavy). I didn't change any scipy imports in algorithms or opflow, because these aren't imported by default, and I thought they might be a bit more work for lesser benefit.

The first commit in this patch also introduces many new HAS_X variables, to unify handling of optional dependencies all the way through the codebase. Previously, many modules were doing ad-hoc testing with

try:
    import optional_dependency
    HAS_OPTIONAL = True
except ImportError:
    HAS_OPTIONAL = False

class MyClass:
    def __init__(self):
        if not HAS_OPTIONAL:
            raise MissingOptionalLibraryError(...)

This form has a couple of downsides:

1. We try to import the optional dependency at import time, which adds to the weight of import qiskit if the dependency is installed, even if it won't be used.
2. It clutters many functions with error-handling code, even though they are doing the same thing.
3. Several modules were repeating the exact same code in a couple of places.

The first commit introduces some new lazy-loader objects (LazyImportTester and LazySubprocessTester), which can be evaluated as Booleans, and will lazily test the presence of the dependencies. This is similar to the old HAS_MATPLOTLIB variable. There is a new module qiskit.utils.optionals that exposes many HAS_X variables of these types. In addition to the lazy Boolean checkers, they also come with unified error handling: modules now can do

@HAS_MATPLOTLIB.require_in_call
def needs_matplotlib(self):
    import matplotlib

def chooses_based_on_argument(self, arg):
    if arg == "text":
        ...
    elif arg == "matplotlib":
        HAS_MATPLOTLIB.require_now("matplotlib drawer")
        import matplotlib
        ...

Both require forms will raise a suitable MissingOptionalLibraryError if the dependency is not available, at the point of call - the decorator raises when the inner function is called, not when the decorator is applied.

Moving the variables into qiskit.utils.optionals is also nice for the test suite - instead of it having to repeat the code or import names from random locations, there's a simple namespace where all the objects exist (with documentation on each one in Sphinx).

Fix #7498, as a side-effect - all the instances where initialisation code was logging are now deferred to their usage sites, so the logs will actually make sense and be useful. You can import every single module defined in Terra without us issuing a log message now.

[coveralls]

Pull Request Test Coverage Report for Build 1783494894

• 423 of 548 (77.19%) changed or added relevant lines in 54 files are covered.
• 23 unchanged lines in 13 files lost coverage.
• Overall coverage increased (+0.2%) to 83.354%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
qiskit/opflow/gradients/hessian.py	5	6	83.33%
qiskit/transpiler/passes/routing/algorithms/bip_model.py	4	5	80.0%
qiskit/utils/classtools.py	64	65	98.46%
qiskit/visualization/circuit_visualization.py	5	6	83.33%
qiskit/visualization/gate_map.py	6	7	85.71%
qiskit/visualization/matplotlib.py	4	5	80.0%
qiskit/visualization/pulse/interpolation.py	0	1	0.0%
qiskit/visualization/pulse/matplotlib.py	3	4	75.0%
qiskit/visualization/utils.py	4	5	80.0%
qiskit/algorithms/optimizers/nlopts/nloptimizer.py	2	4	50.0%

Files with Coverage Reduction	New Missed Lines	%
qiskit/algorithms/optimizers/nlopts/nloptimizer.py	1	40.63%
qiskit/circuit/parameterexpression.py	1	87.4%
qiskit/circuit/qpy_serialization.py	1	87.14%
qiskit/tools/jupyter/init.py	1	0%
qiskit/tools/jupyter/job_watcher.py	1	0%
qiskit/tools/jupyter/progressbar.py	1	0%
qiskit/transpiler/passes/optimization/crosstalk_adaptive_schedule.py	1	9.36%
qiskit/visualization/pass_manager_visualization.py	1	18.84%
qiskit/transpiler/passes/optimization/hoare_opt.py	2	11.61%
qiskit/visualization/pulse/matplotlib.py	2	13.17%

Totals
Change from base Build 1781777612:	0.2%
Covered Lines:	52223
Relevant Lines:	62652

---

💛 - Coveralls

[woodsp-ibm]

I guess one downside of the new way, from a user perspective, I write code with a bunch of imports and it all seems fine as it loads and starts running. But instead of it failing fast as it used to, I now find it fails some time later if some optional dependency is not present when it finally goes to construct/uses something. The code may have expended significant computational resources by that point. And I might fix that and run again, go further and hit another missing optional. The fail early that was done before would catch the optional dependence early; but has other considerations that you point out such as loading resources that are never used in practice.

[jakelishman]

This doesn't actually change anything about that - before, all the imports would silently fail immediately, setting (e.g.) _HAS_SKQUANT = False, and then you wouldn't get an error til you instantiated the class. In this form, you still get the error in the exact same place, but you also don't pay the import cost til then either.

[woodsp-ibm]

Ah right, as you say the behavior in terms of it failing later is unaltered by your change. I know in Aqua we used to indicate the failure to load but guess that was removed as it may have been too noisy if you did not care about installing the optional.

[jakelishman]

Yeah, I don't know when it changed, but it was probably the right call - this PR identified 26 different optional dependencies across Terra alone. The code from Aqua seems to be responsible for about 4 of them, which is a bit more reasonable, but still, they're pretty niche optimisers, and I'd imagine most people wouldn't be bothered. In general, you should get the error when you instantiate the optimiser classes, which is likely going to be one of the first things you do.

[mtreinish]

Awesome thanks for doing this. The API around managing optional dependencies is so much cleaner here and much easier to manage than the ad-hoc solution we were doing before. It's nice enough that I'm slightly tempted to violate the stable branch policy so we can start using this in downstream qiskit packages sooner rather than later :) That coupled with fixing the stray debug logging and the import time improvements this really nice!

Just have a few questions and comments inline

[mtreinish]

Oh, too bad I was excited when I opened #7498 that we'd have the first use case for module level getattr now that we're >=3.7 but I guess not. Having this comment here is useful for explaining this I should have added one for the lazy aer and ibmq wrappers

[jakelishman]

Oh, I forgot that module-level getattr was available now - the getattr I'm referring to here is the class one (i.e. why I defined all the methods manually). I think it might end up a bit convoluted here - it also needs to be reachable from qiskit.quantum_info.synthesis, and we'd need to protect an internal Qiskit import in other places, which would be a bit weird.

[mtreinish]

This is a good catch but kind of an unrelated change.

[jakelishman]

Oh yeah, it's because I put in a Sphinx cross-ref to it somewhere (I think in the HAS_Z3 docs?), and it didn't link correctly - it looks totally unrelated, but there was some method in the madness.

[mtreinish]

Is there a reason to not deprecate these now?

[jakelishman]

The alternate paths aren't available in Terra 0.19, so we can't warn until the alternative is already in place, and I know that both Ignis and ibmq-provider import HAS_MATPLOTLIB from here (technically qiskit.tools.visualization, but that's just a pass-through to here).

[mtreinish]

Wow, something is still using qiskit.tools.visualization we made that move like 3 years ago. :)

[jlapeyre]

A couple of lines in CONTRIBUTING.md might be useful. Something about when to add a dependency to requirements.txt and when/how to add it to optionals.py.
(This looks so useful, I'm surprised there's no package for it, or at least how-to for something similar)

[jakelishman]

Added a section into CONTRIBUTING in fd74223.

The dependency-checker classes themselves I suppose could be split off into a separate package, but there would be some additional work around making the error handling nice and extensible there. I think the majority of the benefit comes from defining all the HAS_X variables, and those would still need to be written by each project, as in utils.optionals.

[mtreinish]

This LGTM. I think everything is covered here. Thanks for doing this. I left a few questions/comments inline but nothing work blocking this over. Some things we can always adjust in follow ups as necessary, especially given the size of this PR it'll be good to merge this sooner rather than later.

[mtreinish]

Heh, I appreciate the thoroughness on this class and the details look good to me. I'm just wondering if these edge cases came up in practice in the use of the class decorator, or this was just to be defensive against something needs them currently. Like I guess I'm wondering if something is trying to use _WrappedMethod on a dict subclass or something.

[jakelishman]

They come up far more than you'd expect - things like wrapping method from type and object are part of the core usage - wrapping a non-existent __init__ needs to access something from object, and wrapping __init_subclass__ (which we do in the test classes) needs to access something from type. Wrapping classmethod and staticmethod instances I think already happens in what we use it for now.

Using builtins as the wrappers happened when I was designing the class in the interpreter - I tried to use print as a wrapper for a really poor man's debugger.

[mtreinish]

A potential follow-on, not something we should do in this pr. Is to add a skip test decorator to the lazy importer class. So we can just make this @optionals.HAS_JAX.test_requires or something

[jakelishman]

I'm not 100% sure on that - it would couple the objects to our testing framework, and that's not ideal for something we're publicly exporting.

[mtreinish]

Well I was thinking more just having the decorator raise unittest.SkipTest which is in stdlib and while it's specifically part of unittest framework I think all the python testing frameworks support it. But it's not critical and I agree a gray area of whether we want that as part of the api. We can discuss it later if we think it's something worthwhile

[mtreinish]

Is there an advantage to this vs L71? This should be equivalent to self.assertFalse(test_generator())

[jakelishman]

Yeah, it's a slightly superfluous test - I'm not 100% sure what I was thinking, but I suspect it was along the lines of "check a few possible implicit Boolean contexts". Strictly, all three tests here should be identical, I think. Happy to remove it (and the similar test on line 65 if you'd prefer to reduce noise.

[mtreinish]

Marking this as on hold until we get all the 0.19.2 PRs backported (just to avoid potential merge conflicts). After we're ready to release 0.19.2 I'll tag this as automerge.