Populate namespace: lazy_loader

[rcomer]

Since Iris v3.1 and #4174, some Iris subpackages/modules that used to be available automatically no longer are and must be imported explicitly. Some users have raised concerns about this (#4683 and various Yammer threads).

I stumbled across this package that attempts to square the circle of convenient namespaces and import performance:
https://github.com/scientific-python/lazy_loader

It seems its authors are keen for this approach to be adopted across the Scientific Python Ecosystem:
https://scientific-python.org/specs/spec-0001/

In our case, it would mean a user could do something like

import iris

wibble = iris.cube.CubeList()

and iris.cube would be imported when it is accessed.

I'm not suggesting we should immediately adopt this because

• lazy_loader is a new, two-man project and not [yet?] widely adopted
• It's possible that once the initial move to Iris v3.1 has bedded in, Iris users will stop caring about needing the extra imports.

But maybe something to have in mind depending how things develop.

[jamesp]

I think, in general, it is a bad idea to monkey patch core functionality of the language. Not always, pytest is a good example, but we need a really good reason.

Reasons I think this is not desirable:

1. It is "magic". As a user you're not quite sure how it works, and it is different to other libraries you use. Principle of least surprise.
2. This is another dependency
3. it is additional code that is completely orthogonal to the core functionality of iris.

It is regrettable that:

• the import re-org didn't happen at 3.0
• the change was not better communicated

but I'm not sure the answer is more code and more dependencies. I'm willing to have my mind changed by a compelling counterpoint though!

[rcomer]

For completeness, here is the matplotlib discussion that brought me here:
matplotlib/matplotlib#22247

[jamesp]

To be fair, the import system of Python is confusing even without these kind of patches, especially for submodules. For example:

>>> import iris.cube
>>> iris.analysis
<module 'iris.analysis' >
>>> # wtf but I didn't import `iris.analysis`!

As iris/cube.py imported iris.analysis it is now available in the iris super-namespace without being explicitly imported. It's intuitive to think "If I import iris I should also get all child namespaces like iris.cube". But I'm not sure that it is intuitive to think "if I import iris.cube I should also get (some!) sibling namespaces of iris.cube like 'iris.analysis" but that is what happens in Python. 🤷‍♂️

[trexfeathers]

I don't see any way round this either! We could just import the bits we need (from iris.analysis import MEAN), but then those objects would each be available after importing iris.cube. The same would be true if we aliased iris.analysis.

No clean solutions on SO...
https://stackoverflow.com/questions/16509012/hide-external-modules-when-importing-a-module-e-g-regarding-code-completion

[jamesp]

I don't necessarily think we need to find a way around it or to solve it, only to say that the import system is complicated and some users mental model might be wrong based on the empirical experiences they've had in the past. More support and education on this topic would help (especially as I read the matplotlib issue @rcomer posted above to mean that they may want to change the default imports in the future too)

[trexfeathers]

some users mental model might be wrong based on the empirical experiences they've had in the past

Preach! Very well put

[pp-mo]

FYI this idea has been proposed in the "scientific python org" : SPEC0001
( No idea how important or influential that proposed "umbrella" group actually is, or might be ? )
The history on the proposal page is well worth reading.
The point is, they are suggesting that lazy-loading should become standard for scientific python libraries, as part of a set of general principles they are proposing

Re the dependency issue : The suggested 'lazy-loader' is only one possible solution : we can instead easily provide our own, since the basic idea is quite simple.
The way it occurs in the source, it makes slightly life more complicated for developers, if anyone, not really for users.

I agree that providing lazy-loading is sort of "not what Python does"
But then, the "magical" appearance of subpackages is what Python does, and IMHO it does remain a constant problem for (non-expert) users.
The question, really, is how we address the problem, in terms of how we support our users.

I think there are only a few practical options

1. wild-west, as it was (probably not)
   • some subpackages are always present, others not.
   • we didn't "support" this -- results were not always stable between versions
2. minimal imports, as we currently have.
   • encourage users to use "safe" explicit imports for everything
   • but we can't prevent users from relying on side-effect loading of subpackages
   • with isort, we are now seeking to minimise unused imports, which solves a different problem.
     It may help somewhat in forcing the issue into user understanding, but it doesn't solve the problem.
3. guarantee presence of common "basic subpackages" as standard : support, document and maintain those
   • advice + caveats, just as above, except that some few things can be relied on
4. lazy-loading to hide "the problem"
   The idea that all sub-packages are implicitly always "available", while still being loaded incrementally, is certainly a fix for the problem.
   However, it has a cost, and it may not be such a good idea to "save users from themselves" in this way, if it simply defers learning of good practice. I note that in the "SPEC0001" proposal, they suggest still recommending the use of full imports

[jamesp]

What if we reverted the change and go back to import iris.cube by default in iris __init__.py? We shouldn't kid ourselves that this is a real-world "performance hit" on importing iris, as surely iris is entirely pointless without creating a cube, at which point you need to import iris.cube...

[rcomer]

I understood that the speedup was a happy side-effect rather than the main point of the change, which was more about the mess of circular imports. I may have misunderstood.

[rcomer]

I also think I do prefer 3.1 for working interactively...

>>> import iris  # irritated waiting for this
>>> cubes = iris.load(my_file)  # expect this to take a while, so I'll wander off and browse GitHub...

[trexfeathers]

See GeoVista for a worked example of using lazy_loader. Would be great to look into this for Iris.

[bjlittle]

This recent conversation across on pyvista is kinda relevant ... just cross-referencing for awareness 👍