Numba dependency

[CommonClimate]

Pyleoclim currently depends on Numba for the Weighted Wavelet Z-transform implementation (see wavelet.py), but there are concerns that numba is not under very active development, yield potential incompatibilities with NumPy down the line.

Since our use of Numba is pretty much restricted to one loop, I am wondering if there might be better options out there. @rgommers would you care to weigh in? Note that we would have used CPython if we knew any C, as the original WWZ code from ETH's James Kirchner is in C.

[rgommers]

@CommonClimate I don't see too much of a worry with Numba maintenance at the moment. It's better maintained than any of the alternatives (like Cython or Pythran) I'd say, and that won't change any time soon.

More importantly, while Numba is a fairly heavy runtime dependency, that is still much better than incorporating C/Cython code probably. That would mean you'd either have a significant burden of producing O(30) binary wheels if you distribute via PyPI, or have your users now needing a compiler if you build from source (especially problematic if you have Windows users). I'd personally try hard to stay pure Python only for as long as you can.

Two other options come to mind:

1. If kirchner_numba isn't needed internally by too much other code and isn't key to the whole package, you could make it an optional feature to avoid the runtime dependency if that's a worry. Basically that means moving import numba inside the function and adding an informative message when the import fails because numba is not installed.
2. If you'd want to write it in C/Cython without the cost of having to build binary wheels, perhaps there's a package you can contribute that C/Cython code to. E.g., it may fit into Pywavelets. That's a safe dependency, e.g. scikit-image depends on it.

[CommonClimate]

Thanks @rgommers for these insights. I'll confer with @fzhu2e for the best course of action. It's definitely good to know that numba is well maintained for the foreseeable future.

The current version requires NumPy <1.22, which naturally created an error when I tried to update NumPy. It looks like that issue is being worked on, but I was curious why that restriction popped up for Numba and not other NumPy-dependent packages (i.e. pretty much everything in our stack). Is this a package management (pip vs conda) thing?

[rgommers]

No, it's not dependent on the package manager. Numba itself raises an exception, so all packages managers must pin to <1.22), and I verified that conda-forge indeed does this.

This is done for all (fragile) Numba dependencies, see numba/numba#7661. Numba 0.55.0 came out before NumPy 1.22.0, hence that pin to avoid breakage. I think the reason why other packages either don't pin at all or have an upper bound further into the future (like SciPy) is that Numba is quite sensitive to NumPy internals changes, and they don't want a new NumPy release to break an already released Numba version.

There could be some package manager dependency there in principle, because conda-forge, Spack and other package managers allow editing metadata after release while PyPI does not (so a lack of upper bounds is harder to fix there).

All this said, the Numba authors could have bumped the pin for 0.55.1, which was released after NumPy 1.22.0. I will open an issue to ask about that, because I agree the current situation is annoying.

[rgommers]

Actually no, it's not changed in 0.55.1 because there are a bunch of problems still that need to be resolved: numba/numba#7754. That issue illustrates that Numba needs quite some work to be adapted to changes in NumPy.

[CommonClimate]

Indeed - it does make sense. I suppose we'll have to be patient that all these issues have been worked out - I will be better about not rushing to the latest NumPy version next time. Thank you for your explanations, which help me navigate this complex terrain!