micropip.install("<...>") does not uphold custom PyPI indices

[agriyakhetarpal]

A description of the issue

Last year, #74 added support for setting a custom index URL that adheres to the PEP-503-based simple repository APIs for the purpose of installing files from an index different from PyPI.

However, I cannot seem to make this work while setting up interactive examples on Read the Docs for the development version of a package, here is a reproducer: PyWavelets/pywt#728

Some steps to reproduce

In the Examples section on the Read the Docs PR preview for a simple API reference, say, here: https://pywavelets--728.org.readthedocs.build/en/728/ref/wavelets.html#wavelet-families

I am trying to run this code snippet inside a code block in the Pyodide-enabled JupyterLite notebook (that can be opened with the "Try it in your browser" button):

import micropip
await micropip.install("pywavelets", index_urls="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple")

and I even tried this:

micropip.set_index_urls(urls="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple")
await micropip.install("pywavelets")

but micropip.list() still prints out (note: truncated output):

PyWavelets         | 1.4.1   | pyodide

rather than the version I am trying to install (1.6.0rc1 at the time of writing), which doesn't make sense – so I tried reloading the package with importlib.reload() and importlib.invalidate_caches(), but to no avail – it is still 1.4.1. This suggests that the micropip.install() invocation might be failing silently.

Expected behaviour

It would be great to see the development version of PyWavelets getting downloaded and installed (i.e., v1.6.0). For interactive examples using jupyterlite-pyodide-kernel, it is not possible to pre-install wheels (unlike xeus-kernel, where pre-loading wheels is possible through emscripten-forge), so we are adding an admonition that displays the command to execute an installation of the wheels for interested users.

---

P.S. A workaround I am considering is to download the WASM wheel by overriding the Read the Docs documentation build process, place it into an appropriate location as listed in https://jupyterlite.readthedocs.io/en/latest/howto/pyodide/wheels.html, and proceed to install from the wheel file from there. However, being a workaround – this has negative implications and is too hacky to use, since it would be expected for the functionality added to just work.

[agriyakhetarpal]

I suppose #51 and #64 are related to this behaviour and their completion will bring the functionality to update an already installed package? Even so, with the index_urls argument – an error message is not shown (which is likely another bug).

[ryanking13]

@agriyakhetarpal Thanks for the report.

What happens is, pypi.anaconda.org does not set CORS header on the response, so micropip cannot download the package from there. Maybe some pyscript folks can help about this? (maybe @fpliger?) I think Roman had contacted Anaconda before about this but I am not very sure... (Related: pyodide/pyodide#3049 (comment))

So because of CORS issue, micropip silently fallbacks to pyodide-lock.json and download PyWavelets 1.4.1 that is listed in the lockfile.

[agriyakhetarpal]

Right, thanks, @ryanking13 for the extra logs. I did try to download PyWavelets from pypi.anaconda.org previously locally, directly from the URL, and I had faced this issue about the lack of CORS headers. I don't think it should be worked around, though – couldn't it potentially be the cause of security issues?

[fpliger]

@ryanking13 thanks for the ping!

Let me check with the team and will report back.

/Cc @chenglee @jezdez

[ryanking13]

Great! Thanks for the help, Fabio!!

[lesteve]

I think Roman had contacted Anaconda before about this but I am not very sure...

Yes, I can confirm Roman contacted Anaconda before 😉.

As a scikit-learn developer who worked on adding interactivity to the scikit-learn documentation and improving Pyodide support for scikit-learn and scipy, I would be super interested to see CORS headers set on anaconda.org, although I can imagine there are complexities involved.

This would allow to have more control on the scikit-learn version inside JupyterLite and match the one for the documentation. For example right now if you go to this scikit-learn example the documented scikit-learn version is 1.4.1 but the scikit-learn version inside JupyterLite is 1.3.1 which is what Pyodide 0.25.1 comes with. So this particular example gives confusing errors because it documents new features in 1.4 features and the scikit-learn version inside JupyterLite is 1.3.1.

Also Numpy has started to publish Pyodide nightly wheels on anaconda.org see https://anaconda.org/scientific-python-nightly-wheels/numpy/files/ and numpy/numpy#26134. Having CORS headers set on anaconda.org will help a lot making them more conveniently usable, e.g. with micropip.install(..., index_urls='https://pypi.anaconda.org/...) or %pip install inside a notebook.

[agriyakhetarpal]

To add to this, I'm not sure how much work it would require on JupyterLite's end but to make this step automated it would be great to set the index URL (or a list of them if one wants to choose multiple ones) in conf.py and pre-load these wheels automatically with the Pyodide kernel instead of asking users to do it in a cell manually. It's probably not the right place to request that feature, but I wanted to post this comment to highlight what would mostly be the plan with these nightly wheels going forward.

[fpliger]

Just to keep folks informed, I haven't dropped the ball here, just going through internal communications

[lesteve]

As a follow-up to the CORS headers on anaconda.org, I posted pyodide/pyodide#3049 (comment) to try to keep the conversation in a single place 😉