WIP: CI: switch Emscripten job to Meson #24603

rgommers · 2023-08-31T15:13:17Z

This is one of the last CI jobs to be converted to Meson. Still a work in progress.

Note that we need to cross-compile here. Using pyodide build fails on not having a cross file. The source() function in pyodide_build/cli/build.py needs a --backend-flags parameter so that we can pass config-settings arguments to the backend. In the meantime, we use python -m build ....

One can also try this locally with:

$ python -m build -wnx -Csetup-args=--cross-file=\$PWD/tools/ci/emscripten.meson.cross  -Csetup-args=-Dallow-noblas=true

as long as both pyodide-build and the exact version of Emscripten (currently 3.1.32, can be installed with https://emscripten.org/docs/tools_reference/emsdk.html) are available. The configure stage of the build looks fine, the build itself fails halfway through due to:

include/python3.11/pyport.h:601:2: error: "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."

That issue is due to a static assert in pyconfig.h, also discussed at: pyodide/pyodide#2494 (comment)

See also pyodide/pyodide#2238 and the patches to _numpyconfig.h in https://github.com/pyodide/pyodide/tree/main/packages/numpy.

tl;dr this needs some work, and it seems like cross-compilation is hard because Pyodide doesn't ship its own Python interpreter that we can actually target during a direct cross build.

Cc @hoodmane @rth. I hope I can bother you with this. If I got any of the above wrong or if you've got a suggestion to get past the LONG_BIT error, I'd love to hear it. Also, a couple of questions:

If I didn't overlook a way to pass --config-settings flags to the backend in the pyodide build CLI, would you accept a PR to Pyodide to add that? It should match backend-flags in meta.yaml recipes I think.
Since scikit-image can be cross-compiled, I had expected to be able to make that work for numpy too. But after reading this comment, that may not be true? Is that still the case, and do you see a road to a regular cross-compile working for NumPy? It's actually not clear to me how the distutils-based build works right now, it doesn't seem to be doing the "playback" thing explained in that comment.

[skip ci]

rgommers · 2023-08-31T15:16:44Z

tools/ci/emscripten.meson.cross

+endian = 'little'
+
+[properties]
+longdouble_format = 'IEEE_QUAD_LE'


This is the only addition compared to the upstream emscripten.meson.cross file in Pyodide - necessary for now.

Most of this file can be removed, only this is needed:

[properties] longdouble_format = 'IEEE_QUAD_LE'

hoodmane · 2023-08-31T16:04:11Z

cc @ryanking13 @henryiii

rgommers · 2023-09-11T09:26:48Z

I decided that it's easier to finish this only after upgrading the build of numpy that Pyodide does in-tree to Meson and 1.26.x first. I've made a start on that, but I need to do some more digging. There was also a pip bug that made that a little more difficult. And pyodide/pyodide#4118 is the same LONG_BIT error as I'm seeing in this PR.

Next steps I plan to take (not necessarily in this order):

disable this CI job and remove all setup.py files
finish unvendoring meson-python from this repo after the next meson-python release
upgrade numpy in Pyodide to 1.26.0 + Meson
if needed, ensure the out-of-tree Pyodide build can take the same backend-flags as the in-tree build
re-enable this CI job on the main branch

rth · 2023-09-11T09:39:47Z

Thanks for working on this @rgommers ! I don't have useful suggestions about the errors you are facing but I agree it would probably be easier to do this in the in-tree build first. Also cc @lesteve who worked on the scikit-image meson build.

If I didn't overlook a way to pass --config-settings flags to the backend in the pyodide build CLI, would you accept a PR to Pyodide to add that?

I think overall we could be happy to accept any changes needed to make numpy with meson work. But since pyodide build aims to be a drop-in replacement for python -m build, we need to double check in particular whether this would still be OK with cibuildwheel, or if there is some other recommended way of passing those config settings.

rgommers · 2023-09-11T09:47:08Z

But since pyodide build aims to be a drop-in replacement for python -m build, we need to double check in particular whether this would still be OK with cibuildwheel, or if there is some other recommended way of passing those config settings.

That should be fine for cibuildwheel I think. python -m build -Ckey=value will add key=value to the config_settings dict and then pass that on to the backend. -C/--config-setting is implemented in build, and -C/--config-settings in pip (note the extra s for pip in the long form). I am missing that flag from pyodide build --help.

ryanking13 · 2023-09-11T12:36:38Z

That should be fine for cibuildwheel I think. python -m build -Ckey=value will add key=value to the config_settings dict and then pass that on to the backend. -C/--config-setting is implemented in build, and -C/--config-settings in pip (note the extra s for pip in the long form). I am missing that flag from pyodide build --help.

Currently pyodide build passes extra command line arguments to config-settings (backend-flags), so pyodide build key1=val1 is equivalent to python -m build -C key1==val1 (it is not documented sorry). But yes, we should change it to -C/--config-settings to match the behavior of pypabuild and pip.

rgommers · 2023-09-11T15:44:20Z

Thank you @ryanking13, that's exactly what I needed.

Initial inspiration was taken from numpy/numpy#24603.

Moving it to Meson is going to take a bit of time, and in the meantime we have to disable it, because it's the only job that still needs the `setup.py` based build and we'd like to remove support for building with `setup.py`.

hoodmane · 2024-02-09T18:29:10Z

@ryanking13 Do you know what the status of this is? It would be good if we could look into this again. I would look into it myself but so far I don't know much about meson...

rgommers · 2024-02-09T20:56:26Z

I've been meaning to get back to this PR. I believe with @ryanking13's changes for improved Meson/meson-python support in Pyodide, this should be unblocked. Timing wise I realistically won't revisit it before the NumPy 2.0 release settles down though (any week now ...).

ryanking13 · 2024-02-11T02:09:45Z

Yes, now that we're building numpy with meson in-tree, I don't think it's going to be hard to make it work in out-of-tree too. We've recently added a patch for python-meson >= 0.15 so I think we need to release pyodide-build (0.25.1).

before the NumPy 2.0 release settles down

Maybe emscripten CI should also wait for numpy 2.0 to be released? I've seen Pyodide builds break in many cases when packages do major releases, so it might be better to wait and see if the numpy 2.0 build works in-tree before applying it.

rgommers · 2024-02-11T15:59:05Z

Maybe emscripten CI should also wait for numpy 2.0 to be released? I've seen Pyodide builds break in many cases when packages do major releases, so it might be better to wait and see if the numpy 2.0 build works in-tree before applying it.

That seems reasonable to me. EDIT: no real reason not to do it now though, it's just not a priority compared to 2.0-critical tasks.

rgommers

I added some comments to this file about what is still relevant. @agriyakhetarpal is going to have a fresh attempt at a new/updated CI job.

rgommers · 2024-02-23T11:22:04Z

.github/workflows/emscripten.yml

@@ -52,14 +52,23 @@ jobs:
          actions-cache-folder: emsdk-cache

      - name: Install pyodide-build
-        run: pip install "pydantic<2" pyodide-build==$PYODIDE_VERSION
+        run: pip install "pydantic<2" build pyodide-build==$PYODIDE_VERSION


build isn't needed anymore, I think pyodide-build will do fine.

rgommers · 2024-02-23T11:24:07Z

.github/workflows/emscripten.yml

-          cp pyproject.toml.setuppy pyproject.toml
-          CFLAGS=-g2 LDFLAGS=-g2 pyodide build
+          # Note that we need to cross-compile here. Using `pyodide build`
+          # fails on not having a cross file. The `source() function in


This is no longer true, the cross file is shipped by pyodide now.

Update: still true actually, the cross file won't get applied as long as we have vendored-meson.

We also need the fix from pyodide/pyodide#4502 to actually pick up vendored-meson. With that, an out of tree build should work.

I am on it – I shall be adding a patch file to port the above PR into vendored-meson, as discussed on Slack.

rgommers · 2024-02-23T11:24:43Z

.github/workflows/emscripten.yml

+          # tl;dr this needs some work, and it seems like cross-compilation is
+          # hard because Pyodide doesn't ship its own Python interpreter that
+          # we can actually target during a direct cross build.
+          python -m build --wheel -Csetup-args=--cross-file=$PWD/tools/ci/emscripten.meson.cross -Csetup-args=-Dallow-noblas=true


The -Dallow-noblas=true is no longer needed, the fallback to internal code when BLAS is missing is automatic.

The -Csetup-args=--cross-file=$PWD/tools/ci/emscripten.meson.cross will still be needed I think, only to add the longdouble format, just in slightly different format. From this comment I think it should be:

pyodide build setup-args="--cross-file=$PWD/tools/ci/emscripten.meson.cross"

Pyodide bundles emscripten.meson.cross used in in-tree build, so probably you can try that one too.

PYODIDE_MESON_CROSS_FILE=`pyodide config get meson_cross_file` pyodide build -C "setup-args=\"${PYODIDE_MESON_CROSS_FILE}\""

rgommers · 2024-02-23T11:28:08Z

.github/workflows/emscripten.yml

@@ -77,4 +86,4 @@ jobs:
        run: |
          source .venv-pyodide/bin/activate
          cd ..
-          python numpy/runtests.py -n -vv
+          pytest --pyargs numpy -m "not slow"


This change is still needed.

rgommers · 2024-02-23T11:28:25Z

meson_cpu/meson.build

@@ -106,6 +107,7 @@ max_features_dict = {
  's390x': S390X_FEATURES,
  'arm': ARM_FEATURES,
  'aarch64': ARM_FEATURES,
+  'wasm32': {},


the changes in this file will still be needed.

rgommers · 2024-02-23T11:28:58Z

tools/ci/emscripten.meson.cross

+endian = 'little'
+
+[properties]
+longdouble_format = 'IEEE_QUAD_LE'


Most of this file can be removed, only this is needed:

[properties] longdouble_format = 'IEEE_QUAD_LE'

Copied with updates and suggestions received from numpy#24603 on 23/02/2024.

This commit performs the following actions: 1. Adds WASM builds to the CPU family for Meson configurations, but without SSE or SIMD instructions. 2. Removes CPU feature detection message for an unsupported architecture. 3. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target (cross-builds). 4. Enables run for Emscripten/Pyodide wheels by setting the `if:` condition to `true`. 5. Uses recursive submodules to ensure that vendored-meson is received. 6. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a separate Emscripten folder to store relevant files) 7. Adds a patch for vendored-meson detection for Pyodide and applies this Pyodide-meson patch in the Emscripten CI jobs 8. Builds wasm32 wheels without BLAS and LAPACK support (see numpy#24750 (comment)) 9. Forces coloured and prettified outputs for test runs Some of these changes have been copied with updates and suggestions received from numpy#24603 on 23/02/2024 and authorship is preserved with this commit. Co-Authored-By: Ralf Gommers <[email protected]>

This commit performs the following actions: 1. Adds WASM builds to the CPU family for Meson configurations, but without SSE or SIMD instructions. 2. Removes CPU feature detection message for an unsupported architecture. 3. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target (cross-builds). 4. Enables run for Emscripten/Pyodide wheels by setting the `if:` condition to `true`. 5. Uses recursive submodules to ensure that vendored-meson is received. 6. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a separate Emscripten folder to store relevant files) 7. Adds a patch for vendored-meson detection for Pyodide and applies this Pyodide-meson patch in the Emscripten CI jobs 8. Builds wasm32 wheels without BLAS and LAPACK support (see numpy#24750 (comment)) 9. Forces coloured and prettified outputs for test runs Some of these changes have been copied with updates and suggestions received from numpy#24603 on 23/02/2024 and authorship is preserved with this commit. [skip cirrus] [skip circle] [skip azp] Co-Authored-By: Ralf Gommers <[email protected]>

rgommers · 2024-02-27T16:59:54Z

This PR is superseded by gh-25894, things seem to work there. So I'll close this PR. Thanks for the help everyone!

hoodmane · 2024-02-28T00:32:53Z

Thanks for working on this @rgommers and @agriyakhetarpal!

This commit performs the following actions: 1. Adds WASM builds to the CPU family for Meson configurations, but without SSE or SIMD instructions. 2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target (cross-builds). 3. Enables run for Emscripten/Pyodide wheels by setting the `if:` condition to `true`. 4. Uses recursive submodules to ensure that vendored-meson is received. 5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a separate Emscripten folder to store relevant files) 6. Adds a patch for vendored-meson detection for Pyodide and applies this Pyodide-meson patch in the Emscripten CI jobs 7. Builds wasm32 wheels without BLAS and LAPACK support (see numpy#24750 (comment)) 8. Forces coloured and prettified outputs for test runs Some of these changes have been copied with updates and suggestions received from numpy#24603 on 23/02/2024 and authorship is preserved with this commit. [skip cirrus] [skip circle] [skip azp] Co-Authored-By: Ralf Gommers <[email protected]> Co-authored-by: Hood Chatham <[email protected]>

Initial inspiration was taken from numpy/numpy#24603.

* Test emscripten workflow inspired by NumPy Adds a CI job to build and test scikit-image in a Pyodide virtual environment through WASM wheels. Following this, it would be possible to include JupyterLite notebooks in the documentation in order to run scikit-image's code snippets which come with docstring-based examples. Initial inspiration was taken from numpy/numpy#24603. Co-authored-by: Lars Grüter <[email protected]> Co-authored-by: Stefan van der Walt <[email protected]>

CI: switch Emscripten job to Meson

328a9a4

[skip ci]

rgommers added the component: CI label Aug 31, 2023

rgommers marked this pull request as draft August 31, 2023 15:13

github-actions bot added the 25 - WIP label Aug 31, 2023

rgommers commented Aug 31, 2023

View reviewed changes

rgommers added this to the 2.0.0 release milestone Sep 9, 2023

rgommers mentioned this pull request Sep 9, 2023

Update numpy to 1.25.2 pyodide/pyodide#4125

Merged

3 tasks

rth mentioned this pull request Oct 24, 2023

Bump numpy version to 1.26.1 and use meson backend pyodide/pyodide#4240

Merged

ryanking13 mentioned this pull request Nov 19, 2023

Add --config-setting flag to pyodide build command. pyodide/pyodide#4308

Merged

2 tasks

lagru added a commit to scikit-image/scikit-image that referenced this pull request Dec 11, 2023

Test emscripten workflow inspired by NumPy

da23139

Initial inspiration was taken from numpy/numpy#24603.

lagru mentioned this pull request Dec 11, 2023

Build package for Pyodide out of tree scikit-image/scikit-image#7265

Closed

rgommers modified the milestones: 2.0.0 release, 2.1.0 release Jan 31, 2024

rgommers commented Feb 23, 2024

View reviewed changes

agriyakhetarpal added a commit to agriyakhetarpal/numpy that referenced this pull request Feb 23, 2024

Enable run for Emscripten/Pyodide wheels

3b6f382

Copied with updates and suggestions received from numpy#24603 on 23/02/2024.

agriyakhetarpal added a commit to agriyakhetarpal/numpy that referenced this pull request Feb 23, 2024

Enable run for Emscripten/Pyodide wheels

2e3c066

Copied with updates and suggestions received from numpy#24603 on 23/02/2024.

agriyakhetarpal mentioned this pull request Feb 27, 2024

CI, TST: Re-enable Emscripten/Pyodide CI job for NumPy #25893

Closed

agriyakhetarpal mentioned this pull request Feb 27, 2024

CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy #25894

Merged

rgommers removed this from the 2.1.0 release milestone Feb 27, 2024

rgommers closed this Feb 27, 2024

agriyakhetarpal pushed a commit to agriyakhetarpal/scikit-image that referenced this pull request Mar 18, 2024

Test emscripten workflow inspired by NumPy

12a2aec

Initial inspiration was taken from numpy/numpy#24603.

lesteve mentioned this pull request Apr 16, 2024

CI Use Meson for Pyodide build scikit-learn/scikit-learn#28845

Closed

agriyakhetarpal mentioned this pull request May 17, 2024

ENH: Start building WASM wheels against newer Pyodide (version 0.26.0) #26467

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: CI: switch Emscripten job to Meson #24603

WIP: CI: switch Emscripten job to Meson #24603

rgommers commented Aug 31, 2023

rgommers Aug 31, 2023

rgommers Feb 23, 2024

hoodmane commented Aug 31, 2023

rgommers commented Sep 11, 2023

rth commented Sep 11, 2023

rgommers commented Sep 11, 2023

ryanking13 commented Sep 11, 2023

rgommers commented Sep 11, 2023

hoodmane commented Feb 9, 2024

rgommers commented Feb 9, 2024

ryanking13 commented Feb 11, 2024

rgommers commented Feb 11, 2024 •

edited

Loading

rgommers left a comment

rgommers Feb 23, 2024

rgommers Feb 23, 2024

rgommers Feb 23, 2024

agriyakhetarpal Feb 23, 2024

rgommers Feb 23, 2024

rgommers Feb 23, 2024

ryanking13 Feb 24, 2024

rgommers Feb 23, 2024

rgommers Feb 23, 2024

rgommers Feb 23, 2024

rgommers commented Feb 27, 2024

hoodmane commented Feb 28, 2024

WIP: CI: switch Emscripten job to Meson #24603

WIP: CI: switch Emscripten job to Meson #24603

Conversation

rgommers commented Aug 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hoodmane commented Aug 31, 2023

rgommers commented Sep 11, 2023

rth commented Sep 11, 2023

rgommers commented Sep 11, 2023

ryanking13 commented Sep 11, 2023

rgommers commented Sep 11, 2023

hoodmane commented Feb 9, 2024

rgommers commented Feb 9, 2024

ryanking13 commented Feb 11, 2024

rgommers commented Feb 11, 2024 • edited Loading

rgommers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgommers commented Feb 27, 2024

hoodmane commented Feb 28, 2024

rgommers commented Feb 11, 2024 •

edited

Loading