Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add supported platforms to the build workflow for GitHub Actions #409

Closed
kitao opened this issue Aug 28, 2022 · 64 comments
Closed

Add supported platforms to the build workflow for GitHub Actions #409

kitao opened this issue Aug 28, 2022 · 64 comments
Labels
help wanted Extra attention is needed

Comments

@kitao
Copy link
Owner

kitao commented Aug 28, 2022

For Pyxel 1.8.0, I've updated the build.yml to build Python wheels for each platform automatically:
https://github.com/kitao/pyxel/blob/main/.github/workflows/build.yml

But for now it only supports x86_64 Windows, x86_64 Linux, aarch64 Mac and x86_64 Mac.

I would like to support other platforms such as i686 Windows and aarch64 Linux, but that would require cross-compiling for Rust-SDL2 (references SDL2) and PyO3 (references Python) and I haven't been able to do that yet.

I look forward to any information or advice on how to do that.

@kitao
Copy link
Owner Author

kitao commented Aug 29, 2022

I released Pyxel 1.8.0. But I would like to add wheels for other platforms through the development of the 1.8 series.
I look forward to your information and support!

@messense
Copy link

messense commented Aug 29, 2022

The current x86_64 Linux wheel is not portable, it will only work on Ubuntu 20.04 and later. (In other words it requires glibc 2.31.)

You might want to check out the manylinux project to build more portable Linux wheels if you care to support Ubuntu 18.04 or other Linux distros with older glibc.

@kitao
Copy link
Owner Author

kitao commented Aug 29, 2022

@messense Thank you for your advice.
It seems that if I use manylinux project, I can also support aarch64.
Is there any good example uses it in GitHub Actions?

@messense
Copy link

Well, it's just Docker images, you can use them on GitHub Actions via the container option for x86_64: https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container

aarch64 requires QEMU so you'd need to pair https://github.com/docker/setup-qemu-action with https://github.com/addnab/docker-run-action, it's going to be very slow though.

@kitao
Copy link
Owner Author

kitao commented Aug 29, 2022

@messense Thank you for the information.
Actually I have an another question. If I use ubuntu-18.04 with Python3.7+ instead of ubuntu-latest (just like I did in the previous version of Pyxel), glibc-related issue will be addressed?

@merwok
Copy link
Contributor

merwok commented Aug 29, 2022

Here is the tool that many people use, and works in github actions: https://github.com/pypa/cibuildwheel

@kitao
Copy link
Owner Author

kitao commented Aug 29, 2022

@merwok I've already checked the action but I couldn't find the way to use Makefile which includes pre-process and post-process instead of normal setup.py build.
Is there any way to use make command with it?
And Rust-SDL2 also build SDL2 with CMake in the build process. It should also be taken care.

@merwok
Copy link
Contributor

merwok commented Aug 29, 2022

In that case, you would need to call cibuildwheel from the makefile, between your pre- and post-process steps.

@kitao
Copy link
Owner Author

kitao commented Aug 30, 2022

@messense Though I'm not familiar with zig, is it realistic to cross-compile with zig? If it can cover both SDL2 with CMake and Rust with Maturin, it is ideal.

@messense
Copy link

cmake-rs does not have good support for cross compiling right now: rust-lang/cmake-rs#158

@messense
Copy link

The good news is that with my cmake-rs patch, it compiles with maturin and zig, and runs fine:

$ maturin build --target aarch64-unknown-linux-gnu --release --zig -o dist
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
⚠️  Warning: skipped unavailable python interpreter 'python3.8' from pyenv
🐍 Not using a specific python interpreter
📡 Using build options manifest-path from pyproject.toml
   Compiling cmake v0.1.48 (https://github.com/messense/cmake-rs.git?branch=cross-compile#15ab68ba)
   Compiling pyo3-ffi v0.16.5
   Compiling pyo3 v0.16.5
   Compiling jpeg-decoder v0.1.22
   Compiling jpeg-decoder v0.2.6
   Compiling sysinfo v0.23.13
   Compiling sdl2-sys v0.35.2
   Compiling tiff v0.6.1
   Compiling tiff v0.7.3
   Compiling image v0.23.14
   Compiling image v0.24.3
   Compiling noise v0.7.0
   Compiling sdl2 v0.35.2
   Compiling pyxel-engine v1.8.0 (/Users/messense/Projects/pyxel/crates/pyxel-engine)
   Compiling pyxel-wrapper v1.8.0 (/Users/messense/Projects/pyxel/crates/pyxel-wrapper)
    Finished release [optimized] target(s) in 30.23s
📦 Built wheel for abi3 Python ≥ 3.7 to dist/pyxel-1.8.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

$ docker run --rm -it -v $(pwd):/io -w /io ubuntu:22.04
root@a49eb76a6a8d:/io# apt update && apt install -y python3-pip
root@a49eb76a6a8d:/io# pip3 install dist/pyxel-1.8.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Processing ./dist/pyxel-1.8.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Installing collected packages: pyxel
Successfully installed pyxel-1.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@a49eb76a6a8d:/io# python3
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyxel
>>> pyxel
<module 'pyxel' from '/usr/local/lib/python3.10/dist-packages/pyxel/__init__.py'>
>>> pyxel.pyxel_wrapper
<module 'pyxel.pyxel_wrapper' from '/usr/local/lib/python3.10/dist-packages/pyxel/pyxel_wrapper.abi3.so'>

The patch is

diff --git a/crates/pyxel-wrapper/Cargo.toml b/crates/pyxel-wrapper/Cargo.toml
index ee2eb477..b7766e6a 100644
--- a/crates/pyxel-wrapper/Cargo.toml
+++ b/crates/pyxel-wrapper/Cargo.toml
@@ -6,7 +6,7 @@ edition = "2021"
 description = "Python wrapper for Pyxel, a retro game engine for Python"
 repository = "https://github.com/kitao/pyxel"
 license = "MIT"
-readme = "../../docs/README-abspath.md"
+readme = "../../README.md"
 categories = ["game-engines", "graphics", "multimedia"]
 keywords = ["game", "gamedev", "python"]
 
@@ -21,5 +21,8 @@ pyxel-engine = { path = "../pyxel-engine", version = "1.8.0" }
 [target.'cfg(not(target_os = "emscripten"))'.dependencies]
 sysinfo = "0.23"
 
+[patch.crates-io]
+cmake = { git = "https://github.com/messense/cmake-rs.git", branch = "cross-compile" }
+
 [package.metadata.maturin]
 name = "pyxel.pyxel_wrapper"
diff --git a/pyproject.toml b/pyproject.toml
index 47c5ff79..756f3932 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -6,7 +6,7 @@ build-backend = "maturin"
 name = "pyxel"
 version = "1.8.0"
 description = "A retro game engine for Python"
-readme = "../../docs/README-abspath.md"
+readme = "README.md"
 requires-python = ">=3.7"
 license = { file = "../../LICENSE" }
 authors = [{ name = "Takashi Kitao", email = "[email protected]" }]

Had to patch readme location locally otherwise it does not build.

@kitao
Copy link
Owner Author

kitao commented Aug 30, 2022

@messense Great news! Thank you for making a patch for CMake.

There are two reasons why I use Makefile now. One is to make a README without relative link for PyPI according to the latest README (in pre-process), the other is to copy the built SDL2.dll from release folder to the python/pyxel folder (in post-process).
The former can be reduced by changing the time to make README-abspath. Regarding the latter, if there is a way to do it, I can use pure Maturin without Makefile and it means I can also use your maturin-action.

Do you know whether it is possible to copy some built artifact (e.g. SDL2.dll) while compiling a wheel with Maturin?

@messense
Copy link

Do you know whether it is possible to copy some built artifact (e.g. SDL2.dll) while compiling a wheel with Maturin?

No, it's not possible. I think It might be better to statically link SDL2 on Windows, then you don't need to copy anything.

@kitao
Copy link
Owner Author

kitao commented Aug 30, 2022

@messense
Yeah, it's what I'd really like to realize.
But it seems that it is impossible to link SDL2.lib to other dll statically on MSVC.
On the other hand, it seems that it is impossible to make ABI3 Python dll on MinGW.
If I misunderstood the situation, please let me know.

@messense
Copy link

messense commented Aug 30, 2022

But it seems that it is impossible to link SDL2.lib to other dll statically on MSVC.

I'm not sure about this, upstream rust-sdl2 has trouble building on Windows with bundled and static-link features, see https://github.com/Rust-SDL2/rust-sdl2/runs/7975840847?check_suite_focus=true

And this thread says it possible to link statically on Windows: http://forums.libsdl.org/viewtopic.php?p=39311#39311 , and this one: libsdl-org/SDL#846

@messense
Copy link

But it seems that it is impossible to link SDL2.lib to other dll statically on MSVC.

Fixed in Rust-SDL2/rust-sdl2#1252

@kitao
Copy link
Owner Author

kitao commented Aug 30, 2022

I am amazed at the speed of your actions.
So I can try it with your Rust-SDL2 fix-windows-static branch for now?

@messense
Copy link

So I can try it with your Rust-SDL2 fix-windows-static branch for now?

Yes please, note that maturin support cross compiling to Windows, so you can just use maturin build --target x86_64-pc-windows-msvc --release -o dist on macOS/Linux to test it out on your computer.

@kitao
Copy link
Owner Author

kitao commented Aug 31, 2022

@messense According to your advice, I'm modifying Pyxel's build.yml to build wheels with manylinux container.
For aarch64 platform with QEMU, is there any good way to get source code and install rustup? I'd like to use command if possible.

@messense
Copy link

Now that you've patched cmake-rs, you can just compile aarch64 linux wheels with --zig, install zig is easy: pip install ziglang

[patch.crates-io]
cmake = { git = "https://github.com/messense/cmake-rs.git", branch = "cross-compile" }
sdl2 = { git = "https://github.com/messense/rust-sdl2.git", branch = "fix-windows-static" }

Note that my patches to rust-sdl2 are merged in master, consider to change it to upstream master branch.

@kitao
Copy link
Owner Author

kitao commented Aug 31, 2022

Thank you, @messense .
I’ll change to use the latest rust-sdl2.
When I cross-compile with —zig option for aarch64, should I keep to use manylinux container? Sorry for many questions!

@messense
Copy link

When I cross-compile with —zig option for aarch64, should I keep to use manylinux container? Sorry for many questions!

No need to use manylinux container when using --zig because zig can target any glibc version, we set it to the version specified by --compatibility option which defaults to manylinux2014 for aarch64 Linux.

@kitao
Copy link
Owner Author

kitao commented Aug 31, 2022

In the latest build.yml of Pyxel, I'm using --zig option for cross-copilation, but 'bits/sys_errlist.h' file not found error occurs.
@messense Do you know that I'm missing?

@messense
Copy link

I'm not sure, I haven't tried compile it on Linux with zig yet, but it worked for me on macOS before.

@messense
Copy link

messense commented Sep 1, 2022

I don't have the time to investigate recently, meanwhile you can also try my messense/manylinux2014-cross:aarch64 docker image for cross compiling linux aarch64 wheels without --zig.

@kitao
Copy link
Owner Author

kitao commented Sep 1, 2022

@messense
Though --zig option is not used, thanks to your container, it seems that every build for seven platforms succeeded.
As soon as I have time, I will do an operation check test.

@kitao
Copy link
Owner Author

kitao commented Sep 1, 2022

It's not easy... I checked this artifacts built with the latest build.yml
pyxel-wheels.zip

Platform Result
x86_64 Windows Worked
i686 Windows Not checked yet
aarch64 macOS ImportError: dlopen(/Users/takashi/Downloads/venv/lib/python3.10/site-packages/pyxel/pyxel_wrapper.abi3.so, 0x0002): symbol not found in flat namespace (___isPlatformVersionAtLeast)
x86_64 macOS Worked
aarch64 Ubuntu Python can't find pyxel module after installing pyxel wheel. (No module named pyxel error)
x86_64 Ubuntu Not checked yet

It seems that cross-compiled version wheel is missing something.
And I'm wondering why aarch64 macOS version doesn't work because it worked in the previous version's build.yml.

@kitao
Copy link
Owner Author

kitao commented Sep 2, 2022

Thank you.
The version of maturin is updated on manylinux and now I can import Pyxel after installing the wheel.

Then, now I'm facing "No available video device error" while initializing SDL2.
I've seen this error before and I changed the rust-sdl2 setting for Linux to only unsafe-texture with installing libsdl2-dev at that time. (To be honest, I didn't know why it worked. Perhaps because cmake's setting was changed?)

Now I'm thinking about how to handle it for manylinux.

@messense
Copy link

messense commented Sep 2, 2022

Then, now I'm facing "No available video device error" while initializing SDL2.

Sounds like compiling from source did not enable X11 support?

@kitao
Copy link
Owner Author

kitao commented Sep 2, 2022

Ah yes, I think so.
I'm wondering there's some way to enable it with static build.

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

If SDL2 is built from source code, it seems that it depends on the current environment and dummy display driver is set to be used. So I removed bundled and static features from Linux environment and installed SDL2 package separately for dynamic link.
For installing SDL2 package, I just used sudo apt install -y libsdl2-dev but compilation failed because the compiler couldn't find libSDL2. I think it should be installed in a way that suits cross-compilation on messense/manylinux2014-cross.
Could you suggest proper way to install external packages for cross-compilation?

@messense
Copy link

messense commented Sep 3, 2022

The Docker image contains a working GCC cross compiler, so I think you could compile and install SDL2 to it's sysroot before building wheel.

There is a zlib build example: https://github.com/messense/manylinux-cross/blob/6ebafe2c05ea608e688dc5744b48ee594af35c2c/manylinux2014/aarch64/Dockerfile#L98-L108

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

According to this instructions, I arranged the following commands:

git clone https://github.com/libsdl-org/SDL
cd SDL
mkdir build
cd build
../configure
make
sudo make install

to

- name: Build and install SDL2
  run: |
    export CC=$TARGET_CC && \
    export AR=$TARGET_AR && \
    export RANLIB=$TARGET_RANLIB && \
    git clone --depth 1 https://github.com/libsdl-org/SDL.git && \
    cd SDL && \
    mkdir build && \
    cd build && \
    CFLAGS="-O3 -fPIC" ../configure --prefix=/usr/${{ matrix.target }}/ && \
    make -j4 && \
    sudo make -j4 install && \
    cd ../.. && \
    rm -rf SDL

In the case of x86_64-unknown-linux-gnu, it says ranlib command not found.

mkdir -p -- /usr/x86_64-unknown-linux-gnu/lib/cmake/SDL2
/usr/bin/install -c -m 644 sdl2-config.cmake /usr/x86_64-unknown-linux-gnu/lib/cmake/SDL2
/usr/bin/install -c -m 644 sdl2-config-version.cmake /usr/x86_64-unknown-linux-gnu/lib/cmake/SDL2
./libtool: line 1737: x86_64-unknown-linux-gnu-ranlib: command not found
make: *** [Makefile:189: install-lib] Error 127

Regarding aarch64-unknown-linux-gnu, configure fails due to cross compiling.

configure: error: in `/__w/pyxel/pyxel/SDL/build':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
checking whether we are cross compiling... 
Error: Process completed with exit code 77.

I'll continue to check the cross-compilation way for SDL2.

P.S.
It seems that I missed some parameters for cross compile.
https://forum.clockworkpi.com/t/tutorial-cross-compile-sdl2-c-applications-from-linux-to-clockworkos/3211

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

I specified --build and --host for cross compilation, then compilation progressed.
Now the same error occurs on both aarch64 and x86_64.

./libtool: line 1737: aarch64-unknown-linux-gnu-ranlib: command not found

@messense
Copy link

messense commented Sep 3, 2022

It's really odd, ranlib does exist:

root@bc3c11800dac:~/SDL# aarch64-unknown-linux-gnu-ranlib
Usage: aarch64-unknown-linux-gnu-ranlib [options] archive
 Generate an index to speed access to archives
 The options are:
  @<file>                      Read options from <file>
  --plugin <name>              Load the specified plugin
  -D                           Use zero for symbol map timestamp (default)
  -U                           Use an actual symbol map timestamp
  -t                           Update the archive's symbol map timestamp
  -h --help                    Print this help message
  -v --version                 Print version information
aarch64-unknown-linux-gnu-ranlib: supported targets: elf64-littleaarch64 elf64-bigaarch64 elf32-littleaarch64 elf32-bigaarch64 elf32-littlearm elf32-bigarm elf64-little elf64-big elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex

@messense
Copy link

messense commented Sep 3, 2022

Perhaps you can force it use absolute path: export RANLIB=$(which $TARGET_RANLIB)

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

Thank you.
I read this article: https://stackoverflow.com/questions/23078282/ranlib-not-found
And removed sudo, then it worked now.

But other error occurred while linking.

"/github/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-c21be34a5cae8449.rlib" "-Wl,-Bdynamic" "-lSDL2" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/github/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/__w/pyxel/pyxel/crates/pyxel-wrapper/target/x86_64-unknown-linux-gnu/release/deps/libpyxel_wrapper.so" "-Wl,--gc-sections" "-shared" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: /usr/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/4.8.5/../../../../x86_64-unknown-linux-gnu/bin/ld: cannot find -lSDL2

Library path should be modified...?

@messense
Copy link

messense commented Sep 3, 2022

You can try export RUSTFLAGS="-L/usr/${{ matrix.target }}/lib" and I think you might need to enable the static-link feature, otherwise it will try to link to libSDL2 dynamically.

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

If I use static feature, the SDL2 which uses dummy display device instead of X11 is linked and it doesn't work on actual Linux environment. (It is the same result with bundled and static features).

On the other hand, if I don't specify static feature, making wheel failed.
So I think I need to find out how to build SDL2 which can be adapted to users' video environment dynamically.

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

In Pyxel 1.7.2. I built Linux binary only with static feature of rust-sdl2 and it worked in cooperate with SDL2 installed by users.
I'm wondering why the current build process failed due to the lack of .so file. Maturin tries to include it?

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

I'll try SDL2 configure with --enable-x11-shared and --enabled-wayland-shared

@messense
Copy link

messense commented Sep 3, 2022

I'm wondering why the current build process failed due to the lack of .so file. Maturin tries to include it?

For manylinux compliance you need to bundle everything except a list of allowed shared libraries. You can use --skip-auditwheel to skip it.

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

Thanks to your advices. Finally cross-compiled aarch64 wheel worked on my aarch64 Linux VM!
I also found that the wheel name changed from manylinux to linux and I see the reason is against manylinux compliance.
In that case, is it more proper to use --manylinux off option instead of --skip-auditwheel?

@messense
Copy link

messense commented Sep 3, 2022

In that case, is it more proper to use --manylinux off option instead of --skip-auditwheel?

Actually you should use --manylinux 2014 along with --skip-auditwheel to pretend to be a manylinux wheel, otherwise linux wheel can not be published to PyPI.

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

I understood. I added --manylinux 2014 option as well. Thank you!

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

I've released Pyxel 1.8.1 includes new platforms. Thank you for your cooperation!

@kitao kitao closed this as completed Sep 3, 2022
Repository owner moved this from Issues to Done in @messense's open-source contributions Sep 3, 2022
@merwok
Copy link
Contributor

merwok commented Sep 3, 2022

along with --skip-auditwheel to pretend to be a manylinux wheel

It would be best to not publish non-complicant manylinux wheels!

IMO the goal should be to have proper manylinux wheels (with libs bundled), or a complete sdist with pyproject.toml that lets pip build from source.

@kitao
Copy link
Owner Author

kitao commented Sep 3, 2022

@messense
Actually I'm still investigating the difference between Pygame and Pyxel from the perspective of SDL2 support.
It seems that the difference is pre-installed packages in docker container.

Here is the Pygame's Dockerfile:
https://github.com/pygame/pygame/blob/main/buildconfig/manylinux-build/docker_base/Dockerfile-aarch64#L14-L22

Is it possible I can install same packages when running your container or I should make my own Dockerfile?
I tried sudo apt install -y libx11-dev-like things, but it seems that SDL2 couldn't find it while configuring.

I would appreciate it if you could give me the information. Thank you.

@kitao kitao reopened this Sep 3, 2022
Repository owner moved this from Done to In Progress in @messense's open-source contributions Sep 3, 2022
@kitao
Copy link
Owner Author

kitao commented Sep 4, 2022

I checked the SDL2's makefile and found that there's no easy way to make SDL2 recognize libX11 on cross-compile environment.
I close this issue. (I may try it again in the future...)

@kitao kitao closed this as completed Sep 4, 2022
Repository owner moved this from In Progress to Done in @messense's open-source contributions Sep 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants