cross compilation issue with PyO3 for macOS from linux

[mockersf]

My goal is to cross compile a python library made with Py03 for macOS from linux.

My Setup

• python 3.9.0 built on macOS (ideally will be cross compiled later, but for now it works this way). I install it in a custom folder, and copy everything from this install to my linux (to folder /pythons/p3.9.0)
• linker and ar from https://github.com/tpoechtrager/osxcross, with setup in .cargo/config.toml (instructions on how to do it loosely adapted from https://wapl.es/rust/2019/02/17/rust-cross-compile-linux-to-macos.html)

cross building the library works, I can copy it manually to a macOS machine and use it:

$ PYO3_CROSS_INCLUDE_DIR="/pythons/p3.9.0/include" PYO3_CROSS_LIB_DIR="/pythons/p3.9.0/lib" cargo build --target x86_64-apple-darwin
    Finished dev [unoptimized + debuginfo] target(s) in 1.55s

My Issue

just replacing cargo by maturin doesn't work:

$ PYO3_CROSS_INCLUDE_DIR="/pythons/p3.9.0/include" PYO3_CROSS_LIB_DIR="/pythons/p3.9.0/lib" maturin build --target x86_64-apple-darwin
🔗 Found pyo3 bindings
💥 maturin failed
  Caused by: Finding python interpreters failed
  Caused by: Failed to get information from the python interpreter at python3.7
  Caused by: sys.platform in python, linux, and the rust target, Target { os: Macos, arch: X86_64 }, don't match ಠ_ಠ

it seems maturin is calling python to get the current architecture, which makes it impossible to cross compile as a macOS python won't reply the correct arch on linux as it just won't run:

PYO3_CROSS_INCLUDE_DIR="/pythons/p3.9.0/include" PYO3_CROSS_LIB_DIR="/pythons/p3.9.0/lib" maturin build --target x86_64-apple-darwin -i /pythons/p3.9.0/bin/python3
🔗 Found pyo3 bindings
💥 maturin failed
  Caused by: The given list of python interpreters is invalid
  Caused by: /pythons/p3.9.0/bin/python3 is not a valid python interpreter
  Caused by: Trying to get metadata from the python interpreter '/pythons/p3.9.0/bin/python3' failed
  Caused by: Exec format error (os error 8)

My Workaround

My first idea was to setup a false python that will reply whatever maturin is expecting. After some investigation, it seems maturin just want the result of https://github.com/PyO3/maturin/blob/master/src/get_interpreter_metadata.py. I ran this script on a macOS and created my custom python interpreter:

$ cat mypython
#!/bin/bash

echo '{"major": 3, "minor": 9, "abiflags": "", "interpreter": "cpython", "ext_suffix": ".cpython-39-darwin.so", "abi_tag": "39", "m": true, "u": false, "d": false, "platform": "darwin"}'

$ PYO3_CROSS_INCLUDE_DIR="/pythons/p3.9.0/include" PYO3_CROSS_LIB_DIR="/pythons/p3.9.0/lib" maturin build --target x86_64-apple-darwin -i /src/mypython
🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at /src/mypython
📦 Built source distribution to /src/target/wheels/mylibrary-0.1.0.tar.gz
   Compiling pyo3 v0.12.3
error: failed to run custom build command for `pyo3 v0.12.3`

Caused by:
  process didn't exit successfully: `/src/target/debug/build/pyo3-238519a86ddb11a6/build-script-build` (exit code: 1)
--- stderr
Error: "Py_ENABLE_SHARED is not defined"

💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit code: 101": `cargo rustc --message-format json --lib --target x86_64-apple-darwin -- -C link-arg=-undefined -C link-arg=dynamic_lookup`

It seems PyO3 doesn't build the same way when called from maturin. Some more investigation later, Py03 is now executing a python script to parse the _sysconfigdata of the target python: https://github.com/PyO3/pyo3/blob/master/build.rs#L236

Now, with my new and improved custom python, it works 🎉:

$  cat mypython
#!/bin/bash

if [[ $2 == import* ]]; then
	echo '{"major": 3, "minor": 9, "abiflags": "", "interpreter": "cpython", "ext_suffix": ".cpython-39-darwin.so", "abi_tag": "39", "m": true, "u": false, "d": false, "platform": "darwin"}'
else
	exec python3 "$@"
fi
$ PYO3_CROSS_INCLUDE_DIR="/pythons/p3.9.0/include" PYO3_CROSS_LIB_DIR="/pythons/p3.9.0/lib" maturin build --target x86_64-apple-darwin -I /src/mypython
🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at /src/mypython
📦 Built source distribution to /src/target/wheels/mylibrary-0.1.0.tar.gz
   Compiling pyo3 v0.12.3
   Compiling mylibrary v0.1.0 (/src/mylibrary)
    Finished dev [unoptimized + debuginfo] target(s) in 13.82s
📦 Built wheel for CPython 3.9 to /src/target/wheels/mylibrary-0.1.0-cp39-cp39-macosx_10_7_x86_64.whl

This mypython script, when called by maturin, will reply the expected architecture, and when called by PyO3 will use the local python3 to interpret the target python header.

Better Fix?

This is a very hacky fix, and not one I would want to use long term... From my understandings of PyO3 and maturin, I see three possible fixes:

• disable maturin arch verification, possibly behind a flag like --trust-me-dont-verify-my-arch
• check why PyO3 cross compilation doesn't work the same way when called from maturin
• use the same arch verification in PyO3 and maturin

Thank you for reading this far!

[konstin]

I think the best solution would be checking for the PYO3_CROSS environment variables and disabling the checks if those are present.

[ravenexp]

I've successfully cross-compiled a PyO3 "abi3" extension wheel for Windows from Linux by using the following python "interpreter" saved as wpython.sh

#!/bin/sh                                                                                                                                                                            

echo '{"major": 3, "minor": 9, "abiflags": null, "interpreter": "cpython", "ext_suffix": ".pyd", "abi_tag": null, "m": false, "u": false, "d": false, "platform": "windows", "base_prefix": "/usr/x86_64-w64-mingw32"}'

Maturin command line:

maturin build --target x86_64-pc-windows-gnu -i ./wpython.sh

When targeting abi3 on Windows, it seems like the only relevant piece of information the interpreter provides to maturin is "base_prefix", while the rest can be inferred from --target.

It would be nice if maturin could recognize PYO3_CROSS_LIB_DIR in place of "base_prefix", so that maturin build would work as seamlessly as cargo build when targeting Windows from Linux.

[konstin]

@ravenexp Could you please try latest master and check if it works? I tried to add support for abi3+windows+cross compiling in 9177092. This case seems to be easy enough to support, but I fear that the general cross compiling support is out of scope.

[ravenexp]

I've successfully compiled my extension with the latest master.

Everything seems to work, but this comment has caught my attention.

PyO3 defines it as follows

• PYO3_CROSS_LIB_DIR: This variable must be set to the directory containing the target's libpython DSO and the associated _sysconfigdata*.py file.

On my system it is /usr/x86_64-w64-mingw32/lib

But running Windows python in WINE yields:

>>> sys.base_prefix
'/usr/x86_64-w64-mingw32'

There is no /lib at the end, as I expected.
In fact I've got the JSON blob for my "python" shell script by running this python script in WINE:
https://github.com/PyO3/maturin/blob/master/src/get_interpreter_metadata.py

I believe 9177092 is not semantically correct in its "base_prefix" handling, yet it seems to work just fine.

[ravenexp]

Ok, I think I've figured it out.

Linking appeared to work because libpython3.dll.a was conveniently installed in /usr/x86_64-w64-mingw32/lib, which is in the default library search path of the cross compiler x86_64-w64-mingw32-gcc. The output of x86_64-w64-mingw32-gcc -print-search-dirs contains /usr/lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/.

I moved libpython3.dll.a to another location, and pointed PYO3_CROSS_LIB_DIR to it. Now cargo build works, but maturin build fails with:

💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit code: 101": `cargo rustc --message-format json --manifest-path Cargo.toml --lib --target x86_64-pc-windows-gnu -- -L native=/test/lib\libs`

As expected, \libs was incorrectly appended to the PyO3 cross library path. Also the path separator used is not valid for the build platform. Using PathBuf::push() instead of format! in

maturin/src/compile.rs

Line 70 in cbf3c93

format!("native={}\\libs", python_interpreter.base_prefix.display());

would solve this.

[konstin]

Thank you for investigating! I've changed it so that PYO3_CROSS_LIB_DIR is used like {base_prefix}/libs and I've changed the hardcoded backslash to a .join() (5dce1cb)

[ravenexp]

I've retested with the latest master. Both cargo build and maturin build work now for x86_64-pc-windows-gnu target.

Thank you for supporting this valuable edge case. Having to use WINE on the build system to run the Windows python executable to cross-compile a python module just didn't feel right to me. Now cross-compiling python extensions for Windows is as simple as cross-compiling pure Rust programs.

[tafia]

@ravenexp I'm very interested in cross compiling to windows too.

Would you mind posting the necessary steps to have a environment ready to cross compile (I will be using centos7 if this is relevant)? Something like a Dockerfile would be great!

I understand this is probably out of scope from maturin but I bet lot of people will find it useful.

[Agile86]

Here is my version:
Dockerfile

FROM rust

RUN apt-get update && \
	apt-get install -y mingw-w64 && \
	rustup target add x86_64-pc-windows-gnu

RUN apt install -y llvm-dev libclang-dev clang

RUN apt-get install -y python3 curl && \
	curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
	python3 get-pip.py && \
	rm get-pip.py

RUN pip install maturin

mypython:

#!/bin/bash

if [[ $2 == import* ]]; then
	echo '{"major": 3, "minor": 7, "abiflags": null, "interpreter": "cpython", "ext_suffix": ".pyd", "abi_tag": null, "m": false, "u": false, "d": false, "platform": "windows", "base_prefix": "/usr/x86_64-w64-mingw32"}'
else
	exec python3 "$@"
fi

main script:

#!/bin/bash

set -ex

IMAGE_NAME=python-mingw-cross-build

docker build --tag $IMAGE_NAME .

docker run $IMAGE_NAME /bin/bash -c '\
	 PYO3_CROSS_PYTHON_VERSION=3.7 \
	 PYO3_CROSS_LIB_DIR="$PWD/Python37/libs" \
	maturin build --target x86_64-pc-windows-gnu -i ./mypython --release'

• Python37/libs/python3.lib - abi3 in my case or put python37.lib
• In case of error - can't find python3 lib need to add build.rs with same path:

fn main() {
    println!(r"cargo:rustc-link-search=Python37/libs");
}

[ravenexp]

rustup toolchain install stable-x86_64-pc-windows-gnu

Why are you installing a Windows toolchain on linux? It can only be used in Windows or via WINE.
Adding the Windows target for the default linux toolchain is enough.

maturin build --target x86_64-pc-windows-gnu -i ./mypython --release

-i ./mypython hack is no longer needed with pyo3-13.1 and after 9177092 and 5dce1cb were added.

Non-abi3 extensions seem much more difficult to cross-compile because of the python interpreter ABI zoo.
I haven't tried to do that yet tbh.

I hope maturin-0.9.1 will be released in the nearest future solving the Windows cross-compilation case for good.

[tafia]

Thanks @Agile86 I've successfully managed to cross compile too!

On wsl, I've used a windows conda (/mnt/c/Users/<user>/.conda/envs/<name>/libs) and it worked like a charm!

[messense]

Cross compilation has been improved with #454 , for example

maturin/.github/workflows/test.yml

Lines 147 to 166 in 4b928f0

	test-cross-compile:
	name: Test Cross Compile
	runs-on: ubuntu-latest
	strategy:
	matrix:
	platform: [
	{ target: "aarch64-unknown-linux-gnu", arch: "aarch64" },
	{ target: "armv7-unknown-linux-gnueabihf", arch: "armv7" },
	]
	steps:
	- uses: actions/checkout@v2
	- name: Build Wheels
	run: |
	echo 'curl -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable --profile minimal
	source ~/.cargo/env
	rustup target add ${{ matrix.platform.target }}
	export PYO3_CROSS_LIB_DIR=/opt/python/cp36-cp36m/lib
	cargo run --target x86_64-unknown-linux-gnu -- build -i python3.9 --release --out dist --no-sdist --target ${{ matrix.platform.target }} -m test-crates/pyo3-mixed/Cargo.toml
	' > build-wheel.sh
	docker run --rm -v "$PWD":/io -w /io messense/manylinux2014-cross:${{ matrix.platform.arch }} bash build-wheel.sh

[ribeaud]

Any news here on cross-compiling to Mac OS X? To me, the workaround looks quite nasty and I am looking for a proper solution. Thanks.

[messense]

I've verified that cross compile for macOS from linux using osxcross works now.

[messense]

Any news here on cross-compiling to Mac OS X? To me, the workaround looks quite nasty and I am looking for a proper solution. Thanks.

#817 added support for cross compiling to macOS with the --zig option, it should work out of the box for abi3 wheels, but for non-abi3 wheels you still need to build or download macOS Python (from https://github.com/indygreg/python-build-standalone for example).

[ravenexp]

@ravenexp I'm very interested in cross compiling to windows too.

Would you mind posting the necessary steps to have a environment ready to cross compile (I will be using centos7 if this is relevant)? Something like a Dockerfile would be great!

I understand this is probably out of scope from maturin but I bet lot of people will find it useful.

I've finally got around to put my Windows cross-compilation hacks into a rust crate and publish it: https://github.com/ravenexp/python3-dll-a

It generates abi3-compatible Python DLL import libraries for the MinGW compiler without going through the pains of cross-compiling CPython itself using MinGW.

[messense]

@ravenexp That looks interesting, I wonder does it work with MSVC targets?

[ravenexp]

@ravenexp That looks interesting, I wonder does it work with MSVC targets?

No, MinGW and MSVC use different import library formats even though they both use MSCOFF object files.

AFAIK, MSVC does not have anything like MinGW dlltool, which allows the user to generate the import library without building the DLL itself at the same time.

This crate is a hack for cross-compiling Windows extension modules from Linux. When building on Windows, you can just use the host Python interpreter as usual.

[messense]

@ravenexp What about llvm-dlltool?

[ravenexp]

@ravenexp What about llvm-dlltool?

That looks feasible, but is llvm-dlltool installed by default for the x86_64-pc-windows-msvc rustc target?
AFAIK there is a push to move to lld in place of MSVC's link.exe for Windows, but it's not stable yet rust-lang/rust#71520

Also, there is rust-lang/rust#58713, which will probably make the whole import library thing obsolete.

[messense]

but is llvm-dlltool installed by default for the x86_64-pc-windows-msvc rustc target?

No, it's not installed by default.

AFAIK there is a push to move to lld in place of MSVC's link.exe for Windows

We can enable lld for Windows cross compiling in maturin if we want, but it'd only work for pure Rust projects.

Also, there is rust-lang/rust#58713, which will probably make the whole import library thing obsolete.

Great! Will it work for both Windows msvc and gnu targets?

[ravenexp]

I'll try to add basic llvm-dlltool support to python3-all-a and see how it goes. I've never used x86_64-pc-windows-msvc toolchain before, so it'll probably take me some time to test this configuration.

Also, there is rust-lang/rust#58713, which will probably make the whole import library thing obsolete.

Great! Will it work for both Windows msvc and gnu targets?

Yes, it will work for all Windows targets and will allow importing DLL symbols by name without any import libraries. PyO3 has to be modified to support this feature, though.

[ravenexp]

I've added llvm-dlltool support to the latest python3-dll-a and everything seems to work at least when rust-lld is used as the linker. I'll try linking with link.exe as well, but there's nothing I can do in case that fails.

python3-dll-a now covers x86_64-pc-windows-gnu, i686-pc-windows-gnu, x86_64-pc-windows-msvc, i686-pc-windows-msvc and aarch64-pc-windows-msvc compile targets and should be (hopefully) production ready.
I'll write up a PyO3 issue and if that goes well I'll try to integrate it into pyo3-build-config.

@messense Thank you for suggesting llvm-dlltool! It really helped to turn my one day project into something usable by others.