Missing arm build for linux

[thomasaarholt]

Are you using Python or Rust?

Python

What version of polars are you using?

latest, 0.13.3

What operating system are you using polars on?

Debian 10

Describe your bug.

I'm trying to install polars in a python docker container, where rust/cargo is not installed.

We use these docker images regularly at work. During pip install polars, it raises the following error, that it requires rust. Is this correct?
From the github readme, I do not get the impression that rust is a dependency, though I understand it is built using rust. My colleague was very surprised when I told him, and suggested I post here.

Error:

 > [3/3] RUN pip install polars==0.13.3:
#5 1.787 Collecting polars==0.13.3
#5 1.909   Downloading polars-0.13.3.tar.gz (729 kB)
#5 2.543   Installing build dependencies: started
#5 19.12   Installing build dependencies: finished with status 'done'
#5 19.12   Getting requirements to build wheel: started
#5 19.16   Getting requirements to build wheel: finished with status 'done'
#5 19.16     Preparing wheel metadata: started
#5 19.19     Preparing wheel metadata: finished with status 'error'
#5 19.19     ERROR: Command errored out with exit status 1:
#5 19.19      command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_jualw5a
#5 19.19          cwd: /tmp/pip-install-mpfc2sos/polars_a0e8fca0fa6e478fa5565d66ba22b23f
#5 19.19     Complete output (6 lines):
#5 19.19
#5 19.19     Cargo, the Rust package manager, is not installed or is not on PATH.
#5 19.19     This package requires Rust and Cargo to compile extensions. Install it through
#5 19.19     the system's package manager or via https://rustup.rs/
#5 19.19
#5 19.19     Checking for Rust toolchain....

What are the steps to reproduce the behavior?

Save the following file as Dockerfile in a directory on your computer.

FROM python:3.9-slim-buster
WORKDIR /usr/src/app
RUN pip install polars==0.13.3

From the above directory, call docker build -t test-polars .

What is the actual behavior?

Error, as above

What is the expected behavior?

Either it should be made clear that rust is a dependency, or it should install without error :)

[ritchie46]

There are prebuilt binaries. Also for linux maybe your timing was unlycky and the binary was not pushed yet but the source was?

[thomasaarholt]

So, I tried removing the version specification in the Dockerfile (removing ==0.13.3), and when 0.13.3 fails to install, pip continues "down the list" of versions. I stopped it after 0.10.0 - it's producing the same error message each time.

I've tried pulling the regular FROM python:3.9 image as well, with no change.

I can't check on my personal mac to see what would happen there, since I have rust installed.

I do observe that when I pip install polars, it downloads a 700 KB file, not a larger binary.

[ritchie46]

Are you doing this on a Mac? @ghuls didn't we have this discussion on discord last week?

[ghuls]

I assume pip is too old and does not support binary wheels or does not use them by default.

pip install -U --only-binary polars

[ritchie46]

I assume pip is too old and does not support binary wheels or does not use them by default.

pip install - U --only-binary polars

I am going to add this one to the README!

[thomasaarholt]

I think I realised the issue: I'm doing this on an arm mac. There are only prebuilt binaries for x86.
Could you please add a build target for arm?

Edit:
I'm not particularly experienced at building python, but I thought I'd add a few pointers:
We want this rust target: aarch64-unknown-linux-gnu. Unfortunately, manylinux support for arm is not great. It seems like its doable, but slow.

You may be better off recommending me to build it myself by first installing rust.

[thomasaarholt]

Just to backup my request with reasons:
A lot of ML-businesses run linux docker images to make their development as platform-agnostic as possible (ironically, complicating things in this situation). Hence I'm running arm-linux on arm-mac. One could alternatively run x86-linux emulated on arm-mac via rosetta (apple's emulation software), but it's much slower. And I'm certain you will get this request again :)

[thomasaarholt]

And for reference, the image is running pip 21.2.4, which is definitely new enough :) (just to be certain, I did try updating pip, and got the same result)

I assume pip is too old and does not support binary wheels or does not use them by default.

pip install -U --only-binary polars

I think you want this instead :) The above one raises an error.

pip install -U polars --only-binary polars

[thomasaarholt]

This github action lets you run on arm (aarch74) via emulation. Might be helpful for building.
https://github.com/uraimo/run-on-arch-action

[ritchie46]

I have found the discussion I meant earlier. As I understand it you run docker on a M1 chip. And we have precompiled manylinux binaries for intel/amd chips. You can tell docker to emulate those:

https://forums.docker.com/t/run-x86-intel-and-arm-based-images-on-apple-silicon-m1-macs/117123

Now I am not really known of all target types. Do and can we precompile manylinux for arm chips?

[ritchie46]

For mac we have already target m1 and intel/amd chips. So this would need adding manylinux2014_aarch64 to our linux builds. Any help on this would be greatly appreciated. The current manylinux build is here: https://github.com/pola-rs/polars/blob/master/.github/workflows/create-py-release-manylinux.yaml

[ghuls]

And for reference, the image is running pip 21.2.4, which is definitely new enough :) (just to be certain, I did try updating pip, and got the same result)

I assume pip is too old and does not support binary wheels or does not use them by default.

pip install -U --only-binary polars

I think you want this instead :) The above one raises an error.

pip install -U polars --only-binary polars

I posted from mobile, so couldn't really test. But indeed that pip version is more than recent enough. It probably would be a problem for the previous version of debian.

This also exists:
https://cibuildwheel.readthedocs.io/en/stable/faq/#apple-silicon

Cross compiling with maturin does not seem to work yet (using zig):
PyO3/maturin#375

[ritchie46]

I believe it should work with: https://github.com/messense/maturin-action

[ritchie46]

Here it is:

https://files.pythonhosted.org/packages/be/89/d204a3870fc97ade655404da3eea15e8a68c1897b7b0d919c27cd02b8701/polars-0.13.4-cp36-abi3-manylinux_2_24_aarch64.whl

From now on they will be included in every release.

[thomasaarholt]

Wow! That was fast! Thank you!!

[messense]

Cross compiling with maturin does not seem to work yet (using zig):

@ghuls It does now! Implemented in PyO3/maturin#817.

[ghuls]

@messense So it will work in the next release of maturin?

[messense]

Yes, it's already released in a pre-release version so you can try it now: pip3 install -U --pre maturin.

[ghuls]

It does not seeem to work by default:

maturin build -m py-polars/Cargo.toml --target aarch64-apple-darwin --no-sdist -o wheels -i python --zig

As it is missing some MacOS header files. Seems like zig fetch headers does not have them in:
https://github.com/ziglang/fetch-them-macos-headers/tree/main/headers/aarch64-macos.12-gnu/CommonCrypto

[messense]

What header files are missing? Maybe we should report the issue to zig?

[ghuls]

At least this one:

warning: In file included from c_src/mimalloc/src/static.c:24:
warning: c_src/mimalloc/src/random.c:198:10: fatal error: 'CommonCrypto/CommonRandom.h' file not found
warning: #include <CommonCrypto/CommonRandom.h>
warning:          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: 1 error generated.

error: build failed
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `cargo rustc --message-format json --manifest-path py-polars/Cargo.toml --target aarch64-apple-darwin --lib -- -C link-arg=-undefined -C link-arg=dynamic_lookup -C link-args=-Wl,-install_name,@rpath/polars.cpython-39-x86_64-linux-gnu.so`