Python on Nix claims to be manylinux compatible

[RonnyPfannschmidt]

Issue description

thanks to the rules in https://www.python.org/dev/peps/pep-0513/#platform-detection-for-installers
python on nix is considered to be manylinux compatible

in order to elevate the issue it is necessary to create a _manylinux python module that contains manylinux1_compatible = False

[FRidh]

That should be a matter of a simple patch to the interpreters adding that file.

[FRidh]

do you have a script I could run to test whether our Python is not anymore manylinux compatible?

[RonnyPfannschmidt]

@FRidh just typed this one up - it can be pulled into a single command

import _manylinux
assert not _manylinux.manylinux1_compatible

[FRidh]

I've pushed a fix to staging (63ec006). As soon as staging is merged into master the unstable channel will have the fix.

[exarkun]

Could someone explain why it's not manylinux compatible? It would be nice to be able to use manylinux1 wheels on Nix.

Is it because of https://www.python.org/dev/peps/pep-0513/#the-manylinux1-policy or something deeper?

[FRidh]

Good question. I am not sure anymore the exact reason (we should have documented that) but if I am correct the main issue is that wheels can depend on certain libraries which we can provide, but which will be at a different location. When we build from source we hardcode the path to these shared libraries. In the case of a compiled wheel, we would have to patch after installation or pass LD_LIBRARY_PATH.

[RonnyPfannschmidt]

nixos doesn't provide the requested libraries/symbol versions off-hand, also the more strict/save shared object location pinning requires exact library locations - so the only way to try and support manylinux1 would be a patched python that correctly sets up the linker, but at the same time that would weaken one of the benefits of nix/nixos - exact and consistent linkage

[exarkun]

Many manylinux1 binaries don't actually link to any external libraries. The declaration in Nix's _manylinux prevents even these from being installed. pip itself takes the stance that as long as libc is compatible it will go ahead and do the installation - and if additional libraries are linked against, it's the user's responsibility to deal with it. It would be a lot more convenient of Nix to take a stance like this.

[dpkp]

I think it makes little sense to disable manylinux1 compatibility in the name of trying to avoid inconsistent linking / external dependencies. The practical reality is that any package installed with pip -- whether manylinux1 or not -- could link to external libraries. If a user intends to use pip, they are already exiting the comfort of nix-managed packaging and should be treated as assuming those risks.

Nonetheless, for users looking for a workaround, the compatibility flag can be overridden manually within a virtualenv (though perhaps not with python3.5's native venv module):

python3.5 -m virtualenv venv
echo "manylinux1_compatible = True" > venv/lib/python3.5/_manylinux.py
source venv/bin/activate
python -c 'import _manylinux; print(_manylinux.manylinux1_compatible)'

[acowley]

Is there a way of overriding this restriction without using a virtualenv? I'd like to just take responsibility for fixing up the linkage of a downloaded wheel during a regular nix build.

EDIT: I figured this out for myself, sorry for the noise. I just copied the normal installPhase, but added a line to write a _manylinux.py file as suggested above, and tacked that on to PYTHONPATH before the pip install call.

[thomasjm]

What do you guys think of actually adding manylinux1 support to Nix-generated Python environments?

I created a derivation which gathers together all the required libraries here.

I would envision adding an option enableManylinux1 to mk-python-derivation.nix which adds this bundle of libraries to the LD_LIBRARY_PATH for the resulting Python binaries. (python.withPackages is a thin wrapper around this so you could easily supply the option when setting up a Python env.)

Even without further plumbing, you can use it like this:

myWithPackages = f: let packages = f pythonPackages; in
  python.buildEnv.override {
    extraLibs = packages;
    makeWrapperArgs = ["--set" "LD_LIBRARY_PATH" (makeLibraryPath [(callPackage ./manylinux1.nix {}).package])];
  };

If there's interest I can put together a PR.

[RonnyPfannschmidt]

at first glance this looks interesting, i'd suggest to open a issue and/or pr with what you have (but i currently also dont use nix for personal reasons