python: setuptools namespaces are broken (give collisions)

[domenkozar]

Since bf5d6fb, any combination of packages using setuptools namespaces are broken due to the fact we're not using eggs anymore. Since packages have the same path, they collide.

For example:

collision between `/nix/store/j52ga0vwgrjh607ql4kcs4mwnm4rkhjc-python2.7-logilab-astng-0.24.3/lib/python2.7/site-packages/logilab/__init__.py' and `/nix/store/dqsy6a2nv09j7hcv2a300q6392zbry0z-python2.7-logilab-common-0.61.0/lib/python2.7/site-packages/logilab/__init__.py' at /nix/store/9z6d76pz8rr7gci2n3igh5dqi7ac5xqj-builder.pl line 72.

[domenkozar]

This is a major drawback of current design. Eggs and wheels is how this is handled upstream. We should move forward with wheels, something that @chaoflow is working on.

[vcunat]

I'm just curious – any progress on this?

[domenkozar]

Nah, I still haven't found time to tackle this one. Maybe during my July vacations :-)

[domenkozar]

Nope, nope. Bumping the milestone :/

[domenkozar]

Another side-effect is multiple dependencies in tree that collide: 9815218

Ideally we'd want such tree to error out.

[domenkozar]

We have a similar problem like Haskell here: http://lists.science.uu.nl/pipermail/nix-dev/2015-October/018393.html

[domenkozar]

Wheels PR now reports errors if packages conflict, but setuptools namespaces support is still broken. To reproduce:

nix-shell -p pythonPackages.logilab-constraint -I /home/ielectric/dev --command "python -c 'import logilab.common'" --pure

[domenkozar]

I think this is actually an upstream issue. If two packages that have namespaces are in different PYTHONPATH, it won't work.

See

>>> logilab.__path__
['/nix/store/qsbzp85rx4dglkk5dk70i2ly7fcapcw6-python2.7-logilab-constraint-0.6.0/lib/python2.7/site-packages/logilab']

vs. virtualenv

>>> logilab.__path__
['/home/ielectric/dev/test/lib/python2.7/site-packages/logilab']

which clearly works because both packages are in the same folder.

[domenkozar]

Note that #5096 will solve the problem with collisions.

[jgeerds]

@domenkozar Is there an upstream bug report for it? (different PYTHONPATHs)

[domenkozar]

Nope, not yet. I haven't found enough free time to dig into details to report this.

[FRidh]

Namespaces do seem to work when using python.buildEnv.

$ nix-shell -E 'with import <nixpkgs> {}; with pkgs; (python.withPackages (ps: [ps.logilab-constraint])).env' --command "python -c 'import logilab.common; print(logilab.common)'" --pure                                                                    
<module 'logilab.common' from '/nix/store/pdypsn4vjpkmnqlx8dxcn5lvdfkll691-python-2.7.11-env/lib/python2.7/site-packages/logilab/common/__init__.pyc'> 

nix-shell always sets PYTHONPATH. Where is this implemented?

In #16672 I proposed using buildEnv more often and getting rid of PYTHONPATH when possible.

[domenkozar]

@FRidh pkgs/development/interpreters/python/2.7/setup-hook.sh

[gilligan]

I don't really have a clue about python stuff - @domenkozar is this still a problem ? Maybe @garbas could also help out with the status of this ;)

[FRidh]

Generally, namespace packages work fine. There is only one case, and we can't fix that really. If you make a Python 2 env with namespace packages python2.withPackages), then there will be a collision. The solution is to use python2.buildEnv and pass ignoreCollisions. This is an upstream issue and their solution is PEP 420, which is why it is solved with Python 3. Therefore, marking this as a wontfix.