Distributing multiversion packages in a pex file

[mzdaniel]

Thank you so much for creating this great project!

I am the author of the sphinxserve package that uses pex to significantly simplify distribution. Pex does an awesome job allowing to distribute pure python packages and binary ones for the same python minor versions (eg: 3.4.x) within a unique pex file. As Python 3 gets more adopted, very soon well have systems running 2.7, 3.4, 3.5 and 3.6, each of them potentially requiring different package name dependencies.

I am using the following to create pex files:
pex -v --disable-cache --no-index -f dist -c sphinxserve -o pex/scripts/sphinxserve dist/*

My attempts to create a unique pex containing different python version packages were unsuccessful. As soon as 3.4 binary wheels are added (onto dist containing 3.5 wheels), the following happens::

Could not satisfy all requirements for WheelPackage('file:///Volumes/data/prog/sphinxserve/dist/MarkupSafe-0.23-cp34-cp34m-macosx_10_6_intel.whl'):
WheelPackage('file:///Volumes/data/prog/sphinxserve/dist/MarkupSafe-0.23-cp34-cp34m-macosx_10_6_intel.whl')

Shouldn't this work?

[simeonf]

Daniel - good to bump into you!

Somebody more knowledgable may chime in here... But AFAIK I think you're trying to do something PEX hasn't intended to do - make a binary that targets multiple python runtimes. PEX files are designed to be run by a specific Python as demonstrated by the hashbang interpreter header added to the pex file itself. As with virtualenv you are making an environment with a specific Python version and it is for that python version.

You could in theory make a PEX runnable by multiple python versions if its dependencies were pure python with no version requirements - but this is never true of binary dependencies. Per https://docs.python.org/3/c-api/stable.html the ABI changes for every release of python so I'm guessing it sees having both cp34 and cp35 distributions as incompatible.

[lorencarvalho]

@mzdaniel @simeonf

Pex respects a PEX_PYTHON environment variable that you could exploit to accomplish your goals.

Example:

osx ~ ❯❯❯ pex requests
Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests
>>> ^D
osx ~ ❯❯❯ PEX_PYTHON=python3 pex requests
Python 3.5.1 (default, Dec 23 2015, 18:22:56)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests
>>>

This would require that your code and dependencies be modernized, which hopefully you are doing. If you are writing or depending on code that is not modernized, I suspect you'll need to create distinct pexes for each environment. When building them you can bake in which python interpreter to use via --python and/or --pex-shebang (the former is which python to use to build the pex, and the latter is the shebang line which is prepended to the zip file/pex file).

[mzdaniel]

Nice we are meeting here @simeonf! I should have guessed that :) Yes, pex is really nice and would like using it even more as sphinxserve is OS independent, thought it requires os dependent binary wheels (eg: gevent and friends). Maybe I just misinterpreted the end of the first pex README.rst paragraph?

Thank you @sixninetynine for your comment. Yes, this actually seems the way to run a multiversion pex if it were achievable. @simeonf comment makes me doubt now if it is possible at all.

[kwlzn]

currently the only way to build a "multi-platform" pex (e.g. with both osx and linux wheels) is via pants. the README conveniently leaves this bit out, but it is entirely possible to do this in pex itself - it just hasn't been implemented in the pex CLI yet.

there's an outstanding ticket for supporting both multi-platform and multi-interpreter pexes with some discussion here: #68

and there's a lingering PR to implement this out here: #201

which adds support for multiple rounds of resolution against both multi-interpreter and multi-platform. this is what is needed to make this work if anyone had time to pick this back up.

[kwlzn]

closing this one in favor of #68.

[kwlzn]

if anyone is interested in picking #201 back up and taking it over the finish line, looks like that's up for grabs.

[mzdaniel]

I'm for it :)
Comments on #68