Release more flexible pex binaries.

[jsirois]

Currently we release pex27 and pex37. Since pex is dependency free and supports both python 2.7 and python 3.4-3.7, we would ideally release one binary that could run under all these interpreters.

Typical systems will have on the PATH:

1. python - the default python
2. python2 - the default python 2.*
3. python3 - the default python 3.*
4. python3.7, python2.7, ... - the ~specific pythons

The current release process creates binaries with shebangs from the fourth category which has the upshot of not providing a binary for pythons 3.4, 3.5 or 3.6 even though both the pex27 and pex37 could otherwise run under these pythons.

We should release binaries to support 3.4, 3.5 and 3.6 either by releasing one binary that uses a shebang with python and interpreter constraints to ensure that python falls in the acceptable ranges or else by releasing two binaries as we do today but with more flexible python2 and python3 shebangs (really only needed for python3 since python2.7 is actually all we support in the 2 series). The third alternative of releasing one binary per minor supported version seems wasteful.

[jsirois]

Urgh, --interpreter-constraint is buggy in a few ways foiling ideas one and two.

For idea one, passing multiple --interpreter-constraint does not OR, it ANDS - which the , operator already does and so is not useful. If instead multiple --interpreter-constraint OR'd then we could say --interpreter-constraint='>=2.7,<3' --interpreter-constraint='>=3.4,<4', which combined with a #!/usr/bin/env python shebang would allow for a safe universal pex. See here and note all calls pass meet_all_constraints=True:
https://github.com/pantsbuild/pex/blob/a16f0ecb21ce3526f7d243778507d68219c3a91c/pex/interpreter_constraints.py#L24-L26

For idea one and two, --interpreter-constraint only takes effect if PEX_PYTHON or PEX_PYTHON_PATH is set:
https://github.com/pantsbuild/pex/blob/a16f0ecb21ce3526f7d243778507d68219c3a91c/pex/pex_bootstrapper.py#L141-L150

[jsirois]

I've gone ahead and brute-forced for the 1.6.1 release by including pex{27,3{4,5,6,7}}: https://github.com/pantsbuild/pex/releases/tag/v1.6.1

I'll leave this issue open until I break out --interpreter-constraint issues to track the prerequisites for doing all this the right way.

[jsirois]

@CMLivingston if you can comment on this in the form of either - ack, sounds unintended or, to the contrary, that there was intent behind these behaviors, I'd be grateful.

[CMLivingston]

This was intended. The confusion and issues we are seeing that stem from OR vs AND are largely due to my ignorance of pex use cases (I was relatively new to the codebase when I made these changes), and I was having difficulty trying to understand why anyone would OR at all within the context of the change I was making at the time, which was to move from a single PEX_PYTHON (use this interpreter) -> PEX_PYTHON_PATH (use one of these based on constraints of the binary). That was the use case I was targeting with meet_all_constraints=true - to target a specific interpreter on PPP, assuming that the use case there is to have control over the exact interpreter you need (iirc there was some issues with the , AND once you started getting complicated and went beyond simple ones like CPython>=2.7,<3, but I am not completely sure).

Long story short, it was intended for the use case and I was confused at the time / thought that ORing constraints was a bug, so I added that option in to be more explicit and give finer grain control. Ultimately, I think the custom nature of PPP and PP is different from what you are describing here because it is much more pointed, custom setting regarding which interpreter to use vs. giving the freedom and flexibility to encompass a wide range (this being largely because it grew out of the singular PEX_PYTHON). By design, PPP-based selection will not fall back to $PATH, and will error out if the AND is not satisfied. Admittedly, this was heavily influenced by monorepo philosophy / pants, where we were trying to go from one canonical interpreter to two, feature-incompatible canonical interpreters.

I think one possible solution here would be to fragment the call to matched_interpreters based on this condition https://github.com/pantsbuild/pex/blob/a16f0ecb21ce3526f7d243778507d68219c3a91c/pex/pex_bootstrapper.py#L80 and use meet_all_constraints=False, reserving the AND for PPP/PP only. I need to ramp up on other places this bites us (so there may be a case for never ANDing), but given my current knowledge level I think it would make sense to fragment.

[jsirois]

Thanks @CMLivingston - that helps me move forward without breaking Twitter things.

[jsirois]

Noting that full support for multi-python pex binaries will also require fixing #415.

[stuhood]

Hey @jsirois ! Is there any separable work that could be broken out of this that we (Twitter) could prioritize in order to help get this fixed?

[jsirois]

Yes - all the linked issues here that are not labelled in-progress, so:

• Multi-interpreter feature doesn't work perfectly #415 - which is tightened up in When using resolve_multi, not all combinations of interpreters and platforms work. #658
• The --interpreter-constraint option should OR #655
• If an --interpreter-constraint is set, it should always be honored. #656

If anyone does pitch in, please assign the issue to yourself and label in-progress.

[stuhood]

@jsirois : Ok, thank you! Is there a best choice issue for someone to start with? If you add a few more details to "the most separable" issue, I can commit one of us to finishing it before the end of next week.

[jsirois]

#656 seems teed up already to me. I'm happy to answer questions if that somehow is not explicit enough.

[jsirois]

And fwiw - @CMLivingston should be intimately familiar with #655 and #656 both.