pex.resolver.Untranslateable exception in pex>=1.4.0

[pavdmyt]

I'm building PEX for my project with the following sh script:

#!/bin/bash

PEX_VERBOSE=5 \
    pex \
    -r requirements-pex.txt \
    -o "mycmd.pex" \
    -c mycmd \
    -i "http://my.devpi.index/mycmd/dev" \
    --python="python2.7" \
    --python-shebang="/usr/bin/env python" \
    --platform="macosx-10.10-x86_64" \
    --platform="linux-x86_64" \
    -v

Contents of requirements-pex.txt:

backports.ssl-match-hostname==3.5.0.1  # via docker
blindspin==2.0.1
botocore==1.8.19
certifi==2017.7.27.1      # via requests
chardet==3.0.4            # via requests
docker-pycreds==0.2.1     # via docker
docker==2.7.0
docopt==0.6.2
docutils==0.13.1          # via botocore
idna==2.5                 # via requests
ipaddress==1.0.18         # via docker
jinja2==2.10
jmespath==0.9.0           # via botocore
markupsafe==0.23          # via jinja2
psutil==5.4.2
pymongo==3.6.0
python-dateutil==2.6.0    # via botocore
pyvmomi==6.5.0.2017.5-1
pyyaml==3.12
requests==2.18.4
six==1.10.0               # via docker, docker-pycreds, python-dateutil, pyvmomi, websocket-client
termcolor2==0.0.3
termcolor==1.1.0          # via termcolor2
texttable==1.1.1
urllib3==1.22             # via requests
websocket-client==0.44.0  # via docker
wrapt==1.10.11
mycmd==1.2.3

Using pex 1.4.0 or pex 1.4.1 having the following issue (everything works fine with pex 1.3.2):

$ ./build_pex.sh
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x106060790>
/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for my.devpi.index has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
  backports.ssl-match-hostname 3.5.0.1
  termcolor2 0.0.3
  mycmd 1.2.3
  jmespath 0.9.0
  botocore 1.8.19
  docker-pycreds 0.2.1
  wrapt 1.10.11
  six 1.10.0
  termcolor 1.1.0
  docutils 0.13.1
  certifi 2017.7.27.1
  pymongo 3.6.0
  websocket-client 0.44.0
  blindspin 2.0.1
  urllib3 1.22
  MarkupSafe 0.23
  pyvmomi 6.5.0.2017.5.post1
  python-dateutil 2.6.0
  texttable 1.1.1
  ipaddress 1.0.18
  docker 2.7.0
  PyYAML 3.12
  psutil 5.4.2
  chardet 3.0.4
  Jinja2 2.10
  requests 2.18.4
  docopt 0.6.2
  idna 2.5
pex: Target package WheelPackage('file:///private/var/folders/cc/5p9wct3j6xqgzk0r_lhb68nxjhps7j/T/tmpyKWJej/wrapt-1.10.11-cp27-cp27m-macosx_10_10_x86_64.whl') is not compatible with [('cp27', 'cp27m', 'manylinux1_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'none', 'any'), ('cp2', 'none', 'any'), ('py27', 'none', 'any'), ('py2', 'none', 'any')]
Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/pex-build-May22/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/pex/bin/pex.py", line 694, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/pex/bin/pex.py", line 624, in build_pex
    for dist in resolveds:
  File "/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/pex/resolver.py", line 525, in resolve_multi
    use_manylinux=use_manylinux):
  File "/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/pex/resolver.py", line 451, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/pex/resolver.py", line 248, in resolve
    dist = self.build(package, resolvable.options)
  File "/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/pex/resolver.py", line 321, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/usr/local/var/pyenv/versions/2.7.14/envs/pex-build-May22/lib/python2.7/site-packages/pex/resolver.py", line 206, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///Users/me/.pex/build/wrapt-1.10.11.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

[kwlzn]

it looks like what's happening here is that pex is attempting to translate wrapt 1.10.11 from sdist -> wheel (via e.g. python setup.py bdist_wheel) but that ends up with a platform native wrapt-1.10.11-cp27-cp27m-macosx_10_10_x86_64.whl (presumably because you're doing that step on OSX?) which does not satisfy the linux platform requirement (i.e. cross compilation is not a thing here).

pex usually falls back to this mode only if no linux-compatible bdists are found in the resolver path. it does not look like wrapt provides platform specific wheels on pypi for this version: https://pypi.org/project/wrapt/1.10.10/#files

does your custom index provide a linux-compatible bdist for wrapt-1.10.11 to support this build?

if so, does this improve if you pass an extended platform of --platform="linux-x86_64-cp-27-mu"? this works for me against our internal binary store.

[kwlzn]

(and if so, it seems we could improve the behavior here by widening the ABI permutations generated for non-specific platform specifications. IIRC, right now that keys off of the currently executing interpreters ABI, which would typically be different from OSX (cp27m) -> Linux (cp27mu) - hence the reason explicit specification helps.)

[kwlzn]

pants does not yet consume pex 1.4.x, so there's no way these two particular issues could be related. but in general, https://github.com/pantsbuild/pants is the best place to file and work on pants related tickets.

[DrumMachineLearning]

Sorry about that; comment deleted.

Well, that explains why pants hates my manylinux wheel. Thanks for the heads-up

[kwlzn]

@DrumMachineLearning no worries. 1.4.x was just released and contains a pretty invasive change to the way bdist resolution works, so we're letting that bake/stabilize upstream (here) first - then should be pulling that into pants shortly (sometime in the 1-2 weeks).

[kwlzn]

@pavdmyt can you confirm that the above suggestion (passing --platform="linux-x86_64-cp-27-mu" explicitly) resolves this problem for you in pex 1.4.2? I can only speculate since your repro involves a private index, but would rather be sure before moving forward with a fix.

[pavdmyt]

@kwlzn sorry for delay, here is more info.

I was inspecting this issue at OSX, but initially it started failing at our "build server" (container based on alpine linux) where pex version was not pinned in requirements.txt 🤦‍♂️

At my custom index I have pre-built wheels for OSX and Ubuntu 16.04 (the intention is to run resulted pex application at both systems):

• wrapt-1.10.11-cp27-cp27m-macosx_10_10_x86_64.whl
• wrapt-1.10.11-cp27-cp27mu-linux_x86_64.whl

I've replaced --platform="linux-x86_64" with --platform="linux-x86_64-cp-27-mu" and it helped. Build was successful using pex==1.4.2 👍

Thanks!

[kwlzn]

@pavdmyt no worries! that's great - thanks for confirming.

I have a fix in mind to improve the ergonomics here so that it works as it did in 1.3.x for improved backward compatibility - will try to circle back to that within a week to cut 1.4.3.

until then, would use the extended platform form above as a workaround (or permanent change as that form will continue to work).

[kwlzn]

pex 1.4.3 is now cut, so the behavior here should be back to what you'd see with 1.3.x. let me know if you hit any further issues and I'd be happy to take a look.

[sitaktif]

FYI we had a similar issue and bumping the version from 1.4.1 to 1.4.3 fixed the problem for us. Thanks!