pex.resolvable.InvalidRequirement: Unknown requirement type: .

[wickman]

It used to be possible to clone a python repository and do 'pex .' (in other words, check if path is an exploded source package) and build it. Not sure when this behavior changed -- breaks some packaging scripts I've been using for periscope.

[wickman]

Worth noting: FOO=$(basename $PWD); cd .. && pex $FOO still works fine.

[jsirois]

Things work under 2.6, but fail as described under 2.7 and 3.6 at least:

• 2.6

jsirois@gill ~ $ rm -rf /tmp/flask && git clone -q https://github.com/pallets/flask /tmp/flask && (cd /tmp/flask && PYTHONPATH=~/dev/pantsbuild/pex python2.6 -mpex.bin.pex .)
Python 2.6.9 (unknown, Mar 31 2016, 12:12:48) 
[GCC 5.3.0] on linux4
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import flask
>>> 

• 2.7

jsirois@gill ~ $ rm -rf /tmp/flask && git clone -q https://github.com/pallets/flask /tmp/flask && (cd /tmp/flask && PYTHONPATH=~/dev/pantsbuild/pex python2.7 -mpex.bin.pex .)
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/jsirois/dev/pantsbuild/pex/pex/bin/pex.py", line 599, in <module>
    main()
  File "/home/jsirois/dev/pantsbuild/pex/pex/bin/pex.py", line 578, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/jsirois/dev/pantsbuild/pex/pex/bin/pex.py", line 504, in build_pex
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/home/jsirois/dev/pantsbuild/pex/pex/resolvable.py", line 71, in get
    raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
pex.resolvable.InvalidRequirement: Unknown requirement type: .

• 3.6

jsirois@gill ~ $ rm -rf /tmp/flask && git clone -q https://github.com/pallets/flask /tmp/flask && (cd /tmp/flask && PYTHONPATH=~/dev/pantsbuild/pex python3.6 -mpex.bin.pex .)
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jsirois/dev/pantsbuild/pex/pex/bin/pex.py", line 599, in <module>
    main()
  File "/home/jsirois/dev/pantsbuild/pex/pex/bin/pex.py", line 578, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/jsirois/dev/pantsbuild/pex/pex/bin/pex.py", line 504, in build_pex
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/home/jsirois/dev/pantsbuild/pex/pex/bin/pex.py", line 504, in <listcomp>
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/home/jsirois/dev/pantsbuild/pex/pex/resolvable.py", line 71, in get
    raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
pex.resolvable.InvalidRequirement: Unknown requirement type: .

[kwlzn]

different results on my machine for 2.7:

2.7:

[illuminati pex (master)]$ rm -rf /tmp/flask && git clone -q https://github.com/pallets/flask /tmp/flask && (cd /tmp/flask && PYTHONPATH=~/dev/pex python2.7 -mpex.bin.pex .)
Python 2.7.11 (default, Jan  5 2016, 18:06:15) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import flask; flask.__file__
'/private/var/folders/3t/xkwqrkld4xxgklk2s4n41jb80000gn/T/tmp5UMKoF/.deps/Flask-0.13.dev0-py2.py3-none-any.whl/flask/__init__.py'
>>>

3.6

[illuminati pex (master)]$ rm -rf /tmp/flask && git clone -q https://github.com/pallets/flask /tmp/flask && (cd /tmp/flask && PYTHONPATH=~/dev/pex python3.6 -mpex.bin.pex .)
Traceback (most recent call last):
  File "/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/kwilson/dev/pex/pex/bin/pex.py", line 599, in <module>
    main()
  File "/Users/kwilson/dev/pex/pex/bin/pex.py", line 578, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/kwilson/dev/pex/pex/bin/pex.py", line 504, in build_pex
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/Users/kwilson/dev/pex/pex/bin/pex.py", line 504, in <listcomp>
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/Users/kwilson/dev/pex/pex/resolvable.py", line 71, in get
    raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
pex.resolvable.InvalidRequirement: Unknown requirement type: .

the failure appears related to the setuptools version:

[illuminati pex (master)]$ python2.7 -c 'import setuptools; print(setuptools.__version__)'
30.4.0
[illuminati pex (master)]$ python3.6 -c 'import setuptools; print(setuptools.__version__)'
34.2.0
[illuminati pex (master)]$ python3.6 -m pip install setuptools==30.4.0
Collecting setuptools==30.4.0
  Using cached setuptools-30.4.0-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 34.2.0
    Uninstalling setuptools-34.2.0:
      Successfully uninstalled setuptools-34.2.0
Successfully installed setuptools-30.4.0
[illuminati pex (master)]$ rm -rf /tmp/flask && git clone -q https://github.com/pallets/flask /tmp/flask && (cd /tmp/flask && PYTHONPATH=~/dev/pex python3.6 -mpex.bin.pex .)
Python 3.6.0 (default, Feb 13 2017, 11:02:15) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import flask; flask.__file__
'/private/var/folders/3t/xkwqrkld4xxgklk2s4n41jb80000gn/T/tmp9n4cxpb3/.deps/Flask-0.13.dev0-py2.py3-none-any.whl/flask/__init__.py'
>>> 

[kwlzn]

correlated the start of the issue with 34.x.x - further tracing reveals this relates to an inner setuptools capability miss due to incompatible installed vs required versions:

[illuminati pex (master)]$ (cd /tmp/flask && PYTHONPATH=~/dev/pex python3.6 -mpex.bin.pex .)
0
> /Users/kwilson/dev/pex/pex/installer.py(60)__init__()
-> if not self._interpreter.satisfies(self.capability) and strict:
(Pdb) s
--Call--
> /Users/kwilson/dev/pex/pex/installer.py(82)capability()
-> @property
(Pdb) r
--Return--
> /Users/kwilson/dev/pex/pex/installer.py(85)capability()->['setuptools>=20.0,<34.0']
-> return list(self.mixins().values())
(Pdb) r
> /Users/kwilson/dev/pex/pex/installer.py(61)__init__()
-> raise self.IncapableInterpreter('Interpreter %s not capable of running %s' % (
(Pdb) l
 56       self._installed = None
 57       self._strict = strict
 58       self._interpreter = interpreter or PythonInterpreter.get()
 59       import pdb; pdb.set_trace()
 60       if not self._interpreter.satisfies(self.capability) and strict:
 61  ->       raise self.IncapableInterpreter('Interpreter %s not capable of running %s' % (
 62             self._interpreter.binary, self.__class__.__name__))
 63   
 64     def mixins(self):
 65       """Return a map from import name to requirement to load into setup script prior to invocation.
 66   
(Pdb) self._interpreter.satisfies(self.capability); strict
False
True
(Pdb) self._interpreter
PythonInterpreter('/Users/kwilson/Python/CPython-3.6.0/bin/python3.6', PythonIdentity('CPython', 3, 6, 0), {('flask', '0.13.dev0'): '/private/tmp/flask', ('pex', '1.1.16'): '/Users/kwilson/dev/pex', ('wheel', '0.29.0'): '/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/site-packages', ('six', '1.10.0'): '/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/site-packages', ('setuptools', '34.0.1'): '/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/site-packages', ('pyparsing', '2.1.10'): '/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/site-packages', ('pip', '9.0.1'): '/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/site-packages', ('packaging', '16.8'): '/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/site-packages', ('appdirs', '1.4.0'): '/Users/kwilson/Python/CPython-3.6.0/lib/python3.6/site-packages'})
(Pdb) self.capability
['setuptools>=20.0,<34.0']

which causes a ResolvableDirectory.from_string() check to fail during translation - leading it to attempt conversion to other resolvable types and masking the root cause.

the reason it seems to work in the case of cd .. and passing the full dirname is because in the Resolvable enumeration path this will fall back to using the directory name as a literal ResolvableRequirement (e.g. if the dir name is flask it's roughly the same as doing ./pex flask without a ./flask dir). example:

[illuminati pex (master)]$ mv -f /tmp/flask /tmp/flask_source_code
[illuminati pex (master)]$ (cd /tmp && PYTHONPATH=~/dev/pex python3.6 -mpex.bin.pex flask_source_code)
Could not satisfy all requirements for flask_source_code:
    flask_source_code

ultimately, this scenario should bomb with a clearer indication that the setuptools version is incompatible vs appearing like the dist itself is invalid - or worse, falling back to using paths as literal requirement spec strings.

[habnabit]

Any suggestion on how to mitigate this? Just downgrade setuptools?

[kwlzn]

@habnabit yep, afaict a downgrade to setuptools<34.0 should repair this.

[jsirois]

This has since been fixed with the vendoring of setuptools in #624 as tracked by #607.