Recognize Python packages (wheels, eggs, various dist as archives or installed)

[balusarakesh]

scancode must be able to detect Python packages used by in a project by parsing files like setup.py, .whl files, .eggs files, requirements.txt or pip_requirements.txt.

[sschuberth]

Just for the record, while PR #421 is thankfully merged, support for recognizing Python packages based on *requirements*.txt files is still missing.

[pombredanne]

@sschuberth indeed. There are several limitation in the current code.

1. support for requirements is one: https://github.com/landscapeio/requirements-detector may be what the doctor ordered, may be together with pip proper used as a library and also many of the https://pypi.python.org/pypi?%3Aaction=search&submit=search&term=requirements

2. support for parsing setup.py is somewhat weak too. These could be some pointers:

• https://github.com/msabramo/flake8-exact-pin/blob/master/flake8_exact_pin.py
• https://gitlab.com/abre/setup-freeze/blob/master/setup_freeze.py
• https://github.com/msabramo/setuppycheck/blob/master/setuppycheck.py
• https://github.com/inveniosoftware/requirements-builder/blob/master/requirements_builder/requirements_builder.py
• https://github.com/specialunderwear/setupreader/blob/master/setupreader.py
• https://github.com/pombredanne/setupreader
• https://github.com/ashwoods/python-packr/blob/master/setup_parser.py
• https://github.com/openembedded/openembedded-core/blob/f3e753372baac43d0921186340cf260df056de20/scripts/lib/recipetool/create_buildsys_python.py#L181
• https://github.com/yosi-dediashvili/SubiT/blob/ee154c43c3c0170e82750d428688b4d441f95134/build/setup.py
• https://github.com/mk-fg/fgtk/blob/master/dev/distribute_regen
• https://github.com/noisy/parse_setup.py
• http://stackoverflow.com/questions/24236266/how-to-extract-dependencies-information-from-a-setup-py
• http://stackoverflow.com/questions/27790297/parse-setup-py-without-setuptools
• https://github.com/andymckay/baked/blob/master/baked.py
• https://github.com/mgood/quipu/blob/master/quipu.py
• https://github.com/newville/asteval

3. Parsing actual packages as installed could likely benefit from code/functions existing in distutils, pip and setuptools

[pombredanne]

The basic is there. Addintional updates will be for post 2.0

[pombredanne]

Our hand rolled code for Python packages needs beefing. We should use instead:

1. https://github.com/pombredanne/pkginfo enhanced for pip-installed wheels
2. The requirements parser from pyup by @jayfk https://github.com/pyupio/pyup/blob/master/pyup/requirements.py

And possibly yet more candidates:

• Setup.py dynamic execution in docker:

  • https://github.com/noisy/parse_setup.py (but this needs docker)

• Setup.py through AST:

  • https://github.com/amitsaha/parse_setup_py by @amitsaha though I need to ask about the license

Setup.py through mocking:

• https://github.com/inveniosoftware/requirements-builder/blob/master/requirements_builder/requirements_builder.py by the buddies from invenio BSD-licensed

• https://github.com/specialunderwear/setupreader by @specialunderwear GPL-licensed

Using pip to parse requirements and some other pointers :

• https://github.com/narfman0/parse-requirements-not-suckily
• https://github.com/collective/collective.recipe.pip
• https://github.com/witsch/buildout.packagename/blob/master/buildout/packagename/__init__.py
• https://github.com/msabramo/setuppycheck/blob/master/setuppycheck.py
• https://gitlab.com/abre/setup-freeze/blob/master/setup_freeze.py
• https://github.com/reinout/z3c.dependencychecker
• https://github.com/JulienPalard/pystyle

[jayfk]

Okay, this is now useable for requirements.txt, tox.ini and conda.yml (PyPi dependencies only).

I'm planning to add read-only setup.py support soon.

[pombredanne]

@jayfk Thanks++
So I guess my approach will be to:

1. for conda and tox files invoke with the proper filetypes. Check for exceptions and is_valid
2. for req-like looking text files invoke with the proper filetypes. Check for exceptions and is_valid too. Since there are possibly more filename variety there, I will fish with a wide net and balk if this does not parse and only return errors when this is really a requirements*.txt files. I guess something similar to this: https://github.com/pyupio/pyup/blob/e6d2984efb42be8d195ac1f2c7e6a27505596fdc/pyup/bot.py#L486

[licodeli]

Please refer to this: https://pythonhosted.org/pkginfo/distributions.html#introspecting-unpacked-source-distributions

Our goal is to report first installed Python packages, this is likely a job for pkg_info instead of dparse

[licodeli]

This can be another tool: https://pypi.org/project/wheel-inspect/

[pombredanne]

This is all done now and merged in develop.