Pip-compile ignores setup_dependencies when parsing setup.py

[bgklein]

Similar to #625 it seems that pip-compile ignores setup_dependencies when resolving dependencies. This causes a problem for us as our workflow uses pip-compile to freeze dependencies w/ hashes. Then we use the requirements files generated to pip download all dependencies prior to installing in a network isolated environment.

Environment Versions

1. OS Type - Windows
2. Python version: Python 3.7.1
3. pip version: pip 18.1
4. pip-tools version: pip-compile, version 3.7.0

Steps to replicate

pip-compile --generate-hashes --allow-unsafe --output-file test-windows.txt test.in

Where test.in contains a dependency which has setup_dependencies.

Expected result

Setup dependency to be populated in windows.txt

Actual result

Not included.

[atugushev]

Hello @bgklein,

Thanks for the info! Could you paste here a content of the test.in which causes the bug?

[bgklein]

One example is flake8==3.50 (setup_dependency of pytest-runner).
test.in.txt
test-windows.txt

[atugushev]

I'm a little bit confused, what do you mean by setup_dependency?

[bgklein]

Sorry my bad. put the wrong terminology. setup_requires as in https://github.com/PyCQA/flake8/blob/8acf55e0f85233c51c291816d73d828cc62d30d1/setup.py#L146

[atugushev]

Thanks! Looks like that's a correct behaviour since packages in setup_requires are not supposed to be installed, see a note in docs:

(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the ./.eggs directory if they’re not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)

[bgklein]

Hmm that makes sense. Do you know if it would be possible to make this an optional flag for pip-compile? Otherwise my team will need to add them to our requirements file/maintain them separately as our actual install happens without internet connectivity causing 'They are simply downloaded to the ./.eggs directory if they’re not locally available already' to fail.

[atugushev]

You may simply add pytest-runner to the test.in. Wouldn't it be enough?

[bgklein]

Yea it would be enough just mean more maintenance. Since we do hash checking we specify the versions of our dependencies. This means when upgrading flake8 and/or any other dependencies we will need to dive into the dependency tree to upgrade their respective setup_requires.

[atugushev]

FYI there is no need to specify version in *.in file, that's not how it works. Let the pip-compile pin the packages in requirements.txt.

[bgklein]

While pip-compile does not require pinned versions, we do pin our versioning for our core dependencies. Looking over the setup_requires for most of our dependencies it looks like none of them pin their setup_requires to a specific version so we should be able to leave them unpinned, but it does defeat some of the purpose of pinning our dependencies.

You can close the issue if you'd like, as the solution should work. I still think it'd be helpful to have a flag which would automatically get and add setup_requires as currently we either have to try to build and see failures to determine what's missing or crawl the dependency tree to realize what is in the field, but I will leave teh decision on that to you guys.

[atugushev]

FYI, refs #492.