Second-level requirements for editable reqs disappear after round 1

[voidlily]

In 1.8.0, running pip-compile would include requirements from editable reqs in the output file.

This is a possible regression introduced by 444fba5, when the multi-round requirement calculation was introduced.

Environment Versions

1. OS Type - linux
2. Python version: 3.6.0
3. pip version: 9.0.1
4. pip-tools version: 1.8.1rc3

Steps to replicate

requirements.in:

-e git+git://github.com/pyca/cryptography@master#egg=cryptography

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
-e git+git://github.com/pyca/cryptography@master#egg=cryptography
appdirs==1.4.3            # via setuptools
asn1crypto==0.21.1
cffi==1.9.1
idna==2.5
packaging==16.8           # via setuptools
pycparser==2.17           # via cffi
pyparsing==2.2.0          # via packaging
six==1.10.0               # via packaging, setuptools

# The following packages are considered to be unsafe in a requirements file:
# setuptools

(Note that this includes setuptools reqs as in #445, but unrelated to the issue at hand )

Actual result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
-e git+git://github.com/pyca/cryptography@master#egg=cryptography

Debug output

Using indexes:
  https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  cryptography from git+git://github.com/pyca/cryptography@master#egg=cryptography (from -r requirements.in (line 1))

Finding the best candidates:
  found candidate -e git+git://github.com/pyca/cryptography@master#egg=cryptography (constraint was <any>)

Finding secondary dependencies:

New dependencies found in this round:
  adding ['asn1crypto', '>=0.21.0', '[]']
  adding ['cffi', '>=1.4.1', '[]']
  adding ['idna', '>=2.1', '[]']
  adding ['packaging', '', '[]']
  adding ['setuptools', '>=11.3', '[]']
  adding ['six', '>=1.4.1', '[]']
Removed dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  asn1crypto>=0.21.0
  cffi>=1.4.1
  cryptography from git+git://github.com/pyca/cryptography@master#egg=cryptography (from -r requirements.in (line 1))
  idna>=2.1
  packaging
  setuptools>=11.3
  six>=1.4.1

Finding the best candidates:
  found candidate asn1crypto==0.21.1 (constraint was >=0.21.0)
  found candidate cffi==1.9.1 (constraint was >=1.4.1)
  found candidate -e git+git://github.com/pyca/cryptography@master#egg=cryptography (constraint was <any>)
  found candidate idna==2.5 (constraint was >=2.1)
  found candidate packaging==16.8 (constraint was <any>)
  found candidate setuptools==34.3.1 (constraint was >=11.3)
  found candidate six==1.10.0 (constraint was >=1.4.1)

Finding secondary dependencies:
  packaging==16.8           requires pyparsing, six
  six==1.10.0               requires -
  asn1crypto==0.21.1        requires -
  cffi==1.9.1               requires pycparser
  idna==2.5                 requires -
  setuptools==34.3.1        requires appdirs>=1.4.0, packaging>=16.8, six>=1.6.0

New dependencies found in this round:
  adding ['appdirs', '>=1.4.0', '[]']
  adding ['packaging', '>=16.8', '[]']
  adding ['pycparser', '', '[]']
  adding ['pyparsing', '', '[]']
  adding ['six', '>=1.6.0', '[]']
Removed dependencies in this round:
  removing ['asn1crypto', '>=0.21.0', '[]']
  removing ['cffi', '>=1.4.1', '[]']
  removing ['idna', '>=2.1', '[]']
  removing ['packaging', '', '[]']
  removing ['setuptools', '>=11.3', '[]']
  removing ['six', '>=1.4.1', '[]']
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3                           
Current constraints:
  appdirs>=1.4.0
  cryptography from git+git://github.com/pyca/cryptography@master#egg=cryptography (from -r requirements.in (line 1))
  packaging>=16.8
  pycparser
  pyparsing
  six>=1.6.0

Finding the best candidates:
  found candidate appdirs==1.4.3 (constraint was >=1.4.0)
  found candidate -e git+git://github.com/pyca/cryptography@master#egg=cryptography (constraint was <any>)
  found candidate packaging==16.8 (constraint was >=16.8)
  found candidate pycparser==2.17 (constraint was <any>)
  found candidate pyparsing==2.2.0 (constraint was <any>)
  found candidate six==1.10.0 (constraint was >=1.6.0)

Finding secondary dependencies:
  packaging==16.8           requires pyparsing, six
  appdirs==1.4.3            requires -
  pycparser==2.17           requires -
  six==1.10.0               requires -
  pyparsing==2.2.0          requires -

New dependencies found in this round:
  adding ['six', '', '[]']
Removed dependencies in this round:
  removing ['appdirs', '>=1.4.0', '[]']
  removing ['packaging', '>=16.8', '[]']
  removing ['pycparser', '', '[]']
  removing ['six', '>=1.6.0', '[]']
------------------------------------------------------------
Result of round 3: not stable

                          ROUND 4                           
Current constraints:
  cryptography from git+git://github.com/pyca/cryptography@master#egg=cryptography (from -r requirements.in (line 1))
  pyparsing
  six

Finding the best candidates:
  found candidate -e git+git://github.com/pyca/cryptography@master#egg=cryptography (constraint was <any>)
  found candidate pyparsing==2.2.0 (constraint was <any>)
  found candidate six==1.10.0 (constraint was <any>)

Finding secondary dependencies:
  six==1.10.0               requires -
  pyparsing==2.2.0          requires -

New dependencies found in this round:
Removed dependencies in this round:
  removing ['pyparsing', '', '[]']
  removing ['six', '', '[]']
------------------------------------------------------------
Result of round 4: not stable

                          ROUND 5                           
Current constraints:
  cryptography from git+git://github.com/pyca/cryptography@master#egg=cryptography (from -r requirements.in (line 1))

Finding the best candidates:
  found candidate -e git+git://github.com/pyca/cryptography@master#egg=cryptography (constraint was <any>)

Finding secondary dependencies:
------------------------------------------------------------
Result of round 5: stable, done

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
-e git+git://github.com/pyca/cryptography@master#egg=cryptography

[mattlong]

+1. 1.8.1 is resulting in dependencies of -e reqs being missed. This is a pretty bad backwards-imcompatible regression.

[davidovich]

Does this happen in 1.8.0 ?

[mattlong]

No, it is new in 1.8.1. After some digging, I believe I've found the cause. Bear with me as I may be using incorrect terminology at times since I'm not very familiar with the internals of pip.

For editable reqs, the resolver always calls self.repository.get_dependencies(ireq) whenever it needs to get the dependencies of the package ireq. Under the covers, get_dependencies calls reqset._prepare_file(self.finder, ireq) which is where we jump into pip internals since reqset is an instance of pip.req.req_set import RequirementSet.

_prepare_file sets ireq.prepared = True meaning that on subsequent calls with the same ireq object, an empty list will always be returned. Thus, we lose the dependencies of an editable req when the same ireq obj is used to represent them on subsequent resolver rounds. The quickest fix I can think of would be to set ireq.prepared = False before calling _prepare_file to avoid its short-circuit behavior, but that feels pretty fragile. Seems like it would be cleaner and much more performant to just memoize repositories.pypi.PyPIRepository.get_dependencies.

[davidovich]

Should be fixed by #476.