pipenv does not install all extras

[hoesler]

Pipenv seems to process only the first element in a list of extras.

Given a folder with the following Pipfile (otherwise empty):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
apache-airflow = {extras = ["s3", "druid"], version = "==1.8.2"}

[requires]
python_version = "3.5"

pipenv install installs 53 packages. The package pydruid, which is defined in extra druid, is missing.

Given this Pipfile with inverted order of extras:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
apache-airflow = {extras = ["druid", "s3"], version = "==1.8.2"}

[requires]
python_version = "3.5"

pipenv install installs 52 packages. The packages boto and filechunkio, which are defined in extra s3, are missing.

Describe your environment

1. OS Type: Mac OS 10.13.1
2. Python version: Python 2.7.10
3. Pipenv version: pipenv, version 8.3.2

[techalchemy]

This may be fixed in #1095 but I have not tested...

[kennethreitz]

assuming this is fixed now

[hoesler]

It's not. Tested with the latest version from master.

[techalchemy]

@hoesler are you still using the same exact pipfile? Can you provide the command you ran and terminal output?

[hoesler]

I created a gist for you with an example: https://gist.github.com/hoesler/66f4ab6a7d69d2cadde8bfe9a9582228

Just run docker build -t pipenv1094 https://gist.githubusercontent.com/hoesler/66f4ab6a7d69d2cadde8bfe9a9582228/raw/301bb79912a00485c9124975af2d26ff75d14711/Dockerfile

The last command checks for the boto lib in Pipfile.lock, which fails for pipenv revision 704c622

[sdementen]

I still have the issue with pipenv 10.1.0. pipenv will only install the first item of the extras list.
I see the issue is closed. Has this been fixed ?

[techalchemy]

@sdementen I just looked over your setup.py and I wanted to point out that your extras_require syntax is no longer supported by setuptools or pip and is likely one piece of the cause of your struggles. Specifying the python_version marker as a conditional install flag should not be handled in extras_require but rather in install_requires per the setuptools documentation

This may be regressed but until we remove the other variables (and we did release version 11 as well) it will be hard to untangle. I am submitting a PR to your feature branch with the changes

[sdementen]

tx @techalchemy ! I have commented your PR with a new question (unrelated I think to this issue).

And is the issue here (#1094) solved in pipenv ?

[techalchemy]

@sdementen it was. As I mentioned, it’s unclear if it’s regressed currently

[hoesler]

@techalchemy I don't know if extras_require is the problem here, but it is still mentioned in latest setuptools documentation you provided. Anyway, I think this is still a pipenv problem and I don't understand why this issue got closed.

[techalchemy]

OK this was definitely broken, can confirm. fix pending

[kyle-hamlin]

@kennethreitz I believe this is still an issue for vcs deps. Take for example: apache-airflow = {git = "https://github.com/apache/incubator-airflow.git", ref = "master", extras = ["slack", "postgres", "redis", "crypto", "s3", "celery"]} using pipenv install will not install all extras.

[techalchemy]

@Mokubyow if you want to install something from vcs you need to install it as editable in order to resolve its dependencies.

[kyle-hamlin]

@techalchemy I attempted the install with editable = true extras are still not installed.

Pipfile

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

apache-airflow = {git = "https://github.com/apache/incubator-airflow.git", editable = true, ref = "master", extras = ["slack", "postgres", "redis", "crypto", "s3", "celery"]}


[dev-packages]



[requires]

python_version = "3.6"

pipenv install && pipenv run pip freeze

alembic==0.8.10
-e git+https://github.com/apache/incubator-airflow.git@133e249e05038b25cf15203988c1c78b6974c4e2#egg=apache_airflow
bleach==2.1.2
certifi==2018.1.18
chardet==3.0.4
click==6.7
configparser==3.5.0
croniter==0.3.20
dill==0.2.7.1
docutils==0.14
Flask==0.11.1
Flask-Admin==1.4.1
Flask-Caching==1.3.3
Flask-Login==0.2.11
flask-swagger==0.2.13
Flask-WTF==0.14.2
funcsigs==1.0.0
future==0.16.0
gitdb2==2.0.3
GitPython==2.1.8
gunicorn==19.7.1
html5lib==1.0.1
idna==2.6
iso8601==0.1.12
itsdangerous==0.24
Jinja2==2.8.1
lockfile==0.12.2
lxml==3.8.0
Mako==1.0.7
Markdown==2.6.11
MarkupSafe==1.0
numpy==1.14.2
ordereddict==1.1
pandas==0.22.0
pendulum==1.4.0
psutil==4.4.2
Pygments==2.2.0
python-daemon==2.1.2
python-dateutil==2.7.0
python-editor==1.0.3
python-nvd3==0.14.2
python-slugify==1.1.4
pytz==2018.3
pytzdata==2018.3
PyYAML==3.12
requests==2.18.4
setproctitle==1.1.10
six==1.11.0
smmap2==2.0.3
SQLAlchemy==1.1.18
SQLAlchemy-Utc==0.10.0
tabulate==0.7.7
thrift==0.11.0
tzlocal==1.5.1
Unidecode==1.0.22
urllib3==1.22
webencodings==0.5.1
Werkzeug==0.14.1
WTForms==2.1
zope.deprecation==4.3.0

[techalchemy]

@Mokubyow for editable VCS dependencies, extras aren't resolved into the lockfile (on purpose) -- since the package is installed into editable mode, these are moving targets and must be re-resolved on each installation.

[sdementen]

@techalchemy not sure to understand if this is also applicable when installing the local project (".")...
In any cases, shouldn't we have some warning/raise exception if extras are given but not used ?

[kyle-hamlin]

@techalchemy I agree with @sdementen it felt very much like a bug to me when my extras were not installed without warning even though it is the intended functionality.

[bjmc]

Is there a workaround to specify a VCS install with extras that has dependencies in extras_require in the setup.py? Do I just need to list those dependencies explicitly in the Pipfile?

[techalchemy]

You put it in the egg fragment and we parse it correctly which I’m not sure we are doing yet but there is an open issue on the topic @Mokubyow

[bjmc]

It seems to work the first time when I specify it in the egg fragment during pipenv install ... command line install, but those dependencies don't appear in the Pipfile.lock so they get missed out during subsequent installs from the lockfile.

I can open another issue with a minimal example to reproduce, but I'm afraid it might be a duplicate of a known issue.

[lig]

@techalchemy this doesn't work now on 2018.10.13. Causes a lot of troubles reproducing the environment. Makes pipenv mostly unusable.

[sinistersnare]

Does not work with 2018.10.13, but does with 2018.11.14.

[elliotweiser]

Confirmed that this doesn't work w/ version 2018.11.26. Trying to install molecule[vagrant], which should in turn install python-vagrant.

$ pipenv install 'molecule[vagrant]<3.0.0' --dev && pipenv run pip freeze | grep -i vagrant
Installing molecule[vagrant]<3.0.0...
Adding molecule to Pipfile's [dev-packages]...
✔ Installation Succeeded
Installing dependencies from Pipfile.lock (32d5d4)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 64/64 — 00:00:17
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

(note the lack of output from pipenv run pip freeze | grep -i vagrant)

The only reference to vagrant in the Pipfile.lock is apparently just a piece of metadata, but nothing else:

        "molecule": {
            "extras": [
                "vagrant"
            ],

[techalchemy]

At a glance, that particular package is built with PBR and it looks like their extras aren't landing in their wheels correctly, you can see here: https://pypi.org/pypi/molecule/2.19.0/json

Compare that to the requires-dist section here: https://pypi.org/pypi/vistir/0.2.5/json

You can see the extras in the second package. The JSON API only parses requirements from wheels, and there are wheels uploaded from molecule, so that is certainly how you installed it. If the extra isn't there, pipenv can't install it. Please file an issue upstream, we have regression tests in place for this.

[elliotweiser]

@techalchemy Thanks for the clarification. I think molecule might need a PBR update, anyway. I'll file upstream. Thanks again!

[Chetabahana]

The problem still remain by the latest pipenv version 2018.11.26

Pipfile

celery = {version = "*", extras = ["redis"]}

only install celery, the redis is missing

Step #1:   File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 1009, in __init__
Step #1:     raise ImportError('Missing redis library (pip install redis)')
Step #1: ImportError: Missing redis library (pip install redis