extras_require in setup.py is not correctly evaluated

[matthieucan]

Issue description

When using variables with extras_require in setup.py, pipenv fails to resolve dependencies.

Here's a minimal reproducible example:

from setuptools import find_namespace_packages, setup

install_deps = ["requests"]

dev_deps = ["requests"]
# extras_require = {"dev": ["requests"]}
extras_require = {"dev": dev_deps}

setup(
    name="foo",
    version="0.1.0",
    description="foo",
    packages=find_namespace_packages(where="."),
    install_requires=install_deps,
    extras_require=extras_require,
)

[[source]]
url = 'https://pypi.python.org/simple'
verify_ssl = true
name = 'pypi'

[packages]
foo = {editable = true,extras = ["dev"],path = "."}

[dev-packages]
foo = {editable = true,path = "."}

Expected result

Running pipenv install "-e .[dev]" should complete.

Actual result

See verbose output here https://pastebin.com/DY83M1fL

Steps to replicate

See above.

Note

When not using variables (and using the commented line extras_require = {"dev": ["requests"]} instead of the other one), pipenv runs successfully.

---

$ pipenv --support

Pipenv version: '2020.6.2'

Python installations found:

• 3.7.5: <PATH>/bin/python3
• 3.7.5: <PATH>/bin/python3
• 3.7.5: /usr/bin/python3.7
• 3.7.5: /usr/bin/python3
• 3.7.5: /usr/bin/python3.7m
• 3.7.5: /bin/python3.7
• 3.7.5: /bin/python3
• 3.7.5: /bin/python3.7m
• 2.7.17: /usr/bin/python2.7
• 2.7.17: /usr/bin/python2
• 2.7.17: /bin/python2.7
• 2.7.17: /bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.3.0-55-generic',
 'platform_system': 'Linux',
 'platform_version': '#49-Ubuntu SMP Thu May 21 12:47:19 UTC 2020',
 'python_full_version': '3.7.5',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

• ANDROID_HOME
• CLUTTER_BACKEND
• COLORTERM
• DBUS_SESSION_BUS_ADDRESS
• DEFAULTS_PATH
• DESKTOP_SESSION
• DISPLAY
• EDITOR
• GDMSESSION
• GDM_LANG
• GPG_AGENT_INFO
• GREP_COLORS
• GTK_OVERLAY_SCROLLING
• HOME
• JAVA_HOME
• JDK_HOME
• LANG
• LANGUAGE
• LC_ADDRESS
• LC_IDENTIFICATION
• LC_MEASUREMENT
• LC_MONETARY
• LC_NAME
• LC_NUMERIC
• LC_PAPER
• LC_TELEPHONE
• LC_TIME
• LESS
• LESSOPEN
• LESS_TERMCAP_mb
• LESS_TERMCAP_md
• LESS_TERMCAP_me
• LESS_TERMCAP_se
• LESS_TERMCAP_so
• LESS_TERMCAP_ue
• LESS_TERMCAP_us
• LOGNAME
• LS_COLORS
• MANDATORY_PATH
• NEXUS_PASSWORD
• NEXUS_USERNAME
• OLDPWD
• PAGER
• PAPERSIZE
• PATH
• PWD
• PYTHONSTARTUP
• QT_QPA_PLATFORMTHEME
• SESSION_MANAGER
• SHELL
• SHLVL
• SSH_AGENT_PID
• SSH_AUTH_SOCK
• TERM
• TERMINATOR_DBUS_NAME
• TERMINATOR_DBUS_PATH
• TERMINATOR_UUID
• TMPDIR
• TMUX
• TMUX_PANE
• USER
• VISUAL
• VTE_VERSION
• XAUTHORITY
• XDG_CONFIG_DIRS
• XDG_CURRENT_DESKTOP
• XDG_DATA_DIRS
• XDG_GREETER_DATA_DIR
• XDG_MENU_PREFIX
• XDG_RUNTIME_DIR
• XDG_SEAT
• XDG_SEAT_PATH
• XDG_SESSION_CLASS
• XDG_SESSION_DESKTOP
• XDG_SESSION_ID
• XDG_SESSION_PATH
• XDG_SESSION_TYPE
• XDG_VTNR
• _
• PS1
• VIRTUAL_ENV
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

---

Contents of Pipfile:

[[source]]
url = 'https://pypi.python.org/simple'
verify_ssl = true
name = 'pypi'

[packages]
foo = {editable = true,extras = ["dev"],path = "."}

[dev-packages]
foo = {editable = true,path = "."}

[zeebonk]

@frostming why is this tagged as "Vendored Dependencies"?

[matthieucan]

Hello, any update on this issue? I'm really curious on what kind of parsing is done for this to fail.

[gjabell]

Would also be great to get an update on this issue; it makes it really annoying to use Pipenv to install from setup.py.

[draked2]

I've just encountered this with a slight different minimal example:

Minimal example that pipenv can install with pipenv install "-e .[dev]"

from setuptools import find_packages, setup
import sys

setup(name="foo",
      version="0.1",
      description="A foo utility",
      author="name",
      author_email='[email protected]',
      platforms=["any"], 
      license="BSD",
      extras_require={
          "dev": ["PyQt5" ],
      },
      )

Update to make minimal example break when running pipenv install "-e .[dev]":

from setuptools import find_packages, setup
import sys

setup(name="foo",
      version="0.1",
      description="A foo utility",
      author="name",
      author_email='[email protected]',
      platforms=["any"], 
      license="BSD",
      extras_require={
          "dev": ["PyQt5" if sys.version_info.major >= 3 else "python_qt5"],
      },
      )

[matteius]

Can this be rechecked with pipenv==2022.8.19?

[ringerc]

It looks like extras_require is also handled in a setup.cfg now. But either way, it must be explicitly referenced with the extras=["extra1", "extra2"] option in a Pipfile's [packages] entries or with pipenv install -e 'pkg[extra1,extra2]'

See also #1746

[rrlamichhane]

@ringerc

Can you elaborate on your answer?
I'm trying to use this in my Pipfile and pipenv lock isn't updating the Pipfile.lock file as i change my pyproject.toml.

# pyproject.toml segment
[project.optional-dependencies]
rj = [
    "alembic"
]

# Pipfile segment
[rj]
rj-template = {extras = ["rj"], path = "."}

When I generate my Pipfile.lock using pipenv lock; the old packages under rj are not removed, and alembic is not added.

[ringerc]

@rrlamichhane

# setup.cfg
# ... snip irrelevant ...
[options]
package_dir =
    = .
packages = find:
python_requires = >=3.8
install_requires = file: requirements.txt
[options.extras_require]
dev =
    pip-tools
test =
    pytest
    pytest-cov
    pyhamcrest

$ pipenv install -e '.[dev,test]'

resulting Pipfile autogenerated:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
mypackagename = {editable = true, extras = ["dev", "test"], path = "."}

[requires]
python_version = "3.11"

I don't know if this works with pyproject.toml. I'm completely lost in the amazing mess that Python project dependencies management has turned into, and the way some tools support some subsets of features and config mechanisms, others support different subsets etc.

[rrlamichhane]

I have found out that it works if you add it to packages section, but not for custom defined packages like rj-packages for example. I have opened an issue here:
#5817