Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipenv does not pull in extras if extra name does not match module name #5944

Closed
thehesiod opened this issue Sep 22, 2023 · 8 comments
Closed

Comments

@thehesiod
Copy link

Issue description

If a requirement has extras, with a name that does not match the module name the module will not get added to the lockfile

$ cat Pipfile
[packages]
hypothesis = {extras = ["datetime"]}

[requires]
python_version = "3.11"

Note that the datetime extra should pull in pytz: https://github.com/bennylope/hypothesis-python/blob/master/setup.py#L41

Expected result

Extras specified should get added

Actual result

$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "f1eb1944c7bc948a5bc4e1d5dd0a8e75b99e6b43cad9d3a3c222e3f840a9a3bc"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pip_conf_index_global",
                "url": "https://pypi.fbn.org/simple",
                "verify_ssl": false
            }
        ]
    },
    "default": {
        "attrs": {
            "hashes": [
                "sha256:1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04",
                "sha256:6279836d581513a26f1bf235f9acd333bc9115683f14f7e8fae46c98fc50e015"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==23.1.0"
        },
        "hypothesis": {
            "extras": [
                "datetime"
            ],
            "hashes": [
                "sha256:e1d36522824d62bb3e9fcb7b57dd4a6ca330bb36921324bb19c476bdafabeda7",
                "sha256:e5d75d70f5a4fc372cddf03ec6141237a0a270ed106aeb2156a4984f06d37b0f"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==6.86.2"
        },
        "sortedcontainers": {
            "hashes": [
                "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88",
                "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0"
            ],
            "version": "==2.4.0"
        }
    },
    "develop": {}
}

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).


$ pipenv --support

Pipenv version: '2023.9.8'

Pipenv location: '/opt/homebrew/lib/python3.11/site-packages/pipenv'

Python location: '/opt/homebrew/opt/[email protected]/bin/python3.11'

OS Name: 'posix'

User pip version: '23.2.1'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.4',
 'os_name': 'posix',
 'platform_machine': 'arm64',
 'platform_python_implementation': 'CPython',
 'platform_release': '22.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT '
                     '2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000',
 'python_full_version': '3.11.4',
 'python_version': '3.11',
 'sys_platform': 'darwin'}

System environment variables:

  • SHELL
  • NVM_RC_VERSION
  • XPC_FLAGS
  • HISTCONTROL
  • TERM_PROGRAM_VERSION
  • PKG_CONFIG_PATH
  • HISTSIZE
  • __CFBundleIdentifier
  • SSH_AUTH_SOCK
  • TERM_SESSION_ID
  • GPG_TTY
  • PWD
  • LOGNAME
  • LaunchInstanceID
  • HOME
  • LANG
  • SECURITYSESSIONID
  • TMPDIR
  • FBNSECRETS_FORCE_USER
  • NVM_DIR
  • GEM_HOME
  • TERM
  • USER
  • SHLVL
  • NVM_CD_FLAGS
  • XPC_SERVICE_NAME
  • PS1
  • PATH
  • HOMEBREW_CASK_OPTS
  • OLDPWD
  • TERM_PROGRAM
  • _
  • __CF_USER_TEXT_ENCODING
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /Users/alexmohr/Downloads/google-cloud-sdk/bin:/opt/homebrew/opt/openjdk/bin:/Users/alexmohr/.gem/bin:/opt/homebrew/bin:/opt/homebrew/opt/[email protected]/bin:/opt/homebrew/opt/[email protected]/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin:/Library/Apple/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/homebrew/opt/openjdk/bin:~/.gem/ruby/2.6.0/bin:/Users/alexmohr/dev/fbn.com/ops/docker/ecr/bin:~/Applications/Sublime Text.app/Contents/SharedSupport/bin
  • SHELL: /opt/homebrew/bin/bash
  • LANG: en_US.UTF-8
  • PWD: /tmp/test

Contents of Pipfile ('/private/tmp/test/Pipfile'):

[packages]
hypothesis = {extras = ["datetime"]}

[requires]
python_version = "3.11"

Contents of Pipfile.lock ('/private/tmp/test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "f1eb1944c7bc948a5bc4e1d5dd0a8e75b99e6b43cad9d3a3c222e3f840a9a3bc"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pip_conf_index_global",
                "url": "https://pypi.fbn.org/simple",
                "verify_ssl": false
            }
        ]
    },
    "default": {
        "attrs": {
            "hashes": [
                "sha256:1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04",
                "sha256:6279836d581513a26f1bf235f9acd333bc9115683f14f7e8fae46c98fc50e015"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==23.1.0"
        },
        "hypothesis": {
            "extras": [
                "datetime"
            ],
            "hashes": [
                "sha256:e1d36522824d62bb3e9fcb7b57dd4a6ca330bb36921324bb19c476bdafabeda7",
                "sha256:e5d75d70f5a4fc372cddf03ec6141237a0a270ed106aeb2156a4984f06d37b0f"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==6.86.2"
        },
        "sortedcontainers": {
            "hashes": [
                "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88",
                "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0"
            ],
            "version": "==2.4.0"
        }
    },
    "develop": {}
}
@matteius
Copy link
Member

We need the proper package name for pip resolve/install, that is why.

@thehesiod
Copy link
Author

ya but this works in pipenv 2023.7.23 so this is a regression

@matteius
Copy link
Member

It can be considered a regression, but its not one I intend to "fix" this since was dependent on requirementslib behavior which has since been removed. I don't think its unreasonable to derive the package name from the Pipfile entry -- trying to determine it otherwise from the package itself is an expensive and error prone operation especially considering the many forms of setup.py and the desire to not evaluate code from setup.py during package resolution. Open to patches, but I won't be pursuing a patch for this.

@thehesiod
Copy link
Author

We need the proper package name for pip resolve/install, that is why.

I'm not sure what you mean by you need the proper package name. extras_require is a thing: https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies . The top level key is an alias to one of more real dependencies. From what I see pipenv will pull the req if the alias is the name of a package, but won't dig into the actual requirements for the extra. Definitely seems like a bug. The hypothesis boils down to:

extras_require={
    'datetime':  ["pytz"],
    'fakefactory': ["Faker>=0.7.0,<=0.7.1"],
    'django': ['pytz', 'django>=1.8,<2,!=1.9.*'],
    'numpy': ['numpy>=1.9.0'],
    'pytest': ['pytest>=2.8.0'],
}

if they instead put:

extras_require={
    'pytz':  ["pytz"],

and in the Pipfile we put:

hypothesis = {extras = ["pytz"]}

Remember we need to specify the alias because the alias can point to one or more modules.

pipenv would pull the requirement. This needs to get fixed otherwise we cannot rely on pipenv properly enforcing version requirements across top level modules, and extras.

@thehesiod
Copy link
Author

the original Pipfile is equivalent to pip install hypothesis[datetime], which would follow the alias and also install pytz

@matteius
Copy link
Member

matteius commented Sep 22, 2023

Sorry if there was a misunderstanding, I thought you were saying the name of the requirement in the Pipfile did not match the requirement name. When I try your example the resolve prints out:

[ ] Locking...Warning: WARNING:pipenv.patched.pip._internal.resolution.resolvelib.candidates:hypothesis 6.86.2 does not provide the extra 'datetime'

When I checked the wheel for hypothesis, we can see there is no extras group named datetime in the published wheel. It is named pytz there (I am not sure why). That is what the pypi resolution will use the wheel metadata to determine the extras, so I don't think that is a pipenv bug.

Metadata-Version: 2.1
Name: hypothesis
Version: 6.86.2
Summary: A library for property-based testing
Home-page: https://hypothesis.works
Author: David R. MacIver and Zac Hatfield-Dodds
Author-email: [email protected]
License: MPL-2.0
Project-URL: Source, https://github.com/HypothesisWorks/hypothesis/tree/master/hypothesis-python
Project-URL: Changelog, https://hypothesis.readthedocs.io/en/latest/changes.html
Project-URL: Documentation, https://hypothesis.readthedocs.io
Project-URL: Issues, https://github.com/HypothesisWorks/hypothesis/issues
Keywords: python testing fuzzing property-based-testing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Framework :: Hypothesis
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
Classifier: Operating System :: Unix
Classifier: Operating System :: POSIX
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Education :: Testing
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/x-rst
License-File: LICENSE.txt
Requires-Dist: attrs >=19.2.0
Requires-Dist: sortedcontainers <3.0.0,>=2.1.0
Requires-Dist: exceptiongroup >=1.0.0 ; python_version < "3.11"
Provides-Extra: all
Requires-Dist: black >=19.10b0 ; extra == 'all'
Requires-Dist: click >=7.0 ; extra == 'all'
Requires-Dist: django >=3.2 ; extra == 'all'
Requires-Dist: dpcontracts >=0.4 ; extra == 'all'
Requires-Dist: lark >=0.10.1 ; extra == 'all'
Requires-Dist: libcst >=0.3.16 ; extra == 'all'
Requires-Dist: numpy >=1.17.3 ; extra == 'all'
Requires-Dist: pandas >=1.1 ; extra == 'all'
Requires-Dist: pytest >=4.6 ; extra == 'all'
Requires-Dist: python-dateutil >=1.4 ; extra == 'all'
Requires-Dist: pytz >=2014.1 ; extra == 'all'
Requires-Dist: redis >=3.0.0 ; extra == 'all'
Requires-Dist: rich >=9.0.0 ; extra == 'all'
Requires-Dist: backports.zoneinfo >=0.2.1 ; (python_version < "3.9") and extra == 'all'
Requires-Dist: tzdata >=2023.3 ; (sys_platform == "win32") and extra == 'all'
Provides-Extra: cli
Requires-Dist: click >=7.0 ; extra == 'cli'
Requires-Dist: black >=19.10b0 ; extra == 'cli'
Requires-Dist: rich >=9.0.0 ; extra == 'cli'
Provides-Extra: codemods
Requires-Dist: libcst >=0.3.16 ; extra == 'codemods'
Provides-Extra: dateutil
Requires-Dist: python-dateutil >=1.4 ; extra == 'dateutil'
Provides-Extra: django
Requires-Dist: django >=3.2 ; extra == 'django'
Provides-Extra: dpcontracts
Requires-Dist: dpcontracts >=0.4 ; extra == 'dpcontracts'
Provides-Extra: ghostwriter
Requires-Dist: black >=19.10b0 ; extra == 'ghostwriter'
Provides-Extra: lark
Requires-Dist: lark >=0.10.1 ; extra == 'lark'
Provides-Extra: numpy
Requires-Dist: numpy >=1.17.3 ; extra == 'numpy'
Provides-Extra: pandas
Requires-Dist: pandas >=1.1 ; extra == 'pandas'
Provides-Extra: pytest
Requires-Dist: pytest >=4.6 ; extra == 'pytest'
Provides-Extra: pytz
Requires-Dist: pytz >=2014.1 ; extra == 'pytz'
Provides-Extra: redis
Requires-Dist: redis >=3.0.0 ; extra == 'redis'
Provides-Extra: zoneinfo
Requires-Dist: backports.zoneinfo >=0.2.1 ; (python_version < "3.9") and extra == 'zoneinfo'
Requires-Dist: tzdata >=2023.3 ; (sys_platform == "win32") and extra == 'zoneinfo'

When I try with the git repository, I have to manually manipulate the Pipfile entry to have the extras but that is working:

$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "a53d9843a7d846384f64ea100fdda25f9358f7d2151b5276169fe7adb7deea3f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "hypothesis": {
            "extras": [
                "datetime"
            ],
            "git": "git+https://github.com/bennylope/hypothesis-python.git",
            "markers": "python_version >= '3.8'",
            "ref": "73ad1f3b0ac1da959134b5a9f87659a2b3408c34"
        },
        "pytz": {
            "hashes": [
                "sha256:7b4fddbeb94a1eba4b557da24f19fdf9db575192544270a9101d8509f9f43d7b",
                "sha256:ce42d816b81b68506614c11e8937d3aa9e41007ceb50bfdcb0749b921bf646c7"
            ],
            "version": "==2023.3.post1"
        }
    },
    "develop": {}
}

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5947
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
hypothesis = {ref = "master", git = "git+https://github.com/bennylope/hypothesis-python.git", extras=["datetime"]}

[dev-packages]

[requires]
python_version = "3.9"

The biggest issue I can see is that there should be a way to pipenv install the Pipfile entry with the extras for the git repository example without having to edit the Pipfile. However for the case of the wheels extras groups not matching the setup.py extras groups from 8 years ago -- probably has to do with a recently published wheel of the hypothesis project?

@thehesiod
Copy link
Author

ok perhaps I picked a bad example, we're seeing this same issue at work for our own module, let me make an example w/o pypi

@thehesiod
Copy link
Author

sorry, will close this until I get a test case that buttons down all the hatches :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants