Regression in 2020.11.15 failing to install dependencies defined in setup.py

[ms-lolo]

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.

Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.

Make sure to mention your debugging experience if the documented solution failed.

Issue description

I was investigating this issue: #4355 and noticed that my project that contains all the dependencies in setup.py and has the Pipfile simply install "." in editable mode is no longer installing anything defined in install_requires in my setup.py. However, this behavior is not present in 2020.11.4.

My setup.py file is defining "install_requires": ["numpy"],
and my Pipfile looks like this:

[packages]
my-thing = {path = ".",editable = true}

Expected result

I expect the lock file to contain numpy and numpy to be isntalled.

Actual result

My package is installed but numpy is missing. No errors visible in any of the verbose output.

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

---

$ pipenv --support

Pipenv version: '2020.11.15'

Pipenv location: '/home/myuser/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pipenv'

Python location: '/home/myuser/.pyenv/versions/3.6.8/bin/python3.6'

Python installations found:

• 3.8.5: /usr/bin/python3
• 3.8.5: /usr/bin/python3.8
• 3.8.5: /bin/python3
• 3.8.5: /bin/python3.8
• 3.7.5: /home/myuser/.pyenv/versions/3.7.5/bin/python3.7m
• 3.7.4: /home/myuser/.pyenv/versions/3.7.4/bin/python3.7m
• 3.6.8: /home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A/bin/python3
• 3.6.8: /home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A/bin/python3.6
• 3.6.8: /home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A/bin/python3.6m
• 3.6.8: /home/myuser/.pyenv/versions/3.6.8/bin/python3.6m
• 3.6.8: /home/myuser/.pyenv/versions/3.6.8/bin/python3.6
• 3.6.8: /home/myuser/.pyenv/versions/3.6.8/bin/python3
• 3.6.8: /home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A/bin/python3
• 3.6.8: /home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A/bin/python3.6
• 3.6.8: /home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A/bin/python3.6m
• 3.6.8: /home/myuser/.pyenv/versions/3.6.8/bin/python3.6m
• 2.7.18: /usr/bin/python2
• 2.7.18: /usr/bin/python2.7
• 2.7.18: /bin/python2
• 2.7.18: /bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.8',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.0-52-generic',
 'platform_system': 'Linux',
 'platform_version': '#57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020',
 'python_full_version': '3.6.8',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• LSCOLORS
• SESSION_MANAGER
• PYENV_HOOK_PATH
• QT_ACCESSIBILITY
• COLORTERM
• PYENV_SHELL
• XDG_CONFIG_DIRS
• LESS
• XDG_MENU_PREFIX
• GNOME_DESKTOP_SESSION_ID
• PIPENV_ACTIVE
• TERMINATOR_DBUS_PATH
• GNOME_SHELL_SESSION_MODE
• SSH_AUTH_SOCK
• DIRENV_DIR
• TERMINATOR_UUID
• XMODIFIERS
• DESKTOP_SESSION
• SSH_AGENT_PID
• DEVOPS_PYPI_PW
• PYENV_VERSION
• GTK_MODULES
• PWD
• LOGNAME
• XDG_SESSION_DESKTOP
• XDG_SESSION_TYPE
• GPG_AGENT_INFO
• XAUTHORITY
• GJS_DEBUG_TOPICS
• WINDOWPATH
• HOME
• USERNAME
• IM_CONFIG_PHASE
• LANG
• DEVOPS_BUILDS_PW
• LS_COLORS
• XDG_CURRENT_DESKTOP
• VIRTUAL_ENV
• VTE_VERSION
• INVOCATION_ID
• TERMINATOR_DBUS_NAME
• MANAGERPID
• DIRENV_DIFF
• PYENV_DIR
• GJS_DEBUG_OUTPUT
• XDG_SESSION_CLASS
• TERM
• ZSH
• USER
• PIP_DISABLE_PIP_VERSION_CHECK
• DISPLAY
• APP_NAME
• SHLVL
• PAGER
• QT_IM_MODULE
• PYTHONDONTWRITEBYTECODE
• KUBECONFIG
• XDG_RUNTIME_DIR
• PYENV_ROOT
• DEVOPS_PYPI_URL
• JOURNAL_STREAM
• XDG_DATA_DIRS
• PATH
• GDMSESSION
• DBUS_SESSION_BUS_ADDRESS
• DIRENV_WATCHES
• DEVOPS_PYPI_UN
• GIO_LAUNCHED_DESKTOP_FILE_PID
• GIO_LAUNCHED_DESKTOP_FILE
• OLDPWD
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: /home/myuser/.pyenv/versions/3.6.8/bin:/home/myuser/.pyenv/libexec:/home/myuser/.pyenv/plugins/python-build/bin:/home/myuser/.pyenv/plugins/pyenv-virtualenv/bin:/home/myuser/.pyenv/plugins/pyenv-update/bin:/home/myuser/.pyenv/plugins/pyenv-installer/bin:/home/myuser/.pyenv/plugins/pyenv-doctor/bin:/home/myuser/projects/things/thing/my-thing/bin:/home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A/bin:/home/myuser/.pyenv/shims:/home/myuser/.pyenv/bin:/home/myuser/.local/bin:/home/myuser/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
• SHELL: /usr/bin/zsh
• LANG: en_US.UTF-8
• PWD: /home/myuser/projects/things/thing/my-thing
• VIRTUAL_ENV: /home/myuser/.local/share/virtualenvs/my-thing-sO64gR_A

---

Contents of Pipfile ('/home/myuser/projects/things/thing/my-thing/Pipfile'):

[[source]]
name = "artifacts"
url = "$DEVOPS_PYPI_URL"
verify_ssl = true

[dev-packages]

[packages]
my-thing = {path = ".",editable = true}

[requires]
python_version = "3.6.8"

Contents of Pipfile.lock ('/home/myuser/projects/things/thing/my-thing/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "2e965c720ff5205f879ac66e310611ee278a76f98b98369f07f25eba3620b70b"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6.8"
        },
        "sources": [
            {
                "name": "artifacts",
                "url": "$DEVOPS_PYPI_URL",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "my-thing": {
            "editable": true,
            "path": "."
        },
        "install-requires": {
            "hashes": [
                "sha256:d6229486e085a330041d3cc6f47eeb8d2fd0c255cf81a4b899b287c306deb717"
            ],
            "version": "==0.3.0"
        }
    },
    "develop": {}
}

[frostming]

Would you mind providing the content of setup.py. You can redact the secrets of course

[ms-lolo]

@frostming I was able to really narrow down the issue here. It looks like something is parsing my setup.py file instead of executing it… here are a few examples:

I was able to get things working fine with this very basic example:

from setuptools import setup

setup_kargs = {
    "name": "my-foo",
    "author": "someone",
    "author_email": "[email protected]",
    "description": "",
    "license": "",
    "classifiers": [],
    "packages": ["foo"],
    "package_dir": {"": "src/python"},
    "package_data": {},
    "install_requires": ["numpy"],
    "python_requires": ">=3.6",
    "entry_points": {},
}

setup(**setup_kargs)

however, if you pull out the required packages into a variable and use that, it gets ignored:

from setuptools import setup

install_requires = ["numpy"]

setup_kargs = {
    "name": "my-foo",
    "author": "someone",
    "author_email": "[email protected]",
    "description": "",
    "license": "",
    "classifiers": [],
    "packages": ["foo"],
    "package_dir": {"": "src/python"},
    "package_data": {},
    "install_requires": install_requires,
    "python_requires": ">=3.6",
    "entry_points": {},
}

setup(**setup_kargs)

lastly, it seems like the name of the variable actually matters! this example below actually installs numpy:

from setuptools import setup

numpy = ["numpy"]

setup_kargs = {
    "name": "my-foo",
    "author": "someone",
    "author_email": "[email protected]",
    "description": "",
    "license": "",
    "classifiers": [],
    "packages": ["foo"],
    "package_dir": {"": "src/python"},
    "package_data": {},
    "install_requires": numpy,
    "python_requires": ">=3.6",
    "entry_points": {},
}

setup(**setup_kargs)

just to verify it's using the name of the variable for something meaningful, this example below actually throws an error about failing to find a distribution named foo:

from setuptools import setup

foo = ["numpy"]

setup_kargs = {
    "name": "my-foo",
    "author": "someone",
    "author_email": "[email protected]",
    "description": "",
    "license": "",
    "classifiers": [],
    "packages": ["foo"],
    "package_dir": {"": "src/python"},
    "package_data": {},
    "install_requires": foo,
    "python_requires": ">=3.6",
    "entry_points": {},
}

setup(**setup_kargs)

ERROR: Could not find a version that matches foo (from -r /tmp/pipenv73n16i6vrequirements/pipenv-tor_rpu6-constraints.txt (line 2))
No versions found

let me know if you need any other examples!

[frostming]

@ms-lolo Thanks for that. I think you have already figured out the workaround.
Either of the following solutions will do the trick:

1. Use setup(install_requires=install_requires) directly instead of via a dict.
2. Define requirements literally in install_requires field.

[ms-lolo]

@frostming I tried applying your suggested fix but I think I ran into some other unexpected behavior and now I'm just even more confused and, unfortunately, troubleshooting this is taking entire days and I can't continue.

I'm guessing my previous version of code was only working by chance and was just not doing what I was defining it to do. All I can do is describe my project structure and hope someone else can spend time to reproduce and troubleshoot because I am out of time here.

I have two python packages in one repo, foo-common and foo-contrib. They both use setuptools-scm and are unreleased, so they are both pre-releases. The foo-contrib package depends on foo-common and installs it as a local editable package. Here are some example snippets:

foo-contrib Pipfile

[packages]
foo-common = {path = "../foo-common",editable = true}
foo-contrib = {path = ".",editable = true}

foo-contrib setup.py

setup(
    name=distribution_name,
    author="…",
    author_email="…",
    description="…",
    license="…",
    classifiers=[
        "Private :: Do Not Upload to pypi server",
    ],
    namespace_packages=namespace_packages,
    packages=pkgs,
    package_dir={"": package_dir},
    package_data={"": ["py.typed"]},
    data_files=generate_data_files(),
    install_requires=[
        "foo-common",
    ],
    python_requires=">=3.6",
    entry_points={},
    use_scm_version={"root": "..", "local_scheme": "dirty-tag"}
)

I am now getting an error about failing to find a version that satisfied my dependencies:

ERROR: Could not find a version that matches foo-common (from -r /tmp/pipenva1ka99_prequirements/pipenv-1y3v0uol-constraints.txt (line 6))
Skipped pre-versions: 0.1.dev218, 0.1.dev219, 0.1.dev220, 0.1.dev221, 0.1.dev222, 0.1.dev223, 0.1.dev224, 0.1.dev225, 0.1.dev226, 0.1.dev227, 0.1.dev228, 0.1.dev229, 0.1.dev230, 0.1.dev231, 0.1.dev232, 0.1.dev233, 0.1.dev234, 0.1.dev235, 0.1.dev237, 0.1.dev238, 0.1.dev239, 0.1.dev240, 0.1.dev241, 0.1.dev242, 0.1.dev243, 0.1.dev244, 0.1.dev245, 0.1.dev246, 0.1.dev247, 0.1.dev248, 0.1.dev249, 0.1.dev250, 0.1.dev251, 0.1.dev252, 0.1.dev253, 0.1.dev254, 0.1.dev255, 0.1.dev256, 0.1.dev257, 0.1.dev260, 0.1.dev261, 0.1.dev263, 0.1.dev264, 0.1.dev265, 0.1.dev266, 0.1.dev267, 0.1.dev268, 0.1.dev269, 0.1.dev270, 0.1.dev271, 0.1.dev272, 0.1.dev273, 0.1.dev274, 0.1.dev275, 0.1.dev276, 0.1.dev277, 0.1.dev278, 0.1.dev279, 0.1.dev280, 0.1.dev281, 0.1.dev282, 0.1.dev283, 0.1.dev284, 0.1.dev285, 0.1.dev286, 0.1.dev287, 0.1.dev288, 0.1.dev289, 0.1.dev290, 0.1.dev291, 0.1.dev292, 0.1.dev293, 0.1.dev294, 0.1.dev295, 0.1.dev296, 0.1.dev297, 0.1.dev298, 0.1.dev299, 0.1.dev300, 0.1.dev301, 0.1.dev302, 0.1.dev303, 0.1.dev304, 0.1.dev305, 0.1.dev306, 0.1.dev307, 0.1.dev308, 0.1.dev309, 0.1.dev310, 0.1.dev311, 0.1.dev312, 0.1.dev313, 0.1.dev314, 0.1.dev315, 0.1.dev316, 0.1.dev317, 0.1.dev318, 0.1.dev319, 0.1.dev320, 0.1.dev321, 0.1.dev322

This happens on 2020-11-4 too, so I guess it's related to avoiding the setup(**config) type code from my previous version? Why is pipenv trying to find a release for my package when Pipfile has it listed as a local editable package? I can't specify --pre because that's very much not ok for all packages in my env, and making this a per-package flag is already an open feature request. But I am super confused about why this is only happening now, when I simply shuffled logic around in my setup file without changing the actual config values.

I'm adding this comment here because it's all part of my attempt to get 2020-11-15 to be able to work in my environments. It seems like a different issue, and possibly not a regression.

[ms-lolo]

#1760 is the feature request preventing me from just hiding from this issue and letting pipenv install the latest version of my dependency, regardless of it being unreleased.

[frostming]

Pipenv uses a "resolve then override" method when there are editable requirements with the same name as the released one. So the release version must exist in the first place to make the resolution work.

[ms-lolo]

In that case, do you think actually packaging and releasing a version v0.1.0 would resolve this? Would it fetch the v0.1.0 release and then override it with my local version before locking?

Any idea why this is not what I am observing with my previous version of setup.py? Is it just this same bug manifesting in different ways on the different pipenv versions?

I think I can give this a try relatively easily tomorrow morning, just trying to make sure I understand before I make more of a mess on my end :)

[ms-lolo]

was able to test this today and confirmed that I am back to a functional environment with the workarounds from this thread. #1760 is a much more critical feature request for me now because I cannot require the tagging of a fake, stable versions of all local dependencies in my environments.

thanks for all the help getting me running while this bug is open!

[yrsurya]

#!/usr/bin/env python3

## Modules
import setuptools
from getpass import getuser
import os
from pkg_resources import working_set, Requirement
from setuptools import setup
from subprocess import check_output
import sys
from time import time

## Constants
VERSION = '0.13.7'
REQUIREMENTS = [
        'aws-python',
        'Jinja2'
]


## Funtions
def get_reqs(env, reqs):
    if env == 'prod':
        reqs = [
            f'{r} == {working_set.find(Requirement(r)).version}'
            for r in reqs
        ]
    return reqs

def print_reqs(env, reqs):
    if env == 'prod':
        for r in reqs:
            print(f'+ fixed dependency: {r}', file=sys.stderr)
    else:
        print('+ using latest dependencies', file=sys.stderr)

def get_version(env, version_base):
   # git_id = check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()
    timestamp = int(time())

    if env == 'prod':
        suffix = ''

    elif env == 'dev':
        suffix = f'b{timestamp}'

    else:
        suffix = f'.dev{timestamp}+git{git_id}'

    return f'{version_base}{suffix}'

def get_env():
    env = os.environ.get('ENVIRONMENT')
    if not env:
        try:
            env = getuser()
        except (KeyError, ImportError) as error:
            env = 'local'
    return env


if __name__ == '__main__':
    env = get_env()
    reqs = get_reqs(env, REQUIREMENTS)
    version = get_version(env, VERSION)
    setup(
        name='abc',
        version=version,
        license='BSD 3-Clause',

        packages=['abc'],
        entry_points={
            'console_scripts': [
            ],
        },
        install_requires=reqs,
    )
    print(f'+ version: {version}', file=sys.stderr)
    print_reqs(env, reqs)

[yrsurya]

Hello all , I am facing issue to release from master branch where we have multibranch Jenkins project . I am able to upload succesfully from development branch but while using master branch where ENVIRONMENT=prod facing this issue
AttributeError: 'NoneType' object has no attribute 'version'

while running pipenv install -e .
pipenv --version
pipenv, version 2020.11.15

Is this issue causing because of this bug?
Any help would be appreciate

[frostming]

Fixed on v2021.11.5