Pipenv Lock (Or Install) Does Not Respect Index Specified For A Package

[janneronkko]

Issue description

Index argument for a package is not respected

Expected result

Pipenv only installs (locks) versions from specified package index.

Actual result

Pipenv installs (locks) the package with greatest version from a package index not specified in the Pipfile.

Steps to replicate

Run the following script:

#! /usr/bin/env bash

set -euxo pipefail


function finish {
  set +e

  kill -9 ${PYPI_SERVER1_PID} || true
  kill -9 ${PYPI_SERVER2_PID} || true
}

trap finish EXIT

cat > example.py <<EOF
import pkg_resources

def main():
    print(pkg_resources.get_distribution('example'))
EOF

cat > setup.py <<EOF
import os

import setuptools

setuptools.setup(
    name='example',
    version=os.environ['EXAMPLE_VERSION'],
    description='Example package',
    py_modules=['example'],
    python_requires='>=3.6',
    entry_points={
        'console_scripts': [
            'show-installed-package = example:main',
        ],
    },
)
EOF

EXAMPLE_VERSION=1 python setup.py bdist_wheel
EXAMPLE_VERSION=2 python setup.py bdist_wheel

python -m venv venv
venv/bin/pip install pypiserver

mkdir -p server1 server2
cp dist/example-1-py3-none-any.whl server1/
cp dist/example-2-py3-none-any.whl server2/

function run_pypi_server() {
  venv/bin/pypi-server \
    -p 808${1} \
    -i 127.0.0.1 \
    server${1}/ 2> pypi-server${1}.log &
}

run_pypi_server 1
PYPI_SERVER1_PID=$!

run_pypi_server 2
PYPI_SERVER2_PID=$!

rm -f Pipfile.lock

cat > Pipfile <<EOF
[[source]]
url = "http://127.0.0.1:8081"
name = "server1"

[[source]]
url = "http://127.0.0.1:8082"
name = "server2"

[packages]
example = {version="*", index="server1"}

[dev-packages]

[requires]
python_version = "$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
EOF

pipenv lock
pipenv sync
pipenv run show-installed-package

The script creates two versions of example Python packge (version 1 and 2). Then it starts pypi server; server 1 contains example version 1 and server 2 contains example version 2.

The Pipfile states that example package of any version should be installed from index server1 (that contains only version 1 of example package).

The result is that version 2 of example package is installed (from server 2)

Documentation (https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes) states:

If you’d like a specific package to be installed with a specific package index, you can do the following:

---

$ pipenv --support

Pipenv version: '2020.11.15'

Pipenv location: '/usr/lib/python3.9/site-packages/pipenv'

Python location: '/usr/bin/python'

Python installations found:

• 3.9.2: /usr/bin/python3.9
• 3.9.2: /usr/bin/python3
• 3.7.9: /usr/bin/pypy3
• 2.7.18: /usr/bin/python2.7
• 2.7.18: /usr/bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.11.1-arch1-1',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Tue, 23 Feb 2021 14:05:30 +0000',
 'python_full_version': '3.9.2',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• SESSION_MANAGER
• WINDOWID
• QT_SCREEN_SCALE_FACTORS
• COLORTERM
• XDG_SESSION_PATH
• HISTCONTROL
• TMUX
• HISTSIZE
• LANGUAGE
• LC_ADDRESS
• LC_NAME
• SSH_AUTH_SOCK
• HISTTIMEFORMAT
• SHELL_SESSION_ID
• DESKTOP_SESSION
• LC_MONETARY
• SSH_AGENT_PID
• GTK_RC_FILES
• XCURSOR_SIZE
• CLOUDSDK_PYTHON_ARGS
• EDITOR
• GTK_MODULES
• XDG_SEAT
• PWD
• XDG_SESSION_DESKTOP
• LOGNAME
• XDG_SESSION_TYPE
• XAUTHORITY
• MOTD_SHOWN
• GTK2_RC_FILES
• HOME
• LC_PAPER
• LANG
• HISTFILE
• XDG_CURRENT_DESKTOP
• KONSOLE_DBUS_SERVICE
• CLOUDSDK_ROOT_DIR
• KONSOLE_DBUS_SESSION
• PROFILEHOME
• XDG_SEAT_PATH
• KONSOLE_VERSION
• KDE_SESSION_UID
• CLOUDSDK_PYTHON
• XDG_SESSION_CLASS
• TERM
• LC_IDENTIFICATION
• GOOGLE_CLOUD_SDK_HOME
• USER
• TMUX_PANE
• COLORFGBG
• KDE_SESSION_VERSION
• PAM_KWALLET5_LOGIN
• VISUAL
• DISPLAY
• SHLVL
• LC_TELEPHONE
• LC_MESSAGES
• LC_MEASUREMENT
• XDG_VTNR
• XDG_SESSION_ID
• MOZ_PLUGIN_PATH
• LC_CTYPE
• XDG_RUNTIME_DIR
• LC_TIME
• QT_AUTO_SCREEN_SCALE_FACTOR
• LC_COLLATE
• XCURSOR_THEME
• KDE_FULL_SESSION
• PATH
• HISTFILESIZE
• DBUS_SESSION_BUS_ADDRESS
• KDE_APPLICATIONS_AS_SCOPE
• HG
• MAIL
• LC_NUMERIC
• OLDPWD
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/janne/bin:/usr/lib/colorgcc/bin:/opt/google-cloud-sdk/bin:/home/janne/bin:/usr/lib/colorgcc/bin:/home/janne/bin:/usr/lib/colorgcc/bin:/opt/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/jvm/default/bin
• SHELL: /bin/bash
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: /home/janne/tmp/pipenvbug

---

Contents of Pipfile ('/home/janne/tmp/pipenvbug/Pipfile'):

[[source]]
url = "http://127.0.0.1:8081"
name = "server1"

[[source]]
url = "http://127.0.0.1:8082"
name = "server2"

[packages]
example = {version="*", index="server1"}

[dev-packages]

[requires]
python_version = "3.9"

Contents of Pipfile.lock ('/home/janne/tmp/pipenvbug/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "78e9f526b9a5f7eda8ccc51a8927df178ce4b229e58f42dfa0c4de8ad2b9b07e"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "server1",
                "url": "http://127.0.0.1:8081"
            },
            {
                "name": "server2",
                "url": "http://127.0.0.1:8082"
            }
        ]
    },
    "default": {
        "example": {
            "hashes": [
                "sha256:77b20dfead4f4754da288294f2be8730c70c6835c4e654f0bf4c27905c462cfa"
            ],
            "index": "server1",
            "version": "==2"
        }
    },
    "develop": {}
}

[reinvantveer]

I can second this behaviour. It's causing a lot of trouble recently, since I didn't know about "dependency confusion" as a risk. Now we published a library to an internal package index that has recently gotten a "competitor" at pypi.org.

Pipenv lets me specify the correct index, but doesn't honor it when running pipenv install. I'd like to point out that this is a big security risk and this issue should be labeled as such, I believe.

[reinvantveer]

Thanks @janneronkko for saving me the time to describe the bug in full!

[matteius]

Ah-ha I got it to run. because I changed the path to not use venv I had to also remove that from the pypi-server command. Ok so it ran and generated a Pipfile.lock and the actual package installed is in fact 2 using pipenv==2022.1.8

matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4637$ pipenv run pip freeze
example==2

Also checked it on my pip 22.0.4 branch and that also has the issue.

[matteius]

Actually I am wondering if this will even be possible to exclude pypi from the search indexes within pip.
https://stackoverflow.com/a/67442488

[matteius]

@frostming If you have time to have your attention drawn to this issue, I think it is critically important based on the reproduction. There are a couple issues at play here, but the primary one of this ticket is that the resolver is not respecting the index specifications for a particular requirement.

[matteius]

I messed around with this most of the day without much success, like there was some forward progress but ultimately broke a bunch of tests so not really progress. I did learn these three things:
1.) There is a bug where if the package name exists in pypi then it will takes precedence in the resolver. Since this test was for two private pypis, having the package named example was creating this confusion and causing more headaches so I renamed it to something unique to try and sort out issues 2 & 3.
2.) There is the issue of the Resolver not respecting the specified index so it can in this example it picks version 2 when only version 1 is available in the specified package index.
3.) There is the issue of the Pipenv install not respecting the specified index in the Package.lock when it goes to install that version.

[matteius]

This is going to be fairly hard to solve I think. For 2.) the resolver not respecting the indexes ... we create one pip package finder and one pip resolver to resolve all requirements. Pipenv's Resolver knows about the index_lookup mapping of which packages are restricted to a particular index, but there is no real way to pass this into Pip's resolver. It uses the finder._link_collector.search_scope.index_urls which it includes all of them and kind of has to because it is used to resolved all constraints not just the one with the index restriction. So then Pip doesn't seem to support the same notion of an index_lookup ... not sure how to proceed with a fix at this time.

[matteius]

That being said, I have an experimental branch that changed vendor'd code including pip code to try and solve for this problem. In my branch issue-4637 I have your test example passing. I am running the test suite now locally to see how bad it messes up anything else, but it is bad because it requires changes to pip internals and I can't see the light at the end of the tunnel for what that entails.

[matteius]

Ok I opened a PR, 1 test is failing when run in the group but when I run it singularly it passes every time (but also fails locally if I run the whole group). Will have to investigate that, and what can be done to support what I am trying to do within pip, since I modified the pip internals for part of it: #4983

However, it does seem to fix your issue at least in terms of your example test.

[reinvantveer]

@matteius Thanks for taking the time to do this, I do believe this could fix a serious security issue!

[matteius]

@janneronkko or @reinvantveer this is mostly set, but I've run into some snags adding a unit test to the project for it. May be able to move forward with out that, but its not ideal. @janneronkko example involves running two pypi test servers and generating a test package on the fly. We have a test fixture for a single pypi test server but I think its port is dynamic and some other differences making the script hard to port. I had started something for it but lost those changes in my stash when I had to destroy my local copy and reclone, but its ok because I hadn't gotten very far. If either of you have time to consider the testing angle that could really help prevent future regressions.

[matteius]

I merged the fix in master -- feel free to respond if you have ideas on getting a test in place. Wanted to ensure that the fix gets in the next release however.

[matteius]

2022.3.23 has been released!