Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tox installing the wrong version of pip #3416

Closed
robsdedude opened this issue Oct 21, 2024 · 3 comments
Closed

Tox installing the wrong version of pip #3416

robsdedude opened this issue Oct 21, 2024 · 3 comments

Comments

@robsdedude
Copy link
Contributor

Issue

As the title says, tox seems to be installing the wrong version of pip which then fails as that version of pip is incompatible with the Python version I want to test against.

Environment

Here's a docker file to reproduce the environment:

Dockerfile 
FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
    apt-get install -y locales && \
    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
 && rm -rf /var/lib/apt/lists/*
ENV LANG=en_US.UTF-8

RUN apt-get --quiet update && apt-get --quiet install -y \
    software-properties-common \
    bash \
    python3 \
    python3-pip \
    git \
    curl \
    tar \
    wget \
 && rm -rf /var/lib/apt/lists/*

# Install Build Tools
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev \
        libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev \
        libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
        ca-certificates && \
    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Install pyenv
RUN git clone https://github.com/pyenv/pyenv.git .pyenv
ENV PYENV_ROOT=/.pyenv
ENV PATH="$PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH"

# Setup python version
ENV PYTHON_VERSIONS="3.13 3.7"

RUN for version in $PYTHON_VERSIONS; do \
        pyenv install $version; \
    done
RUN pyenv rehash
RUN pyenv global $(pyenv versions --bare --skip-aliases | sort --version-sort --reverse)

# Install Latest pip and setuptools for each environment
# + tox and tools for starting the tests
# https://pip.pypa.io/en/stable/news/
RUN for version in $PYTHON_VERSIONS; do \
        python$version -m pip install -U pip && \
        python$version -m pip install -U coverage tox; \
    done
Output of pip list of the host Python, where tox is installed 
Package       Version
------------- -------
cachetools    5.5.0
chardet       5.2.0
colorama      0.4.6
coverage      7.6.4
distlib       0.3.9
filelock      3.16.1
packaging     24.1
pip           24.2
platformdirs  4.3.6
pluggy        1.5.0
pyproject-api 1.8.0
tox           4.23.0
virtualenv    20.27.0

Tox file

[tox]
env_list = py{37,313}-{unit,integration,performance}

[testenv]
; Any dependency here will suffice to show the issue. It's just about getting tox to run `pip install`.
deps = pytest
commands =
    unit: python -c "print('All 0 tests passed ;)')"

Output of running tox

Output of tox -rvv 
py37-unit: 260 W remove tox env folder /demo/.tox/py37-unit [tox/tox_env/api.py:325]
.pkg: 328 W remove tox env folder /demo/.tox/.pkg [tox/tox_env/api.py:325]
py37-unit: 392 I could not create app data folder /.local/share/virtualenv due to PermissionError(13, 'Permission denied') [virtualenv/app_data/__init__.py:42]
py37-unit: 392 D app data folder /.local/share/virtualenv has no write access [virtualenv/app_data/__init__.py:46]
py37-unit: 392 D created temporary app data folder /tmp/tmp36qn2959 [virtualenv/app_data/via_tempdir.py:17]
py37-unit: 395 I find interpreter for spec PythonSpec(major=3, minor=7) [virtualenv/discovery/builtin.py:73]
py37-unit: 396 D filesystem is case-sensitive [virtualenv/info.py:25]
py37-unit: 396 I proposed PythonInfo(spec=CPython3.13.0.final.0-64, exe=/.pyenv/versions/3.13.0/bin/python3, platform=linux, version='3.13.0 (main, Oct 21 2024, 10:29:34) [GCC 9.4.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:80]
py37-unit: 396 D discover PATH[0]=/.pyenv/versions/3.13.0/bin [virtualenv/discovery/builtin.py:149]
py37-unit: 397 D discover PATH[1]=/.pyenv/libexec [virtualenv/discovery/builtin.py:149]
py37-unit: 397 D discover PATH[2]=/.pyenv/plugins/python-build/bin [virtualenv/discovery/builtin.py:149]
py37-unit: 397 D discover PATH[3]=/.pyenv/shims [virtualenv/discovery/builtin.py:149]
py37-unit: 398 D get interpreter info via cmd: /.pyenv/shims/python3 /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/discovery/py_info.py FC32MYnXaoGBArZSxiphOlH1ZyZZuE2U pXlyszgDRwudW4yy2ORsJMTqr9vCmQzf [virtualenv/discovery/cached_py_info.py:112]
py37-unit: 494 D wrote python info of %s at (PosixPath('/.pyenv/shims/python3'), PosixPath('/tmp/tmp36qn2959/py_info/1/2dd06b78807279f2ee536ee29dadbdd2bf9c199967de713a0e1014465800b480.json')) [virtualenv/app_data/via_disk_folder.py:151]
py37-unit: 495 D get interpreter info via cmd: /.pyenv/shims/python3.7 /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/discovery/py_info.py p1FwqZqpodGqTeAuZK6M3gzkJp7gUzjx Shu4xtxWUnRtYfYh7U2SypWCpFMEZOZS [virtualenv/discovery/cached_py_info.py:112]
py37-unit: 613 D wrote python info of %s at (PosixPath('/.pyenv/shims/python3.7'), PosixPath('/tmp/tmp36qn2959/py_info/1/8556f02201b4ec3b5a4313fad946ea660348ad7f4f85894eef7f20175a75c2dc.json')) [virtualenv/app_data/via_disk_folder.py:151]
py37-unit: 614 D get interpreter info via cmd: /.pyenv/versions/3.7.17/bin/python3.7 /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/discovery/py_info.py FlFSkQtV1k0eItIlBrapx0oJ0g9TEwNu FL4GuctxvBbOakVJyYTicivlxSakoXne [virtualenv/discovery/cached_py_info.py:112]
py37-unit: 675 D wrote python info of %s at (PosixPath('/.pyenv/versions/3.7.17/bin/python3.7'), PosixPath('/tmp/tmp36qn2959/py_info/1/453d35917d3fc333f631f96df756b6f06a586bc16e7e1a13ec55ed458ee40f02.json')) [virtualenv/app_data/via_disk_folder.py:151]
py37-unit: 675 I proposed PathPythonInfo(spec=CPython3.7.17.final.0-64, system=/.pyenv/versions/3.7.17/bin/python3.7, exe=/.pyenv/shims/python3.7, platform=linux, version='3.7.17 (default, Oct 21 2024, 10:30:24) \n[GCC 9.4.0]', encoding_fs_io=utf-8-UTF-8) [virtualenv/discovery/builtin.py:80]
py37-unit: 675 D accepted PathPythonInfo(spec=CPython3.7.17.final.0-64, system=/.pyenv/versions/3.7.17/bin/python3.7, exe=/.pyenv/shims/python3.7, platform=linux, version='3.7.17 (default, Oct 21 2024, 10:30:24) \n[GCC 9.4.0]', encoding_fs_io=utf-8-UTF-8) [virtualenv/discovery/builtin.py:82]
py37-unit: 704 I create virtual environment via CPython3Posix(dest=/demo/.tox/py37-unit, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:50]
py37-unit: 704 D create folder /demo/.tox/py37-unit/bin [virtualenv/util/path/_sync.py:12]
py37-unit: 705 D create folder /demo/.tox/py37-unit/lib/python3.7/site-packages [virtualenv/util/path/_sync.py:12]
py37-unit: 705 D write /demo/.tox/py37-unit/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:33]
py37-unit: 705 D 	home = /.pyenv/versions/3.7.17/bin [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 705 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 705 D 	version_info = 3.7.17.final.0 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 705 D 	virtualenv = 20.27.0 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 705 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 705 D 	base-prefix = /.pyenv/versions/3.7.17 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 705 D 	base-exec-prefix = /.pyenv/versions/3.7.17 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 706 D 	base-executable = /.pyenv/versions/3.7.17/bin/python3.7 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 706 D symlink /.pyenv/versions/3.7.17/bin/python3.7 to /demo/.tox/py37-unit/bin/python [virtualenv/util/path/_sync.py:32]
py37-unit: 706 D create virtualenv import hook file /demo/.tox/py37-unit/lib/python3.7/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:91]
py37-unit: 707 D create /demo/.tox/py37-unit/lib/python3.7/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:94]
py37-unit: 707 D ============================== target debug ============================== [virtualenv/run/session.py:52]
py37-unit: 707 D debug via /demo/.tox/py37-unit/bin/python /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:200]
py37-unit: 707 D {
  "sys": {
    "executable": "/demo/.tox/py37-unit/bin/python",
    "_base_executable": null,
    "prefix": "/demo/.tox/py37-unit",
    "base_prefix": "/.pyenv/versions/3.7.17",
    "real_prefix": null,
    "exec_prefix": "/demo/.tox/py37-unit",
    "base_exec_prefix": "/.pyenv/versions/3.7.17",
    "path": [
      "/.pyenv/versions/3.7.17/lib/python37.zip",
      "/.pyenv/versions/3.7.17/lib/python3.7",
      "/.pyenv/versions/3.7.17/lib/python3.7/lib-dynload",
      "/demo/.tox/py37-unit/lib/python3.7/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "3.7.17 (default, Oct 21 2024, 10:30:24) \n[GCC 9.4.0]",
  "makefile_filename": "/.pyenv/versions/3.7.17/lib/python3.7/config-3.7m-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/.pyenv/versions/3.7.17/lib/python3.7/os.py'>",
  "site": "<module 'site' from '/.pyenv/versions/3.7.17/lib/python3.7/site.py'>",
  "datetime": "<module 'datetime' from '/.pyenv/versions/3.7.17/lib/python3.7/datetime.py'>",
  "math": "<module 'math' from '/.pyenv/versions/3.7.17/lib/python3.7/lib-dynload/math.cpython-37m-x86_64-linux-gnu.so'>",
  "json": "<module 'json' from '/.pyenv/versions/3.7.17/lib/python3.7/json/__init__.py'>"
} [virtualenv/run/session.py:53]
py37-unit: 740 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/tmp/tmp36qn2959) [virtualenv/run/session.py:57]
py37-unit: 742 D install pip from wheel /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/seed/wheels/embed/pip-24.2-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py37-unit: 742 D install setuptools from wheel /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/seed/wheels/embed/setuptools-75.2.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py37-unit: 742 D install wheel from wheel /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/seed/wheels/embed/wheel-0.44.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py37-unit: 744 D build install image for wheel-0.44.0-py3-none-any.whl to /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/wheel-0.44.0-py3-none-any [virtualenv/seed/embed/via_app_data/pip_install/base.py:47]
py37-unit: 744 D build install image for setuptools-75.2.0-py3-none-any.whl to /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/setuptools-75.2.0-py3-none-any [virtualenv/seed/embed/via_app_data/pip_install/base.py:47]
py37-unit: 744 D build install image for pip-24.2-py3-none-any.whl to /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/pip-24.2-py3-none-any [virtualenv/seed/embed/via_app_data/pip_install/base.py:47]
py37-unit: 769 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/wheel-0.44.0-py3-none-any/wheel to /demo/.tox/py37-unit/lib/python3.7/site-packages/wheel [virtualenv/util/path/_sync.py:40]
py37-unit: 786 D copy /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/wheel-0.44.0-py3-none-any/wheel-0.44.0.virtualenv to /demo/.tox/py37-unit/lib/python3.7/site-packages/wheel-0.44.0.virtualenv [virtualenv/util/path/_sync.py:40]
py37-unit: 787 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/wheel-0.44.0-py3-none-any/wheel-0.44.0.dist-info to /demo/.tox/py37-unit/lib/python3.7/site-packages/wheel-0.44.0.dist-info [virtualenv/util/path/_sync.py:40]
py37-unit: 792 D generated console scripts wheel wheel3.7 wheel3 wheel-3.7 [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
py37-unit: 900 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/pip-24.2-py3-none-any/pip-24.2.dist-info to /demo/.tox/py37-unit/lib/python3.7/site-packages/pip-24.2.dist-info [virtualenv/util/path/_sync.py:40]
py37-unit: 905 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/pip-24.2-py3-none-any/pip to /demo/.tox/py37-unit/lib/python3.7/site-packages/pip [virtualenv/util/path/_sync.py:40]
py37-unit: 922 D copy /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/setuptools-75.2.0-py3-none-any/distutils-precedence.pth to /demo/.tox/py37-unit/lib/python3.7/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:40]
py37-unit: 923 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/setuptools-75.2.0-py3-none-any/pkg_resources to /demo/.tox/py37-unit/lib/python3.7/site-packages/pkg_resources [virtualenv/util/path/_sync.py:40]
py37-unit: 929 D copy /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/setuptools-75.2.0-py3-none-any/setuptools-75.2.0.virtualenv to /demo/.tox/py37-unit/lib/python3.7/site-packages/setuptools-75.2.0.virtualenv [virtualenv/util/path/_sync.py:40]
py37-unit: 930 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/setuptools-75.2.0-py3-none-any/setuptools to /demo/.tox/py37-unit/lib/python3.7/site-packages/setuptools [virtualenv/util/path/_sync.py:40]
py37-unit: 1027 D copy /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/pip-24.2-py3-none-any/pip-24.2.virtualenv to /demo/.tox/py37-unit/lib/python3.7/site-packages/pip-24.2.virtualenv [virtualenv/util/path/_sync.py:40]
py37-unit: 1028 D generated console scripts pip-3.7 pip pip3.7 pip3 [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
py37-unit: 1056 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/setuptools-75.2.0-py3-none-any/_distutils_hack to /demo/.tox/py37-unit/lib/python3.7/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:40]
py37-unit: 1056 D copy directory /tmp/tmp36qn2959/wheel/3.7/image/1/CopyPipInstall/setuptools-75.2.0-py3-none-any/setuptools-75.2.0.dist-info to /demo/.tox/py37-unit/lib/python3.7/site-packages/setuptools-75.2.0.dist-info [virtualenv/util/path/_sync.py:40]
py37-unit: 1058 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
py37-unit: 1058 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:63]
py37-unit: 1061 D write /demo/.tox/py37-unit/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:33]
py37-unit: 1062 D 	home = /.pyenv/versions/3.7.17/bin [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1062 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1062 D 	version_info = 3.7.17.final.0 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1062 D 	virtualenv = 20.27.0 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1062 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1062 D 	base-prefix = /.pyenv/versions/3.7.17 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1062 D 	base-exec-prefix = /.pyenv/versions/3.7.17 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1062 D 	base-executable = /.pyenv/versions/3.7.17/bin/python3.7 [virtualenv/create/pyenv_cfg.py:38]
py37-unit: 1064 W install_deps> python -I -m pip install pytest [tox/tox_env/api.py:427]
Traceback (most recent call last):
  File "/.pyenv/versions/3.7.17/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/.pyenv/versions/3.7.17/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/__main__.py", line 22, in <module>
    from pip._internal.cli.main import main as _main
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_internal/cli/main.py", line 11, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_internal/build_env.py", line 18, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_internal/cli/spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_internal/utils/logging.py", line 13, in <module>
    from pip._vendor.rich.console import (
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_vendor/rich/console.py", line 41, in <module>
    from pip._vendor.typing_extensions import (
  File "/demo/.tox/py37-unit/lib/python3.7/site-packages/pip/_vendor/typing_extensions.py", line 1039
    def TypedDict(typename, fields=_marker, /, *, total=True, closed=False, **kwargs):
                                            ^
SyntaxError: invalid syntax
py37-unit: 1181 C exit 1 (0.12 seconds) /demo> python -I -m pip install pytest pid=151 [tox/execute/api.py:286]
  py37-unit: FAIL code 1 (0.92 seconds)
  evaluation failed :( (1.01 seconds)

So you can see that tox/virtualenv picks up pip from the python3.13 installation (which is the one that is used to invoke tox).

y37-unit: 742 D install pip from wheel /.pyenv/versions/3.13.0/lib/python3.13/site-packages/virtualenv/seed/wheels/embed/pip-24.2-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]

That fails because that pip version dropped support for Python 3.7.

There's a workaround here:
You can add

download = true

to the [testenv] section of the tox.ini file.
However, this workaround does not work when combined with use_develop = true.

Minimal example

I've cobbled together a minimal reproducer repository https://github.com/robsdedude/tox-pip-reproducer
@gaborbernat
Copy link
Member

https://tox.wiki/en/4.23.0/faq.html#testing-end-of-life-python-versions

@robsdedude
Copy link
Contributor Author

robsdedude commented Oct 21, 2024 •
edited
Loading

Thanks. That indeed fixes it.

I suggest updating the docs, though. I checked and here it states

  • cross-Python compatible: tox requires CPython 3.7 and higher, but it can create environments 2.7 or later

Without mentioning that special fiddling is required for this to work. That's where I stopped looking into the docs.
Sorry for the noise.

@gaborbernat
Copy link
Member

Sure, PR welcome to fix that.

@robsdedude robsdedude mentioned this issue Oct 21, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gaborbernat @robsdedude