Large number of test failures on Windows local environment

[notatallshaw]

Description

When trying to make first simple pull request for #10415 I am unable to get tests mostly working following this guide on Windows: https://pip.pypa.io/en/latest/development/getting-started/

My final output is something like:

= 705 failed, 1455 passed, 51 skipped, 20 xfailed, 1 xpassed, 3496 warnings, 32 errors in 576.33s (0:09:36) =

I do not have this problem on Linux (under WSL 2). where only a few tests failed following the same instructions:

= 8 failed, 2211 passed, 24 skipped, 15 xfailed, 6 xpassed, 3256 warnings in 89.93s (0:01:29) =

Expected behavior

Tests in Windows should work when following the docs instructions

pip version

pip 21.3.dev0

Python version

Python 3.9.7

OS

Windows 10 Version 21H1 (OS Build 19043.1165)

How to Reproduce

Follow https://pip.pypa.io/en/latest/development/getting-started/ as best as possible.

Step 1 - Install Binary Requirements

1. Install latest version of Python from Python Foundation
2. Install Git
3. Install SVN
4. Install Mercurial
5. Install Bazaar

Step 2 - Setup Pip Virtual Environment and test depenencies

(see attached env-setup.txt for exact copy of commands and output)

1. git clone https://github.com/pypa/pip
2. cd pip
3. py -m venv venv
4. venv\Scripts\activate
5. py -m pip install -e .
6. py -m pip --version
7. py -m pip install tox

Step 3 - Run Tests

(see attached run-tests.txt for exact copy of commands and output, I had problems capturing the output so eventually I used "tee" to do this, hopefully it didn't mess up the output too much not to understand)

8. tox -e py39 -- -n auto

Output

See attached files above. (duplicated below by @pradyunsg)

• run-tests.txt
• env-setup.txt

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Immediate thoughts: I don't install svn, Mercurial or Bazaar, so those tests might fail. Also, I call my local venv .venv. You don't have to follow that, but venv will shadow the stdlib venv module, which might cause weird issues.

Beyond that, I'd have to look at the outputs, the issue says "see attached files" but I don't know where they are attached.

[notatallshaw]

Immediate thoughts: I don't install svn, Mercurial or Bazaar,

I thought installing them is weird but that's what the getting started page says to do.

Also, I call my local venv .venv. You don't have to follow that, but venv will shadow the stdlib venv module, which might cause weird issues

I get the same issue calling my venv something else and/or locating it in a different folder. The reason I called it "venv" is I'm trying to follow the getting started guide to the letter and that's what it says to do.

Beyond that, I'd have to look at the outputs, the issue says "see attached files" but I don't know where they are attached.

Where I've written "attached run-tests.txt for exact copy" the "run-tests.txt" is a hyper link to the attachment. I'm not used to attaching hyper links on GitHub this was the best I could do to make them visible. The output is over 6 MBs so I couldn't paste it in to the text box.

[uranusjr]

From a quick look it seems like an environment issue. A ton of errors are reported by PipTestEnvironment.run(), it can’t run python nor git. So I’ll definitely start there, by first making sure your git does run, and the tox-populated virtual environment is functioning properly (maybe try .tox/py39/scripts/python -m sysconfig).

[notatallshaw]

From a quick look it seems like an environment issue. A ton of errors are reported by PipTestEnvironment.run(), it can’t run python nor git.

Git is available and working as you can see from my env-setup.txt

Python is not available in my normal environment but is available in the virtual environment which is activated when I run the tests. Does tox stay in the virtual environment or does it break out when it runs subprocesses?

try `.tox/py39/scripts/python -m sysconfig

Sure I'll give that a try when I'm back home later today.

[uranusjr]

Regarding the inability to run python -m pip, I wonder if https://bugs.python.org/issue38905 is related.

For Git, is your git pointing to git.exe or git.cmd or something else?

[pradyunsg]

Can you try running with --use-venv passed through through pytest? That would be: tox -e py -- --use-venv -n auto.

[pradyunsg]

My guess (based on looking at the file on my phone) is that virtualenv is not creating a valid environment in your system for some reason. For what it's worth, pip's test suite uses a really old version of virtualenv, so that could be implicated in this.

[pradyunsg]

Further, you don't need to install Subversion, Bazaar or Mercurial -- the test suite checks if their executables are available and skips the tests if not.

[pradyunsg]

And, I'll also echo @pfmoore's suggestion of not calling your virtual environment directory venv.

[notatallshaw]

Further, you don't need to install Subversion, Bazaar or Mercurial -- the test suite checks if their executables are available and skips the tests if not.

This is what the documentation tell you to do which after many failures I tried following to the letter https://pip.pypa.io/en/latest/development/getting-started/ :

Running pip’s test suite requires supported version control tools (subversion, bazaar, git, and mercurial) to be installed.

And, I'll also echo @pfmoore's suggestion of not calling your virtual environment directory venv.

As I said in an above comment I've tried with different names and in different locations and got the same errors. But again trying to follow this https://pip.pypa.io/en/latest/development/getting-started/ to the letter:

virtualenv venv # You can also use "py -m venv venv" from python3.3+
venv\Scripts\activate

I'm collecting everyone's responses and I'll try them all when I have some time to work on this again, hopefully tomorrow. Thanks for the input.

[pradyunsg]

Nice, so we've got bugs to fix in the gettign started guide. :)

[notatallshaw]

he tox-populated virtual environment is functioning properly (maybe try .tox/py39/scripts/python -m sysconfig).

Here is the output (now using .venv as my environmental path):

(.venv) PS C:\Users\damia\pypa\pip> .tox/py39/scripts/python -m sysconfig
Platform: "win-amd64"
Python version: "3.9"
Current installation scheme: "nt"

Paths:
        data = "C:\Users\damia\pypa\pip\.tox\py39"
        include = "C:\Users\damia\AppData\Local\Programs\Python\Python39\Include"
        platinclude = "C:\Users\damia\AppData\Local\Programs\Python\Python39\Include"
        platlib = "C:\Users\damia\pypa\pip\.tox\py39\Lib\site-packages"
        platstdlib = "C:\Users\damia\pypa\pip\.tox\py39\Lib"
        purelib = "C:\Users\damia\pypa\pip\.tox\py39\Lib\site-packages"
        scripts = "C:\Users\damia\pypa\pip\.tox\py39\Scripts"
        stdlib = "C:\Users\damia\AppData\Local\Programs\Python\Python39\Lib"

Variables:
        BINDIR = "C:\Users\damia\pypa\pip\.tox\py39\Scripts"
        BINLIBDEST = "C:\Users\damia\pypa\pip\.tox\py39\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".cp39-win_amd64.pyd"
        INCLUDEPY = "C:\Users\damia\AppData\Local\Programs\Python\Python39\Include"
        LIBDEST = "C:\Users\damia\AppData\Local\Programs\Python\Python39\Lib"
        SO = ".cp39-win_amd64.pyd"
        TZPATH = ""
        VERSION = "39"
        abiflags = ""
        base = "C:\Users\damia\pypa\pip\.tox\py39"
        exec_prefix = "C:\Users\damia\pypa\pip\.tox\py39"
        installed_base = "C:\Users\damia\AppData\Local\Programs\Python\Python39"
        installed_platbase = "C:\Users\damia\AppData\Local\Programs\Python\Python39"
        platbase = "C:\Users\damia\pypa\pip\.tox\py39"
        platlibdir = "lib"
        prefix = "C:\Users\damia\pypa\pip\.tox\py39"
        projectbase = "C:\Users\damia\pypa\pip\.tox\py39\Scripts"
        py_version = "3.9.7"
        py_version_nodot = "39"
        py_version_short = "3.9"
        srcdir = "C:\Users\damia\pypa\pip\.tox\py39\Scripts"
        userbase = "C:\Users\damia\AppData\Roaming\Python"

I don't know what this is supposed to look like, but it concerns me it seems to mostly be pointing to my base Python install and not my virtual environment .venv.

Regarding the inability to run python -m pip, I wonder if https://bugs.python.org/issue38905 is related.

Possibly, I definetly reproduce the sys.executable behavior you describe there.

For Git, is your git pointing to git.exe or git.cmd or something else?

git.exe:

(.venv) PS C:\Users\damia\pypa\pip> get-command git

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     git.exe                                            2.31.1.1   C:\Program Files\Git\cmd\git.exe

Can you try running with --use-venv passed through through pytest? That would be: tox -e py -- --use-venv -n auto.

Roughly same number of failures but completes ~8x faster.

[notatallshaw]

After seeing @uranusjr hint about this bug https://bugs.python.org/issue38905 I tried a fresh install of Python but I chose the non-default option of adding Python to the PATH

This seems to have "fixed" the issue and I am getting significantly less failures running this:

23 failed, 2152 passed, 73 skipped, 10 xfailed, 6 xpassed, 3496 warnings in 545.51s (0:09:05)

However this is less than ideal as I was explicitly avoiding putting this version of Python off the PATH as

1. I mostly use conda versions of Python for my actual work, having Python Foundation Python as my default Python is going to make my other development environments awkward
2. As per this discussion I was trying to keep to using py not python: Why does Pip documentation refer Windows to use an executable named "py"? #10433

I'm going to have to have a think about how to proceed with my pip development environment.

This issue can probably be closed based on updating the documentation pull request #10449 unless anyone thinks there's something that can be technically fixed here by pip and not tox.

[uranusjr]

I don’t have python in PATH on my machine, and the test suite works for me.

Regarding the inability to run python -m pip, I wonder if bugs.python.org/issue38905 is related.

Possibly, I definetly reproduce the sys.executable behavior you describe there.

This makes me think the issue may be in Tox; it should use shutil.which() to resolve what Python executable to create its virtual environment with instead of relying on python being resolved correctly by subprocess implemntation.

I’d suggest the next step is to reduce the setup into a minimal reproducing example (blank project + venv + tox + a simplistic test showing the unintended behaviour). Once you succeed in building that, report the issue to Tox.

[notatallshaw]

This makes me think the issue may be in Tox; it should use shutil.which() to resolve what Python executable to create its virtual environment with instead of relying on python being resolved correctly by subprocess implemntation.

I’d suggest the next step is to reduce the setup into a minimal reproducing example (blank project + venv + tox + a simplistic test showing the unintended behaviour). Once you succeed in building that, report the issue to Tox.

I agree, but I don't understand exactly what tox does yet, I'll try and read through some guides on how to set up a project and work on a minimal test case.

[notatallshaw]

Closing this issue, with nox instead of tox there are way less FAILEDs (specifically 8 integration tests and 5 unit tests) running pip's test suite locally on Windows even though Python isn't on the PATH.

What failures there are I will try and identify specifically, for example this issue covers all the unit test fails I get: #10816