pytest>=5.1.0 is unable to locate test nodes in modules when executed programmatically from a wheel installed on a Windows mapped network drive, if more than one node is passed to pytest.main

[paul-ko]

Basic description

I am in a situation where I'd like to be able to run tests from an installed wheel. Because (as I understand things) pytest custom arguments don't work when tests are invoked using a pytest command installed via a wheel (see #1596 and #3517), I am investigating an approach suggested by comments in those tickets, writing a test runner module that invokes pytest programmatically.

I have found that in pytest versions before 5.1.0, the approach I've taken works. However, from 5.1.0 onwards, pytest sometimes cannot locate test modules/nodes when it is executed from a Windows mapped drive when invoked in ways that work on pre-5.1.0 builds.

I mentioned that it "sometimes" happens. From what I've observed, it happens if there are two or more nodes passed to pytest.main. If you call it with a module and a node (e.g., blah/test/test_stuff.py and blah/test/test_other_stuff.py::test_a) it will work, but if you call it with two nodes (e.g., blah/test/test_stuff.py::test_stuff and blah/test/test_other_stuff.py::test_a) it will fail to locate the function in the second module.

Because of the nature of this issue and the version where it started reproducing, this looks like it may be somewhat related to an issue I reported recently, #5825.

pip list output

$ pip list
Package            Version
------------------ -------
atomicwrites       1.3.0
attrs              19.1.0
colorama           0.4.1
importlib-metadata 0.20
more-itertools     7.2.0
packaging          19.1
pip                19.2.3
pluggy             0.12.0
py                 1.8.0
pyparsing          2.4.2
pytest             5.1.2
runnablerepro      0.1.0
setuptools         41.0.1
six                1.12.0
wcwidth            0.1.7
wheel              0.33.6
zipp               0.6.0

Platform info

platform win32 -- Python 3.7.3, pytest-5.1.2, py-1.8.0, pluggy-0.12.0

OS version: Windows Server 2012R2

I've also reproduced this issue on 5.1.0. However, it fails to reproduce on 5.0.1.

Detailed description and repro info

I've created the repo https://github.com/paul-ko/runnable-test-repro to reproduce this issue. To summarize, the repo works like this:

runnablerepro/testdriver.py implements a simple CLI for invoking tests via pytest. It takes one or more positional arguments, which it converts to file paths or pytest nodes, building them in ways that work correctly (in pre-5.1.0 versions of pytest) when executed from an installed wheel. Those paths/nodes are passed to pytest via a call to pytest.main, along with several optional parameters if they are specified.

To set up for repro, do the following:

1. Clone the repo to a mapped network drive on Windows.
2. Create a virtualenv (my practice is to run virtualenv venv from the base directory of the repo)
3. Activate that virtualenv
4. Install pytest (using the repo requirements.txt to install the latest, or pytest==x.x.x to install a specific version)
5. From the base directory of the repo, run python setup.py bdist_wheel to build a wheel.
6. Create and activate a different virtual environment.
7. Install the wheel in that new virtualenv.
8. Run testdriver.exe test/test_fail.py::test_fail_2 test/test_pass.py::test_pass

If you have pytest 5.0.1 installed, you'll see a successful test run like this:

$ testdriver test/test_fail.py::test_fail_2 test/test_pass.py::test_pass
Namespace(file=None, test_targets=['test/test_fail.py::test_fail_2', 'test/test_pass.py::test_pass'], val=None)
args: ['k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_fail.py::test_fail_2', 'k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_pass.py::test_pass']
============================= test session starts =============================
platform win32 -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: K:\me\tmp\runnable
collected 2 items

venv\lib\site-packages\runnablerepro\test\test_fail.py F                 [ 50%]
venv\lib\site-packages\runnablerepro\test\test_pass.py .                 [100%]

================================== FAILURES ===================================
_________________________________ test_fail_2 _________________________________

    def test_fail_2():
>       assert False, 'whoa'
E       AssertionError: whoa
E       assert False

venv\lib\site-packages\runnablerepro\test\test_fail.py:6: AssertionError
hey
===================== 1 failed, 1 passed in 0.37 seconds ======================

However, if you have pytest>=5.1.0 installed, you'll see an error locating a test

$ testdriver test/test_fail.py::test_fail_2 test/test_pass.py::test_pass
Namespace(file=None, test_targets=['test/test_fail.py::test_fail_2', 'test/test_pass.py::test_pass'], val=None)
args: ['k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_fail.py::test_fail_2', 'k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_pass.py::test_pass']
============================= test session starts =============================
platform win32 -- Python 3.7.3, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
rootdir: \\giant\apps\Data\Src\me\tmp\runnable
collected 1 item

hey
============================ no tests ran in 0.10s ============================
ERROR: not found: k:\me\tmp\runnable\venv\lib\site-packages\runnablerepro\test\test_pass.py::test_pass
(no name 'k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_pass.py::test_pass' in any of [<Module test_pass.py>])

The odd thing is that this only seems to affect the second node; if I switch the order of the positional arguments it is now able to find ...test_pass.py::test_pass but not ...test_fail.py::test_fail:

$ testdriver.exe test/test_pass.py::test_pass test/test_fail.py::test_fail
Namespace(file=None, test_targets=['test/test_pass.py::test_pass', 'test/test_fail.py::test_fail'], val=None)
args: ['k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_pass.py::test_pass', 'k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_fail.py::test_fail']
============================= test session starts =============================
platform win32 -- Python 3.7.3, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
rootdir: \\giant\apps\Data\Src\me\tmp\runnable
collected 1 item

hey
============================ no tests ran in 0.07s ============================
ERROR: not found: k:\me\tmp\runnable\venv\lib\site-packages\runnablerepro\test\test_fail.py::test_fail
(no name 'k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_fail.py::test_fail' in any of [<Module test_fail.py>])

This made me suspicious that something wrong with my code was causing this, but when I print the arguments I'm passing to pytest.main everything looks to be in order, and of course there's the fact that my code works with pre-5.1.0 versions of pytest.

Additionally, it works on 5.1.0 and beyond if only one argument contains a node (i.e., the other arguments are just module filepaths):

$ testdriver.exe test/test_pass.py test/test_fail.py::test_fail
Namespace(file=None, test_targets=['test/test_pass.py', 'test/test_fail.py::test_fail'], val=None)
args: ['k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_pass.py', 'k:\\me\\tmp\\runnable\\venv\\lib\\site-packages\\runnablerepro\\test\\test_fail.py::test_fail']
============================= test session starts =============================
platform win32 -- Python 3.7.3, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
rootdir: \\giant\apps\Data\Src\me\tmp\runnable
collected 2 items

. .F                                                                     [100%]

================================== FAILURES ===================================
__________________________________ test_fail __________________________________

    def test_fail():
>       assert False, 'ouch!'
E       AssertionError: ouch!
E       assert False

venv\lib\site-packages\runnablerepro\test\test_fail.py:2: AssertionError
hey
========================= 1 failed, 1 passed in 0.16s =========================