When "Dynamically adding command line options" is added, pytest doesn't collect any tests

[assundaram]

Followed the pytest documentation for "Dynamically adding command line options"

Below is the sample code and command line output.

Contents of conftest.py:

import pytest
import sys
def pytest_cmdline_preparse(args):
    if 'xdist' in sys.modules: # pytest-xdist plugin
        num = 2
        args[:] = ["-n", str(num)] + args

def read_device_list():
    return ["192.168.72.11", "192.168.72.12"]

def pytest_configure(config):
    # read device list if we are on the master
    if not hasattr(config, "slaveinput"):
        config.iplist = read_device_list()

def pytest_configure_node(node):
    # the master for each node fills slaveinput dictionary
    # which pytest-xdist will transfer to the subprocess
    node.slaveinput["ipadr"] = node.config.iplist.pop()

@pytest.fixture(scope="session")
def device(request):
    slaveinput = getattr(request.config, "slaveinput", None)
    if slaveinput is None: # single-process execution
        ipadr = read_device_list()[0]
    else: # running in a subprocess here
        ipadr = slaveinput["ipadr"]
    return Device(ipadr)

class Device:
    def __init__(self, ipadr):
        self.ipadr = ipadr

    def __repr__(self):
        return "<Device ip=%s>" % (self.ipadr)

Contents of test_parallel.py:

import time
import pytest


class Test1:
    def test_device1(self, device):
        time.sleep(2)
        assert 0, device

    def test_device2(self, device):
        time.sleep(2)
        assert 0, device

Command Line output of pytest:

============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.5.2, pytest-3.4.1, py-1.5.2, pluggy-0.6.0
rootdir: /data/SUNDAR/test/parallel_tests, inifile:
plugins: xdist-1.22.1, forked-0.2
gw0 [0] / gw1 [0]
scheduling tests via LoadScheduling

======================================================================================================= no tests ran in 0.64 seconds ========================================================================================================

When I comment the args[:] line in conftest.py and pass the -n2 in command line, things work fine.

def pytest_cmdline_preparse(args):
    if 'xdist' in sys.modules: # pytest-xdist plugin
        num = 2
        #args[:] = ["-n", str(num)] + args

Command Line output of pytest -n2:

============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.5.2, pytest-3.4.1, py-1.5.2, pluggy-0.6.0
rootdir: /data/SUNDAR/test/parallel_tests, inifile:
plugins: xdist-1.22.1, forked-0.2
gw0 [2] / gw1 [2]
scheduling tests via LoadScheduling
FF                                                                                                                                                                                                                                    [100%]
================================================================================================================= FAILURES ==================================================================================================================
____________________________________________________________________________________________________________ Test1.test_device2 _____________________________________________________________________________________________________________
[gw1] linux -- Python 3.5.2 /usr/bin/python3

self = <test_parallel.Test1 object at 0x7f1ad8e33630>, device = <Device ip=192.168.72.11>

    def test_device2(self, device):
        time.sleep(2)
>       assert 0, device
E       AssertionError: <Device ip=192.168.72.11>
E       assert 0

test_parallel.py:12: AssertionError
____________________________________________________________________________________________________________ Test1.test_device1 _____________________________________________________________________________________________________________
[gw0] linux -- Python 3.5.2 /usr/bin/python3

self = <test_parallel.Test1 object at 0x7f46998426a0>, device = <Device ip=192.168.72.12>

    def test_device1(self, device):
        time.sleep(2)
>       assert 0, device
E       AssertionError: <Device ip=192.168.72.12>
E       assert 0

test_parallel.py:8: AssertionError
========================================================================================================= 2 failed in 2.70 seconds ==========================================================================================================

[pytestbot]

GitMate.io thinks possibly related issues are #1150 (Pass dinamically command line option into a test body), #1123 (does pytest support to this way to collect test cases?), #654 (add pass rate option to pytest), #78 (pytest_addoption - Command-line options not added), and #532 (pytest -f does not notice new tests).

[RonnyPfannschmidt]

i just made your output readable, and its clearly collected 2 tests that failed using different devices - please describe what you expected to see, because what i see is exactly what i would expect

[assundaram]

To get the 2 test collected, I have to comment out "args[:] = ["-n", str(num)] + args", and pass -n2 as the command line option with pytest invocation.

See the first command line output, no tests were collected when the -n2 is added to pytest_cmdline_preparse().
"gw0 [0] / gw1 [0]
scheduling tests via LoadScheduling"

[RonnyPfannschmidt]

@assundaram i missed the part in between since your formatting was so messed, let me fix up the original comment again

[nicoddemus]

@assundaram the problem is that you are adding -n 2 to the command-line even for the worker processes, and they are likely blowing up right at the beginning because they don't know about the -n option.

You can add -n 2 to the master worker only using this code:

def pytest_cmdline_preparse(config, args):
    worker = os.environ.get('PYTEST_XDIST_WORKER')
    if 'xdist' in sys.modules and not worker:  # pytest-xdist plugin
        num = 2
        args[:] = args + ["-n" + str(num)] 

Unfortunately you cannot use the hasattr(config, "slaveinput") trick at this point because of how things are initialized in the xdist remote worker: pytest_cmdline_preparse is called during the creation of the Config object (during the remote_initconfig call below), while the "slaveinput" attribute is added only afterwards:

https://github.com/pytest-dev/pytest-xdist/blob/182d1f6433296fc4f8be29880f87e20c150fda59/xdist/remote.py#L194-L200

[assundaram]

@nicoddemus Thanks, that resolved the issue. pytest documentation needs to be updated.

[nicoddemus]

OK thanks for the feedback @assundaram, could you open a new issue with a proposal addition to the docs, or even a PR if you have the time? Thanks, appreciate it!