[Question]How the fixture can communicate that it is terminating so that the master can re-schedule the tests to other workers

[assundaram]

I tried sys.exit() in a session scope fixture but even then the test is marked as Error. Also the same worker was allocated with more tests and they also end up in error. What I am trying to achieve is if a worker terminates then the master should re-schedule the tests to other workers.

Contests of conftest.py:

import pytest
import sys
import os

sys.stdout = sys.stderr

def pytest_cmdline_preparse(args):
    worker = os.environ.get('PYTEST_XDIST_WORKER')
    if 'xdist' in sys.modules and not worker:  # pytest-xdist plugin
        num = 2
        args[:] = ["-sv", "-n", str(num)] + args

def read_device_list():
    return ["192.168.72.11", "192.168.72.12"]

def pytest_configure(config):
    # read device list if we are on the master
    if not hasattr(config, "slaveinput"):
        config.iplist = read_device_list()

def pytest_configure_node(node):
    # the master for each node fills slaveinput dictionary
    # which pytest-xdist will transfer to the subprocess
    node.slaveinput["ipadr"] = node.config.iplist.pop()

@pytest.fixture(scope="session")
def device(request):
    slaveinput = getattr(request.config, "slaveinput", None)
    if slaveinput is None: # single-process execution
        ipadr = read_device_list()[0]
    else: # running in a subprocess here
        ipadr = slaveinput["ipadr"]
    if ipadr == "192.168.72.11":
        print("192.168.72.11 Crashed")
        sys.exit(-11)
    return Device(ipadr)

class Device:
    def __init__(self, ipadr):
        self.ipadr = ipadr

    def __repr__(self):
        return "<Device ip=%s>" % (self.ipadr)

Contents of test_parallel.py:

import time
import pytest


class Test1:
    def test_device1(self, device):
        time.sleep(2)
        assert 1, device

    def test_device2(self, device):
        time.sleep(2)
        assert 1, device

    def test_device3(self, device):
        time.sleep(2)
        assert 1, device

    def test_device4(self, device):
        time.sleep(2)
        assert 1, device

    def test_device5(self, device):
        time.sleep(2)
        assert 1, device

Command line with output:

(base) E:\Personal\13_Materials\Python\RA\parallel_tests>pytest --max-worker-restart=0
============================= test session starts =============================
platform win32 -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- C:\Users\sunda\Anaconda3\python.exe
cachedir: .cache
rootdir: E:\Personal\13_Materials\Python\RA\parallel_tests, inifile:
plugins: xdist-1.22.2, forked-0.2
[gw0] win32 Python 3.6.4 cwd: E:\Personal\13_Materials\Python\RA\parallel_tests
[gw1] win32 Python 3.6.4 cwd: E:\Personal\13_Materials\Python\RA\parallel_tests
[gw0] Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 10:22:32) [MSC v.1900 64 bit (AMD64)]
[gw1] Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 10:22:32) [MSC v.1900 64 bit (AMD64)]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

test_parallel.py::Test1::test_device2
test_parallel.py::Test1::test_device1 192.168.72.11 Crashed

[gw1] [ 20%] ERROR test_parallel.py::Test1::test_device2
192.168.72.11 Crashed
test_parallel.py::Test1::test_device4
[gw1] [ 40%] ERROR test_parallel.py::Test1::test_device4
[gw0] [ 60%] PASSED test_parallel.py::Test1::test_device1 192.168.72.11 Crashed

test_parallel.py::Test1::test_device5
test_parallel.py::Test1::test_device3
[gw1] [ 80%] ERROR test_parallel.py::Test1::test_device5
[gw0] [100%] PASSED test_parallel.py::Test1::test_device3

=================================== ERRORS ====================================
____________________ ERROR at setup of Test1.test_device2 _____________________
[gw1] win32 -- Python 3.6.4 C:\Users\sunda\Anaconda3\python.exe

request = <SubRequest 'device' for <Function 'test_device2'>>

    @pytest.fixture(scope="session")
    def device(request):
        slaveinput = getattr(request.config, "slaveinput", None)
        if slaveinput is None: # single-process execution
            ipadr = read_device_list()[0]
        else: # running in a subprocess here
            ipadr = slaveinput["ipadr"]
        if ipadr == "192.168.72.11":
            print("192.168.72.11 Crashed")
>           sys.exit(-11)
E           SystemExit: -11

conftest.py:35: SystemExit
____________________ ERROR at setup of Test1.test_device4 _____________________
[gw1] win32 -- Python 3.6.4 C:\Users\sunda\Anaconda3\python.exe

request = <SubRequest 'device' for <Function 'test_device4'>>

    @pytest.fixture(scope="session")
    def device(request):
        slaveinput = getattr(request.config, "slaveinput", None)
        if slaveinput is None: # single-process execution
            ipadr = read_device_list()[0]
        else: # running in a subprocess here
            ipadr = slaveinput["ipadr"]
        if ipadr == "192.168.72.11":
            print("192.168.72.11 Crashed")
>           sys.exit(-11)
E           SystemExit: -11

conftest.py:35: SystemExit
____________________ ERROR at setup of Test1.test_device5 _____________________
[gw1] win32 -- Python 3.6.4 C:\Users\sunda\Anaconda3\python.exe

request = <SubRequest 'device' for <Function 'test_device5'>>

    @pytest.fixture(scope="session")
    def device(request):
        slaveinput = getattr(request.config, "slaveinput", None)
        if slaveinput is None: # single-process execution
            ipadr = read_device_list()[0]
        else: # running in a subprocess here
            ipadr = slaveinput["ipadr"]
        if ipadr == "192.168.72.11":
            print("192.168.72.11 Crashed")
>           sys.exit(-11)
E           SystemExit: -11

conftest.py:35: SystemExit
====================== 2 passed, 3 error in 7.19 seconds ======================

(base) E:\Personal\13_Materials\Python\RA\parallel_tests>

[pytestbot]

GitMate.io thinks possibly related issues are #2861 (Question re: order of execution of test fixtures), #367 (Skip tests that require a fixture), #2246 (Question regarding fixture visibility), #2502 (How to provide fixture parameters at test level?), and #230 (Make getting test results from fixtures easier).

[RonnyPfannschmidt]

currently thereis no supported way to stop execution - i beleive it could work to set a session.shouldstop in xdist workers, but i feat that they will jsut restart