pytest began spuriously rejecting pytest.skip in modules in 3.x

[riastradh-probcomp]

In pytest 2.x, I used pytest.skip('...') to skip entire modules of tests that are not enabled by a command-line flag --integration, which have additional import dependencies and take longer to load and run.

This useful functionality was broken in pytest 3.x with commit d81f230, which makes pytest.skip simply crash in this case.

I can't use pytestmark = pytest.mark.skipif(...) because the ensuing code contains imports that must be skipped as well or else pytest will fail trying to load the module -- in other words, I really do want to abort the rest of the module.

How do I restore this useful functionality?

Example. In conftest.py:

def pytest_addoption(parser):
    parser.addoption('--integration', action='store_true',
        help='run integration tests')

In test_foo.py:

import pytest

if not pytest.config.getoption('--integration'):
    pytest.skip('specify --integration to run integration tests')

import nonexistent

def test_nonexistent():
    assert nonexistent.wotsit() == nonexistent.warbleglobber

Output:

% python -m pytest
============================= test session starts ==============================
platform linux2 -- Python 2.7.6, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /tmp/user/1000/20170329/venv/test, inifile:
collected 0 items / 1 errors 

==================================== ERRORS ====================================
_________________________ ERROR collecting test_foo.py _________________________
Using pytest.skip outside of a test is not allowed. If you are trying to decorate a test function, use the @pytest.mark.skip or @pytest.mark.skipif decorators instead.
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.09 seconds ============================
% pip freeze
argparse==1.2.1
py==1.4.33
pytest==3.0.7
wsgiref==0.1.2
% uname -a
Linux ubuntu 3.13.0-66-generic #108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Thanks for submitting an issue!

Here's a quick checklist in what to include:

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[RonnyPfannschmidt]

we made this change after people again and again would use pytest.skip instead of the pytest.mark.skip mark to mark test functions thus skipping whole test modules instead of just a function

however the use-case you present is also valid and i wonder if there is a reasonable way to compromise

[The-Compiler]

The Skipped exception already has allow_module_level=True as argument which is used by pytest.importorskip internally - so you should already be able to do raise pytest.skip.Exception("...", allow_module_level=True).

I think we should just expose that via pytest.skip(...) too.

[RonnyPfannschmidt]

seems like this is a 2 line change + a 30 line test

[riastradh-probcomp]

It would be nice if there were some way to skip the tests that works both in 2.x and 3.x, because I would like to ship our software to users who can run py.test --pyargs xyz to run the tests once installed and who are not necessarily experienced Python developers working in virtualenvs with bleeding-edge versions of everything. So requiring an extra argument allow_module_level=True to pytest.skip won't help for that purpose.

[nicoddemus]

I see, but we don't plan to release a 2.10 release with that support, sorry. You can follow up the discussion that lead to this decision in #607.

As a solution, you can easily provide this function for your users to use:

import pytest

if pytest.__version__ > '3':
    def skip_module(reason):
        raise pytest.skip.Exception(reason, allow_module_level=True)        
else:
    def skip_module(reason):
        pytest.skip(reason=reason)

And then use that instead of the previous pytest.skip(reason='reasons') at the module level.

@riastradh-probcomp would you like to contribute the allow_module_level change? As @RonnyPfannschmidt said should be simple, and it would ship with pytest 3.1 which we plan to release soon.

[nicoddemus]

Closing in favor of #2805