Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Take --pyargs into account to find initial conftests (was: py.test --pyargs my_pkg doesn't run pytest_addoption in my_pkg/conftest.py) #1596

Open
Korijn opened this issue Jun 8, 2016 · 14 comments
Labels
type: enhancement new feature or API change, should be merged into features branch

Comments

@Korijn
Copy link

Korijn commented Jun 8, 2016

As part of our build process, we run the test suite after creating the packaging to check that the installed package works as intended. We recently added an option to py.test for some conditional testing, by placing a function pytest_addoption in the conftest.py file that is in the root of the package (so next to my_pkg/__init__.py). When calling py.test on the source, this works as intended. Tests are skipped when we omit the option, and they are executed when we pass the option.

my_pkg/conftest.py:

def pytest_addoption(parser):
    """Adds options to the pytest parser."""
    parser.addoption("--azure-storage-emulator",
                     action="store_true",
                     help="run tests that depend on the azure storage emulator")

However, when we run py.test --pyargs my_pkg in an environment where the built package is installed, we see the following error message (even though the test suite is run):

=================================== ERRORS ====================================
______________________________ ERROR collecting  ______________________________
..\..\envs\_test\lib\site-packages\my_pkg\storage\tests\test_azure.py:18: in <module>
    not pytest.config.getoption("--azure-storage-emulator"),
..\..\envs\_test\lib\site-packages\_pytest\config.py:1061: in getoption
    raise ValueError("no option named %r" % (name,))
E   ValueError: no option named 'azure_storage_emulator'

All other tests, including those depending on fixtures in the same conftest.py file, simply pass.

We're running on Windows and Linux, and both on dev machines as on CI (AppVeyor and Codeship), this scenario occurs. We're using the following related packages:

    pytest:                         2.9.1-py_1
    pytest-cov:                     2.2.1-py35_0
    pytest-mock:                    0.10.1-py35_0
    pytest-xdist:                   1.14-py35_0
    python:                         3.5.1-4

Running with --traceconfig does show that the conftest.py file is registered:

PLUGIN registered: <module 'my_pkg.conftest' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\my_pkg\\conftest.py'>

full output:

[_test] C:\Users\Korijn\Miniconda3\conda-bld\test-tmp_dir>py.test --traceconfig --pyargs my_pkg
PLUGIN registered: <_pytest.config.PytestPluginManager object at 0x0000027EA83061D0>
PLUGIN registered: <_pytest.config.Config object at 0x0000027EA8999198>
PLUGIN registered: <module '_pytest.mark' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\mark.py'>
PLUGIN registered: <module '_pytest.main' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\main.py'>
PLUGIN registered: <module '_pytest.terminal' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\terminal.py'>
PLUGIN registered: <module '_pytest.runner' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\runner.py'>
PLUGIN registered: <module '_pytest.python' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\python.py'>
PLUGIN registered: <module '_pytest.pdb' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\pdb.py'>
PLUGIN registered: <module '_pytest.unittest' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\unittest.py'>
PLUGIN registered: <module '_pytest.capture' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\capture.py'>
PLUGIN registered: <module '_pytest.skipping' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\skipping.py'>
PLUGIN registered: <module '_pytest.tmpdir' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\tmpdir.py'>
PLUGIN registered: <module '_pytest.monkeypatch' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\monkeypatch.py'>
PLUGIN registered: <module '_pytest.recwarn' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\recwarn.py'>
PLUGIN registered: <module '_pytest.pastebin' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\pastebin.py'>
PLUGIN registered: <module '_pytest.helpconfig' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\helpconfig.py'>
PLUGIN registered: <module '_pytest.nose' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\nose.py'>
PLUGIN registered: <module '_pytest.assertion' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\assertion\\__init__.py'>
PLUGIN registered: <module '_pytest.genscript' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\genscript.py'>
PLUGIN registered: <module '_pytest.junitxml' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\junitxml.py'>
PLUGIN registered: <module '_pytest.resultlog' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\resultlog.py'>
PLUGIN registered: <module '_pytest.doctest' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\doctest.py'>
PLUGIN registered: <module '_pytest.cacheprovider' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\_pytest\\cacheprovider.py'>
PLUGIN registered: <module 'pytest_cov.plugin' from 'c:\\users\\korijn\\miniconda3\\envs\\_test\\lib\\site-packages\\pytest_cov-2.2.1-py3.5.egg\\pytest_cov\\plugin.py'>
PLUGIN registered: <module 'pytest_mock' from 'c:\\users\\korijn\\miniconda3\\envs\\_test\\lib\\site-packages\\pytest_mock-0.10.1-py3.5.egg\\pytest_mock.py'>
PLUGIN registered: <module 'xdist.plugin' from 'c:\\users\\korijn\\miniconda3\\envs\\_test\\lib\\site-packages\\pytest_xdist-1.14-py3.5.egg\\xdist\\plugin.py'>
PLUGIN registered: <module 'xdist.looponfail' from 'c:\\users\\korijn\\miniconda3\\envs\\_test\\lib\\site-packages\\pytest_xdist-1.14-py3.5.egg\\xdist\\looponfail.py'>
PLUGIN registered: <module 'xdist.boxed' from 'c:\\users\\korijn\\miniconda3\\envs\\_test\\lib\\site-packages\\pytest_xdist-1.14-py3.5.egg\\xdist\\boxed.py'>
PLUGIN registered: <_pytest.capture.CaptureManager object at 0x0000027EA95BF208>
PLUGIN registered: <Session 'test-tmp_dir'>
PLUGIN registered: <_pytest.cacheprovider.LFPlugin object at 0x0000027EA95C9E80>
PLUGIN registered: <_pytest.terminal.TerminalReporter object at 0x0000027EA95F6320>
PLUGIN registered: <_pytest.python.FixtureManager object at 0x0000027EA960E7F0>
============================= test session starts =============================
platform win32 -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
using: pytest-2.9.1 pylib-1.4.31
setuptools registered plugins:
  pytest-cov-2.2.1 at c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_cov-2.2.1-py3.5.egg\pytest_cov\plugin.py
  pytest-mock-0.10.1 at c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_mock-0.10.1-py3.5.egg\pytest_mock.py
  pytest-xdist-1.14 at c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_xdist-1.14-py3.5.egg\xdist\plugin.py
  pytest-xdist-1.14 at c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_xdist-1.14-py3.5.egg\xdist\looponfail.py
  pytest-xdist-1.14 at c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_xdist-1.14-py3.5.egg\xdist\boxed.py
active plugins:
    main                : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\main.py
    mark                : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\mark.py
    pytest_mock         : c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_mock-0.10.1-py3.5.egg\pytest_mock.py
    python              : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\python.py
    genscript           : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\genscript.py
    runner              : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\runner.py
    pytest_cov          : c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_cov-2.2.1-py3.5.egg\pytest_cov\plugin.py
    terminal            : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\terminal.py
    skipping            : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\skipping.py
    tmpdir              : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\tmpdir.py
    resultlog           : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\resultlog.py
    lfplugin            : <_pytest.cacheprovider.LFPlugin object at 0x0000027EA95C9E80>
    xdist.boxed         : c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_xdist-1.14-py3.5.egg\xdist\boxed.py
    assertion           : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\assertion\__init__.py
    2743010877904       : <_pytest.config.PytestPluginManager object at 0x0000027EA83061D0>
    nose                : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\nose.py
    helpconfig          : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\helpconfig.py
    pdb                 : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\pdb.py
    cacheprovider       : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\cacheprovider.py
    pytestconfig        : <_pytest.config.Config object at 0x0000027EA8999198>
    pastebin            : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\pastebin.py
    doctest             : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\doctest.py
    junitxml            : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\junitxml.py
    funcmanage          : <_pytest.python.FixtureManager object at 0x0000027EA960E7F0>
    recwarn             : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\recwarn.py
    session             : <Session 'test-tmp_dir'>
    xdist               : c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_xdist-1.14-py3.5.egg\xdist\plugin.py
    terminalreporter    : <_pytest.terminal.TerminalReporter object at 0x0000027EA95F6320>
    capturemanager      : <_pytest.capture.CaptureManager object at 0x0000027EA95BF208>
    xdist.looponfail    : c:\users\korijn\miniconda3\envs\_test\lib\site-packages\pytest_xdist-1.14-py3.5.egg\xdist\looponfail.py
    capture             : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\capture.py
    monkeypatch         : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\monkeypatch.py
    unittest            : C:\Users\Korijn\Miniconda3\envs\_test\lib\site-packages\_pytest\unittest.py
rootdir: C:\Users\Korijn\Miniconda3\conda-bld\test-tmp_dir, inifile:
plugins: cov-2.2.1, mock-0.10.1, xdist-1.14
PLUGIN registered: <module 'my_pkg.conftest' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\my_pkg\\conftest.py'>
PLUGIN registered: <module 'my_pkg.sub_pkg.conftest' from 'C:\\Users\\Korijn\\Miniconda3\\envs\\_test\\lib\\site-packages\\my_pkg\\sub_pkg\\conftest.py'>
collected 352 items / 1 errors

. ................................................................................................................................................................................................................................................................................................................................................................

=================================== ERRORS ====================================
______________________________ ERROR collecting  ______________________________
..\..\envs\_test\lib\site-packages\my_pkg\storage\tests\test_azure.py:18: in <module>
    not pytest.config.getoption("--azure-storage-emulator"),
..\..\envs\_test\lib\site-packages\_pytest\config.py:1061: in getoption
    raise ValueError("no option named %r" % (name,))
E   ValueError: no option named 'azure_storage_emulator'
==================== 352 passed, 1 error in 57.56 seconds =====================

I tried adding a default value:

azure_storage_emulator = pytest.mark.skipif(
    not pytest.config.getoption("--azure-storage-emulator", default=False),
    reason="need --azure-storage-emulator option to run"
)

This avoids the exception, but I still can't pass the option:

[_test] C:\Users\Korijn\Miniconda3\conda-bld\test-tmp_dir>py.test --pyargs my_pkg
============================= test session starts =============================                                                                                                                                                              
platform win32 -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1                                                                                                                                                                        
rootdir: C:\Users\Korijn\Miniconda3\conda-bld\test-tmp_dir, inifile:                                                                                                                                                                         
plugins: cov-2.2.1, mock-0.10.1, xdist-1.14                                                                                                                                                                                                  
collected 363 items                                                                                                                                                                                                                          
. ...........................................................................................................................................................................................................................................
........................................................................................................s.......................                                                                                                             
=================== 362 passed, 1 skipped in 60.17 seconds ====================                                                                                                                                                              
[_test] C:\Users\Korijn\Miniconda3\conda-bld\test-tmp_dir>py.test --pyargs my_pkg --azure-storage-emulator                                                                                                                                     
usage: py.test-script.py [options] [file_or_dir] [file_or_dir] [...]                                                                                                                                                                         
py.test-script.py: error: unrecognized arguments: --azure-storage-emulator                                                                                                                                                                   
  inifile: None                                                                                                                                                                                                                              
  rootdir: C:\Users\Korijn\Miniconda3\conda-bld\test-tmp_dir                                                                                                                                                                                 
[_test] C:\Users\Korijn\Miniconda3\conda-bld\test-tmp_dir>
@nicoddemus nicoddemus added type: bug problem that needs to be addressed status: critical grave problem or usability issue that affects lots of users labels Jun 8, 2016
@nicoddemus
Copy link
Member

Thanks for the report!

@nicoddemus
Copy link
Member

nicoddemus commented Jun 8, 2016

By default pytest_addoption is not called on all conftest.py files, only at the root.

From pytest_addoption:

Note
This function should be implemented only in plugins or conftest.py files situated at the tests root directory due to how py.test discovers plugins during startup.

I think this logic would have to be extended to also consider conftest.py files at the root of packages when using the --pyargs option.

@Korijn
Copy link
Author

Korijn commented Jun 8, 2016

Right, I thought that included the root of packages. Guess not. :)

@nicoddemus nicoddemus changed the title py.test --pyargs my_pkg doesn't run pytest_addoption in my_pkg/conftest.py Consider conftest.py for --pyargs arguments (was: py.test --pyargs my_pkg doesn't run pytest_addoption in my_pkg/conftest.py) Jul 6, 2017
@nicoddemus nicoddemus added type: enhancement new feature or API change, should be merged into features branch and removed type: bug problem that needs to be addressed status: critical grave problem or usability issue that affects lots of users labels Jul 6, 2017
@nicoddemus nicoddemus changed the title Consider conftest.py for --pyargs arguments (was: py.test --pyargs my_pkg doesn't run pytest_addoption in my_pkg/conftest.py) Take --pyargs into account to find initial conftests (was: py.test --pyargs my_pkg doesn't run pytest_addoption in my_pkg/conftest.py) Jul 6, 2017
@jessecarbon
Copy link

Just FYI, I found this workaround which works - https://stackoverflow.com/a/43747114/2149641

However it basically makes your tests a callable module instead of using --pyargs.

@larsks
Copy link

larsks commented Jul 14, 2017

I showed up here to file exactly this bug but I see someone has beat me to it. I want to be able to execute tests from an installed location, where my "root" conftest.py is located in the directory corresponding to mypackage.tests. I was hoping that pytest --pyargs mypackage.tests would locate it correctly.

@bashtage
Copy link

bashtage commented Sep 5, 2018

Any progress on fixing this issue? It has recently cropped up in statsmodels.

@astrojuanlu
Copy link

Also bitten by this while running the statsmodels tests... @al1190

@larsks
Copy link

larsks commented Jul 3, 2019

Hey, hello 2017 larsks! I just into the same problem again and forgot that I had previously filed this bug. @nicoddemus, is this a particularly intractable problem, or just not an interesting one?

@nicoddemus
Copy link
Member

Hey @larsks,

Now that the dust has settled a bit with pytest 5.0, I will see if I can get a shot at this. Thanks for the ping. 👍

@vxgmichel
Copy link

vxgmichel commented Jan 19, 2020

Hi all! I just ran into the same issue, does anyone know a work around?

I couldn't find a way to run the tests within an installed package with custom options, apart from:

  • pointing pytest to the test files using the full path:
    $ pytest myenv/lib/python3.7/site-packages/mypackage/tests --custom-option foo
  • move pytest_addoption to a separate plugin module and provide it to pytest through -p:
    $ pytest -p mypackage.tests.plugin --pyargs mypackage.tests --custom-option foo
  • move pytest_addoption to a separate plugin module and register it using setuptools (it isn't great because this specific module gets loaded at every pytest invocations but it only makes sense for the tests in this particular package)

Oh, and thank you for your work :)

EDIT: there was already an answer mentioning a workaround, I should have paid more attention.

@bashtage
Copy link

I use a pattern where I define a test function in my projects __init__ that looks like:

def test(
    extra_args: Optional[Union[str, List[str]]] = None,
    exit: bool = True,
    append: bool = True,
):
    import sys

    try:
        import pytest
    except ImportError:
        raise ImportError("Need pytest to run tests")

    cmd = ["--tb=auto"]
    if extra_args:
        if not isinstance(extra_args, list):
            pytest_args = [extra_args]
        else:
            pytest_args = extra_args
        if append:
            cmd += pytest_args[:]
        else:
            cmd = pytest_args
    pkg = os.path.dirname(__file__)
    cmd = [pkg] + cmd
    print("running: pytest {}".format(" ".join(cmd)))
    status = pytest.main(cmd)
    if exit:
        sys.exit(status)

This lets me test from an install with

python -c "import pkg;pkg.test() or python -c "import pkg;pkg.test(['-r a','--skip-slow','-n auto',...)

@vxgmichel
Copy link

@bashtage Oh thanks, I didn't notice a similar workaround had already been shared earlier in the thread.

Sorry for the noise 😅

@max-sixty
Copy link

How feasible would it be to allow defining options in a conftest.py within the tests path?

That would mean we didn't need to put the conftest.py file in the root, and so it wouldn't get missed by python's packaging (I think).

@yarikoptic
Copy link

Now that the dust has settled a bit with pytest 5.0, I will see if I can get a shot at this. Thanks for the ping.

We stumbled upon this issue in dandi project as well. Please let us know if dust conditions improved. Cheers.

TimoRoth added a commit to TimoRoth/oggm that referenced this issue Sep 5, 2022
This is a workaround for pytest-dev/pytest#1596
Without this, testing an installed version of oggm is impossible.
TimoRoth added a commit to TimoRoth/oggm that referenced this issue Sep 5, 2022
This is a workaround for pytest-dev/pytest#1596
Without this, testing an installed version of oggm is impossible.
TimoRoth added a commit to TimoRoth/oggm that referenced this issue Sep 5, 2022
This is a workaround for pytest-dev/pytest#1596
Without this, testing an installed version of oggm is impossible.
TimoRoth added a commit to OGGM/OGGM-Docker that referenced this issue Sep 5, 2022
TimoRoth added a commit to TimoRoth/oggm that referenced this issue Sep 5, 2022
TimoRoth added a commit to OGGM/oggm that referenced this issue Sep 5, 2022
This is a workaround for pytest-dev/pytest#1596
Without this, testing an installed version of oggm is impossible.
TimoRoth added a commit to OGGM/oggm that referenced this issue Sep 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement new feature or API change, should be merged into features branch
Projects
None yet
Development

No branches or pull requests

9 participants