Option to suppress exit code 5 (no tests run)

[jakirkham]

For the most part, I think the addition of PR ( #817 ) to have a special exit code to denote no tests run was a good idea. However when I start out a project, I may not have any tests as there is no real code. Still I would like to get everything up and running (i.e. CIs). It would be nice in these cases like this to have an option to suppress this behavior and have exit code 0 returned instead.

[The-Compiler]

FWIW you can do something like

def test_placeholder():
    pass

[RonnyPfannschmidt]

im closing this, as the very reason for having and not suppressing this error code is to learn of broken/gone bad ci setups, its just as easy to simply put a first green test you know you will fill out later than to opt out of something you may forget about

[nicoddemus]

Thanks for the suggestion @jakirkham, but I agree with @RonnyPfannschmidt that we shouldn't add yet another option to pytest to handle this case specially because an alternative solution is trivial and easy to do, while the possible drawbacks can be a pain (forgetting later that the option is set like @RonnyPfannschmidt mentioned).

[mkleina]

But what in case the -m (markers) option is present and provided marker is not found? I am working on the case where multiple test suites are run in particular order using tox.

[testenv:all]
commands =
    pytest testsuite1/ {posargs}
    pytest testsuite2/ {posargs}
    pytest testsuite3/ {posargs}

where {posargs} are arguments passed to tox. When I want to preserve order, but filter out selected marker, for example:

tox -e all -- -m smoketest

pytest command fails when no markers found and no further tests are started because of exit code 5.

Any idea how to substitute this use case?

[blueyed]

@mkleina
You can wrap the command:
sh -c 'pytest testsuite1/ {posargs}; ret=$?; [ $ret = 5 ] && exit 0 || exit $ret'

I am using the following p wrapper myself:

#!/bin/sh
pytest "$@"
ret=$?
if [ "$ret" = 5 ]; then
  echo "No tests collected.  Exiting with 0 (instead of 5)."
  exit 0
fi
exit "$ret"

[RonnyPfannschmidt]

we ought to have a pytest plugin to do that for people that truely need it (since a shell wrapper is pretty anti-portable

[nicoddemus]

Agree with the plugin idea, should be simple to do.

[ChiKenNeg]

Any news?

[RonnyPfannschmidt]

@ChiKenNeg nobody worked on a plugin for this as far as i know, if you'd like to have one, i strongly suggest making one

[yashtodi94]

A little hack. But something on these lines should work fine. (Pytest v4.0.2)

Add this to your conftest.py:

def pytest_sessionfinish(session, exitstatus):
    if exitstatus == 5:
        session.exitstatus = 10 # Any arbitrary custom status you want to return```

[Tadaboody]

@yashtodi94 would you be willing to create a plugin with that solution?

[yashtodi94]

Sure...I'll try submitting a PR by next week

[Tadaboody]

@yashtodi94 I think implementing the plugin in a different repository and linking it here is the norm, because it isn't changing pytest directly.
There's a cookie cutter template for making one that's easy to submit to pytest-dev
https://github.com/pytest-dev/cookiecutter-pytest-plugin
If you need any help let me know!

[yashtodi94]

@yashtodi94 I think implementing the plugin in a different repository and linking it here is the norm, because it isn't changing pytest directly.
There's a cookie cutter template for making one that's easy to submit to pytest-dev
https://github.com/pytest-dev/cookiecutter-pytest-plugin
If you need any help let me know!

Yes, I discovered that after posting my previous comment. Thanks for the help :)

[yashtodi94]

@Tadaboody @RonnyPfannschmidt

First attempt at building a plugin: https://pypi.org/project/pytest-custom-exit-code/

[nicoddemus]

Awsome @yashtodi94, thanks for sharing! 🙃

[dmcnulla]

https://pypi.org/project/pytest-custom-exit-code/

I made a variation of that because sometimes the re-run had zero tests and exited with code 5. Thanks for posting it.

export PYTHONPATH=.:./framework:./integration_tests
export CONFIG=test

python -m integration_tests.purge_queues
pytest -c int_tests.ini --disable-pytest-warnings
ret=$?
if [ "$ret" = 1 ]; then
  python -m integration_tests.purge_queues
  pytest --last-failed --last-failed-no-failures none -c int_tests.ini --disable-pytest-warnings
else
  exit $ret
fi

[joelazar]

Also, here is an example of how to have a workaround for this in a Makefile:

.PHONY: test
test:
	sh -c 'poetry run pytest **.py || ([ $$? = 5 ] && exit 0 || exit $$?)'

[paigeadelethompson]

I guess this works for me, would have been nice if pytest just had an option for it though python -c 'import subprocess, sys; (lambda p: p.returncode == 5 and sys.exit(0) or sys.exit(p.returncode) )(subprocess.run("pytest"))' && echo "ok"

[ardunster]

I just ran into this issue today attempting to amend a commit in which all I did was remove a few print() calls I'd put in for debugging. The commit hook refused to actually rerun tests because nothing materially changed (I assume), even though if ran manually with the same command (pipenv run pytset) I get the output listing pass/skip quantities rather than an error. I didn't want to actually change anything else in that commit, so I ended up using --no-verify, but this doesn't seem like desired behavior.

[SonOfLilit]

I modified @yashtodi94 's solution a bit to only swallow the failure if I passed the -k argument to filter the tests, so I still get a failure if something bad happened to test collection (you may want to check different argument(s) instead).

NO_TESTS_COLLECTED = 5
SUCCESS = 0

def pytest_sessionfinish(session, exitstatus):
    if exitstatus == NO_TESTS_COLLECTED and session.config.getoption("-k"):
        session.exitstatus = SUCCESS```

[Marc--Olivier]

For information, I ran into this issue today while running pytest --cov=. on a directory that had no test. Still, I wanted to collect the code coverage. I personally did not find satisfactory to have to handle the error code 5 or to install the pytest-custom-exit-code in multiple places.
Why does pytest return an error code when there is no test whereas unittest does not?

[nicoddemus]

@Marc--Olivier

Why does pytest return an error code when there is no test whereas unittest does not?

See #812 and #500 for the discussion that led the inclusion of this feature.