Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Pytest based test executor #354

Open
mtreinish opened this issue Oct 19, 2023 · 0 comments · May be fixed by #355
Open

Add Pytest based test executor #354

mtreinish opened this issue Oct 19, 2023 · 0 comments · May be fixed by #355

Comments

@mtreinish
Copy link
Owner

In the python testing world the stdlib untittest testing library and runner are not the defacto standard for running tests, most of the python world uses pytest. In recent years the testing cabal stdlib unittest extension stack that we've built stestr around hasn't been super actively maintained and new features in stdlib unittest aren't supported at all. To enable broader adoption of stestr and also enable using more advanced testing features we should look into adding a mode of operation that leverages pytest as a test runner instead of our unittest based runner. We should still continue to maintain, support, and default to the current test execution model as that's what our users expect. But having a way to configure stestr to use pytest for execution instead would be a great feature. This would enable users to get all the advanced parallelism and scheduling features that stestr provides and the more sophisticated test library features that pytest offers.

The only requirement in practice to make this happen in practice is to have pytest emit a real time subunit stream as it executes tests. If we can teach pytest how to do this then in theory it's just a drop in replacement inside of stestr to call pytest instead of python -m stestr.subunit_runner.run (which is part of stestr's heritage being forked from testr that supports working like a test runner runner).

The open questions for me are on how we package this (do we make pytest a hard requirement) and include a plugin to enable subunit output natively, or do we leave the subunit plugin as an external library and make pytest an optional dependency?

Additional context

@jogo started an implementation of this last year which relied on a fork of an existing pytest-subunit plugin for adding subunit support to pytest. We can take over where he left off and figure out how to best integrate this into stestr.

https://github.com/jogo/pytest-subunit
https://github.com/jogo/stestr/tree/pytest

mtreinish added a commit that referenced this issue Nov 5, 2023
This commit adds a new runner mode to stestr that enables users to run
test suites using pytest instead of unittest. This is an opt-in feature
as for compatibility reasons we'll always default to using unittest, as
all existing stestr users have only been leveraging unittest to run
tests. A pytest plugin that adds a subunit output mode is now bundled
with stestr. When the user specifies running tests with pytest it calls
out to pytest just as stestr does with the integrated unittest extension
today and sets the appropriate flags to enable subunit output.

To facilitate this new feature pytest is added to the stestr
requirements list. I debated making it an optional dependency, but to
make it easier for the significantly larger pytest user base (it's
downloaded ~900x more per month) it seemed simpler to just make it a
hard requirement. But I'm still not 100% on this decision so if there
is sufficient pushback we can start it out as an optional dependency.

Co-Authored-By: Joe Gordon <[email protected]>

Closes: #354
@mtreinish mtreinish linked a pull request Nov 5, 2023 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant