switch to pytest

[flying-sheep]

pytest makes writing tests more fun.

of course targeting unittests at first was an obvious choice, but we should think about if we can switch to something nicer to use.

pytest makes you write top-level functions and assert statements. you don’t need to derive from classes (saving a pointless indentation level) or search for a assert* method.

IMHO for something like tests, fun is crucial. and with unittest, i quickly lose interest and self-discipline when trying to out again if i want self.assertSomething or rather self.assertOMGDoesThisNameNeverEnd. also, as i already see in the IRkernel tests, we now have millions of methods on one class instead of splitting the tests nicely into files.

the data-driven tests showcased in test_ipykernel.py as well as helper functions can be realized via fixtures.

e.g. we could set up a fixture that retrieves a cached kernel instance and runs its data-driven tests on instantiation:

kernel_data = {
    'kernel_name': 'python3',
    'code_hello_world': 'print('hello, world')',
    ...
}

@pytest.fixture(scope='module')
def kernel():
    return jkt.kernel_tester(kernel_data)

and then proceed to test:

def test_thing(kernel):
    reply, output_msgs = kernel.execute(code='import sys; print("oops", file=sys.stderr)')
    assert output_msgs[0]['msg_type'] == 'stream'
    assert output_msgs[0]['content']['name'] == 'stderr'
    assert output_msgs[0]['content']['text'] == 'oops\n'

[takluyver]

Nothing stops JKT users from using py.test as their test runner, and from writing additional tests with py.test. I'm not sold on making JKT itself depend on one specific test framework, because people have different preferences, but we can provide more convenience features to make writing additional tests easier.

[flying-sheep]

it occurred to me that we could add pytest as optional dependency and provide one or more functions that target it.

e.g. the kernel_tester function i drafted above could create the jkt.KernelTests from the data, run it, and then cache and return an object with the execute_helper, km, and kc fields.

[jankatins]

As a standalone package (or as a starting point for c&p work and integration into this package): there is a great cookiecutter package for pytest plugins which per default builds/exports a new fixture. https://github.com/pytest-dev/cookiecutter-pytest-plugin.

[flying-sheep]

hmm, what are plugins good for? can’t we simply define and use a fixture?

or a module-level mark via pytestmark = pytest.mark.jupyter_kernel(data)

[jankatins]

It builds a package which adds all the right things so that you can run py.test and have the fixtures automatically available (via entrypoints).

Just "defining a fixture" would mean that you have to do that in all projects which use it (or at least import something which is then used as fixture)

[flying-sheep]

or at least import something which is then used as fixture

seems easy enough to me, but please tell me more about entry points 😄

[flying-sheep]

funnily this popped up on reddit right now: https://www.reddit.com/r/Python/comments/50nqlp/is_nose_still_relevant_how_about_unittest/

[mariusvniekerk]

So right now you cannot use JKT with pytest due to pytest-dev/pytest#1367