Test setup / tear-down / fixtures?

[jamesharr]

I'm really liking the looks of Hurl and it might replace some internal testing tools we have.

One thing we use a lot in our testing is setup/tear-down using the API before tests. For example, we'll create several objects in the API, perform several reads/asserts to make sure the correct thing happened (all using the same API), then perform tear-down.

We use RobotFramework to do this today, and honestly, we're not using normal Setup/Teardown calls because the setup/tear-down wind up getting used over multiple tests, this setup/tear-down takes some time, and Robot doesn't (seem) to have fixture (setup/tear-down) scoping quite the way that something like PyTest has. What we end up doing is having a "Test" at the beginning of a file that does setup, and one at the end that does tear-down and we lean on tests getting run in series for this to work. Sometimes one test early on might cause a series of subsequent tests to fail, and we just kind of accept that.

I totally understand if this is not on the roadmap and/or a targetted use-case for this tool.

[jcamiel]

Hi @jamesharr

I maybe misunderstand your usage but could you use two Hurl files as a setup/teardown files?
For instance:

$ hurl --test setup.hurl test*.hurl teardown.hurl

or even:

$ hurl setup.hurl
$ hurl --test *.hurl
$ hurl teardown.hurl

In this flow of execution, setup.hurl will be run as a first step and teardown.hurl will be play last. One thing to note is that, in incoming Hurl 5.0.0, tests will be run in parallel by default. In this case:

$ hurl setup.hurl
$ hurl --test *.hurl
$ hurl teardown.hurl

Will not have issues as setup and teardown will be run on first and last order.

But with:

$ hurl --test setup.hurl test*.hurl teardown.hurl

setup.hurl and teardown.hurl could be run in parallel in 5.0.0. You can for a sequential execution with --jobs 1 forcing the tests to be executed one by one.

$ hurl --jobs 1 --test setup.hurl test*.hurl teardown.hurl

[jamesharr]

I suppose this would work for some of the simple cases, but as the test suite starts to grow, we'd have to rely more and more on sequential, linear execution. And admittedly we do some of this today.

We might have setup+tear-down that needs to happen for the entire test suite, setup+teardown that needs to happen for a specific group of tests, and setup+teardown that needs to happen for each test. In the future if you want to support parametrized tests, what if setup/teardown needs to happen on parametrized tests?

We can still do this with sequential execution, but it starts to become cumbersome to manage, we'd need to organize our tests something like this:

000-setup.hurl
suite1/000-setup.hurl
suite1/test1.hurl
suite1/test2.hurl
suite1/test3.hurl
suite1/zzz-teardown.hurl
suite2/000-setup.hurl
suite2/test1.hurl
suite2/test2.hurl
suite2/test3.hurl
suite2/zzz-teardown.hurl
zzz-teardown.hurl

and then execute with:

hurl --jobs 1 --test $(find . -name '*.hurl')

but what if I'm doing dev work and want to execute just suite2 but still needed that global setup/tear-down as well? what about just suite2/test2, but I still need the global setup+teardown and the suite setup/teardown? that starts to get more and more awkward to execute, but to your point it's still doable.

I think your serial execution method would work, but over the long-term of your project, I think it might be worth thinking about test setup/tear-down as a first-class concept if you intend to grow hurl in that sort of a direction. Admittedly, we're doing more complex behavior-style tests and not unit tests, which need a lot more setup/tear-down work.

Pytest handles setup/teardown in a novel way with fixtures and scoping, but honestly, it feels "nifty and useful, but awkward to understand and troubleshoot".