Means for writing test cases more concisely

[tsnorri]

Description

I have a number of scenarios that run basically the same test with different inputs and expected outputs. I would like to write the tests as concisely as possible, i.e. minimise the amount of boilerplate per test. This can be achieved by e.g. using a custom file format for the inputs and outputs and describing the tests in such a file, but this leads to somewhat convoluted implementation, since currently there is no way to maintain state between scenarios or test cases (other than using global variables). Additionally, some sets of inputs need common setup, the results of which I thought would be nice to reuse, since the tests are not meant to evaluate that part of the code. I use the 2.x branch, so there may be some new functionality for this purpose of which I am not aware.

I am not sure what would be the best solution to address this or if it is even worthwhile. A straightforward approach would be to have test suites (as described in an earlier ticket) that would be able to maintain state between test cases or scenarios. Another approach would be describing the scenarios and the required steps separately like Cucumber does. The former solution could have also some other uses cases but would require the library user to implement reading the input, while the latter would presumably require more changes in Catch2.

Additional context

So far I have tried two different options.

• Writing a separate scenario for each test, specifying the inputs as part of the code, and implementing the test as a separate function. To make the code readable, this required some 15 to 20 lines of code per scenario including some boilerplate. I implemented the “common” parts with functions that have a static variable, which (clearly) is not the perfect solution. Having the separate function also has the downside that WHEN and THEN are placed outside the scenario, which makes the code slightly more difficult to follow.
• Writing code that reads the input from a separate file. I replaced each scenario with a section, since the scenario or test case has to be responsible for processing the input. The obvious downside is that the scenario is not actually a scenario. Additionally, running just one section (for e.g. debugging purposes) is not possible with the command line interface but needs to be done by modifying the input file instead.

[jbadwaik]

I do not see what is the state you need to maintain between the test cases given that you are running the same test case multiple times but with different input/output. You mention that Sections can work for you, but for the fact that you cannot run only one of the section at a time.

In that situation, It might be possible to factor out the setup/test code into a function/class. And then have each test case use that function/class to run the test case for that particular set of input and output files.

#include <testmylib/test_checker.hpp>

TEST_CASE("test case 01")
{
  auto input_file = std::string("input01");
  auto output_file = std::string("output01");
  REQUIRE(test_checker(input_file,output_file));
}

TEST_CASE("test case 02")
{
  auto input_file = std::string("input02");
  auto output_file = std::string("output02");
  REQUIRE(test_checker(input_file,output_file));
}

[tsnorri]

Due to the fact that the inputs are quite short, I initially thought about keeping all the inputs in the same file, hence the need for maintaining state. Having separate input files would certainly be an easy solution.

[horenmar]

So you want to have a file like

inputs:
  - name: foo
    key1: value1
    key2: value2
  - name: bar
    key1: value1
    key2: value2
 ...

and then a test case that goes over each input in a row? With the shared state being your position in the input file?

That is exactly what the generators are for. (or just a for loop on top of the test case, advantage of using generators is better reusability and integration with the code)

[tsnorri]

Thanks for the pointer! I’ll look into using generators this way.