Run test case N times with N different inputs

[capsocrates]

Sometimes I want to repeat a test a number of times, but with one (or more) values changed each time. If I try to wrap a TEST_CASE inside a for loop, the for loop just executes on its own before the TEST_CASE executes.

E.g.

//desired behavior

SCENARIO("... some scenario...", "") {
    //do setup here
    WHEN("We do something 10 times") {
        std::vector<mock_class> objs;
        for(const auto counter : {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}) {
            WHEN("We do something") {
                static_member_of_mock_class = counter;
                //do stuff with mock class that modifies objs
            }
        }
    }
}

//how to actually implement desired behavior with Catch

SCENARIO("... some scenario...", "") {
    WHEN("We do something 10 times") {
        for(const auto counter : {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}) {
            std::vector<mock_class> objs;
            static_member_of_mock_class = counter;
            //do stuff with mock class that modifies objs
        }
    }
}

I recognize that a for loop might not jive with the implementation of the TEST_CASEs, but it would be nice to have some built-in feature that allows Catch to run the same test over again just with different inputs/setups.

As an extension to that, it would be nice to have a feature that allowed running a TEST_CASE repeatedly with different combinations of inputs, for instance, given inputs

const auto counters = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
const auto letters = {'a', 'b', 'c', 'd', 'e'};
const auto strings = {"abra", "cadabra"};

COMBINE(counters, letters, strings) {
    std::cout << counter << letter << string << ", ";
    /*
     * outputs 0aabra, 1aabra 2aabra ... 0babra ... 0cabra... 0acadabra ... etc.
     * */
}

[capsocrates]

My solution for now is to wrap the stuff I'm doing in a lambda, with only assertion and logging macros inside (no TEST_CASE macros) and then call that lambda repeatedly in one or more TEST_CASE blocks with different parameters. It is still awkward (especially since the setup has to be performed, one way or another, inside the lambda) but it works.

[philsquared]

This feature is called Generators and is something I've had on a the back-burner for a while. I've recently been working on it again a bit more - but it's still a little way off being ready.
It's also the basis for a push into Property-Based-Testing that I'm working on.

[nabijaczleweli]

Any update on this, @philsquared?

[mlimber]

I am also interested in this. Sometimes this is called data-driven testing (DDT), e.g., in Qt's test framework, or Value-parameterized testing in Google Test.

[roversch]

👍 This would be especially useful for testing numerical code, like the one I'm working on.

For example, I have slightly different data for each test run. I could write a loop around one 'REQUIRE', but for 'SECTION's or 'TEST_CASE's this does not work apparently.

[philsquared]

A little update on this:

1. Although I did get some basic generator stuff working in Catch as-is I kept hitting things that would be either much simpler with C++11, or, in some cases, only possible with C++11 - or even 14! (e.g. I really want it to work with Ranges). So I've decided to push it to Catch2, which will be C++11 (+) only.
2. I thought I'd have a lot more time to work on Catch given my recent move to JetBrains, but so far it's worked out that I've had less time! I'm still confident that that situation will reverse in the near future, but for now it's set me back more than I expected.

So, in summary, it's all still coming - but still not for a while yet, I'm afraid.
I'm not committing to anything but I really would like to have something concrete early next year.

[simonvpe]

Second this. What is the status of Catch2 @philsquared ?

[philsquared]

At time of writing Catch2 is on hold until the Catch "classic" backlog is under control.
We're well on our way towards that now - but a way to go yet.
See: http://www.levelofindirection.com/journal/2017/1/19/catch-up.html

[philsquared]

I believe I have answered this query now (even if not entirely satisfactorily) - and objection to me closing it? (@capsocrates, if you're still watching?)

[mlimber]

I'm ok with closing it, but I'd appreciate a ping here when Catch2 hits the streets with this feature.

[philsquared]

I'm closing this in favour of #850, so be sure to watch that issue for notifications.

[stefanhaller]

I realize this is closed, but I'd like to reply to one particular comment made above by @roversch who said it's not possible to put a SECTION inside a for loop.

We're doing exactly this, and seems to work well for us:

// "production" code:
int timesTwo(int x)
{
  return 2 * x;
}

// test code:
TEST_CASE("timesTwo")
{
  for (const auto [input, expectedOutput] : {
         std::make_tuple(1, 2),
         std::make_tuple(2, 4),
         std::make_tuple(3, 6),
         std::make_tuple(4, 8),
       })
  {
    SECTION("timesTwo with " + std::to_string(input))
    {
      CHECK(timesTwo(input) == expectedOutput);
    }
  }
}

This works so well that I'm actually not sure what the excitement about data-driven tests is about. What am I missing?

(The only slightly annoying thing here is having to manually construct the section names. In this particular example it's easy enough, but often we loop over enum constants, and there's no elegant solution to stringify these.)

[roversch]

Thanks @stefanhaller ! I'll try it out again. It's been a while, so I forgot what exactly went wrong.

[andyborrell]

I think it's worth commenting here that if you do use a SECTION inside a loop, make sure it has a unique name for each iteration of the loop. If you don't use unique names then some tests will be ignored and you might miss failures.