Failure reporting should print out previous SECTIONs for data-driven testing

[jktjkt]

If a testcase differs only in the amount of input data, it would be nice to use SECTION to pass these bits into the object under test, like the following example. However, because the REQUIRE stanza is outside of the nested SECTIONs, error messages do not contain references to the failing sections.

TEST_CASE("check that Foo detect invalid input") {
  std::string input;
  SECTION("no data") {
  }
  SECTION("garbage") {
    input = "xxx";
  }
  SECTION("just two words") {
    input = "foo bar";
  }
  Foo foo;
  foo.setLogLevel(666);
  REQUIRE(!foo.parse(input));
}

This can be worked around by adding additional macros, but it quickly gets ugly when the test involves more than few statements.

TEST_CASE("check that Foo detect invalid input") {
  #define DOIT(...) \
    Foo foo;\
    foo.setLogLevel(666); \
    REQUIRE(!foo.parse(__VA_ARGS__));

  SECTION("garbage") {
    DOIT("xxx")
  }
  SECTION("just two words") {
    DOIT("foo bar")
  }
}

I would like Catch to keep track of those SECTIONs which have been already closed, and to add their names into the test output if expectations fail.

[jruffin]

+1. Many of my tests are either made of common code and REQUIREs that are run against a bunch of input data with corresponding expected output data, or have common post-condition checks that might only fail in a particular run but be OK in the others. On top of that, I use a mocking framework that checks whether everything was called as expected during the destruction of its mock objects at the end...

So far, I have been working around the issue by packing most common code and REQUIREs into fixture methods and calling those in each SECTION that needs it and using a listener to trace and keep the deepest section stack of a run and print it when the mocking framework fails, but neither are nice nor very "catchy" in my opinion.

I have managed to modify Catch to fix the issue with both reporters where this seems sensible (console and JUnit), see #736.

[philsquared]

For the record I agree with this request and it's on my queue

[textshell]

Any news on this? This would be quite useful to have.

[horenmar]

Since generators have been added to Catch, you should either use them for data driven testing, or you make your own reporter.