No way to run an individual section

[wichert]

When debugging I often want to run a specific test method only. This is currently not possible, so as a workaround I find myself often copying a test method to a new temporary test case. That works, but isn't very pleasant.

An option to allow tagging of test methods (and suites if #320 is implemented) in combination with a new option to the test runner to filter methods (-m <filter> perhaps?) would be much appreciated.

[philsquared]

Hi @wichert,

I'm not quite sure what you mean by, "run a specific test method only". First what do you mean by "test method"? Do you mean a TEST_CASE, TEST_CASE_METHOD or something else?

In both of those cases you can run just that test case by specifying it's name on the command line - so I'm presuming I've missed some nuance to what you're talking about (especially with the talk of temporary test cases).

[wichert]

I guess I'm too used to writing Python tests, which makes me use the wrong terminology when dealing with Catch :(

What I mean is: I want to be able to run an individual SECTION. Typically my tests are structured like this:

TEST_CASE("Tokenizer", "[search]") {
    SECTION("Constructors") {
       SECTION("Default constructor") { }
       SECTION("Copy constructor") { }
    }

    SECTION("Unicode handling") { }
}

When I have a bug in the copy constructor I want to be able to run only the Copy constructor section of the Tokenizer test case. Since that is not possible I currently end up copying code around so I get this:

TEST_CASE("XXX Copy constructor", "[DEBUG]") {
  // Copy of Tokenizer->Copy constructor
}

TEST_CASE("Tokenizer", "[search]") {
    SECTION("Constructors") {
       SECTION("Default constructor") { }
       SECTION("Copy constructor") { 
            // Old/original version of what is now in "XXX Copy constructor"
        }
    }

    SECTION("Unicode handling") { }
}

I then just run the tests for the DEBUG tag. Once I'm done I copy the code back to the original section and delete the temporary test case.

[philsquared]

Ah! I wondered if you meant sections.
Yes I keep thinking about how to make this possible.
There are ways to make it work if you uniquely specify the section (by one of several possible means) - which would help in your situation.
AFAICS it's possible in the more general case where you refer to a section name using wildcards and/ or tags.
That's one of the things that's been holding me back (I don't like the inconsistency) - but being able to do the first case is clearly very useful. I'll give it renewed consideration - thanks for bringing it up again.

[vadz]

It would indeed be very useful to be able to use "Tokenizer/Constructors/Copy constructor" on command line, i.e. select the section to run by name.

[vpicaver]

A fix to this issue would be very nice. Some of my test cases take a long time to run, and it would be useful to run a particular section in a test case.

[philsquared]

@vpicaver - you might be better off splitting longer runner tests (which are presumably not unit tests) into separate TEST_CASEs, or possibly TEST_CASE_METHODs (which are more "traditional" x-unit style fixtures).

[vpicaver]

The SECTION helps me set up the problem and re-use code. I'll have to look into TEST_CASE_METHOD

[philsquared]

Yeah, that's what SECTIONs are for and I would usually recommend them - but given your usage and current constraints TEST_CASE_METHOD may be better suited to your needs (you can create a class that use as a fixture that holds the common state and setup code - just like traditional x-unit test runners).

[vpicaver]

Okay cool, makes sense. Catch has saved me a bunch time, btw.

[horenmar]

This is actually in master now, albeit a bit limited, so I am going to close this.

[onqtam]

Since I'm working on the same issue on doctest I'd like to note that the following:

TEST_CASE( "Test" ) {
    cout << "root" << endl;
    SECTION( "sa" ) {
        cout << "sa" << endl;
        SECTION( "sb" ) {
            cout << "sb" << endl;
        }
        SECTION( "sc" ) {
            cout << "sc" << endl;
        }
    }
    SECTION( "sd" ) {
        cout << "sd" << endl;
    }
}

when ran like this: a -c sa -c sb will output the following:

root
sa
sb
root
sa
root

So it enters the test case 3 times - when in reality it should only once.

Not sure this is worth fixing though...

[philsquared]

Hi @onqtam,
Have you implemented -c in doctest too, then?
The reason for this behaviour is that all -c is doing is acting as a filter. It's still attempting to discover all the sections leading up to the first "miss". It could be changed to realise when it has executed all (usually the only) leaf sections within the filter and stop looking for me. Perhaps it should do that - thanks for the observation.

[onqtam]

@philsquared just did - in the dev branch, but mine is a bit different - see the associated issue

Anyway I first noticed the behavior I reported here while implementing the filtering in doctest - then I moved the filtering up a bit inside of my Subcase constructor (analogous to Section in Catch) and I no longer re-enter the test case if all other subcases that were not traversed didn't match the filtering criteria