Tests that accompany code run and produce output at default

[martinmoene]

In the case that unit tests accompany code, user may not expect, or like that at default

• the tests are run,
• the test run produces output (even) if all tests pass.

If tests take a negligible amount of time, running the tests at default may be fine, provided they produce no output on success.

If tests take a non-negligible amount of time, I can imagine people like to skip the tests at default.

Provided running the test at default is retained, it would be nice if the default non-running case could be achieved through a user-supplied main() in combination with a new command line option --run, or --unittest (the flag used by D).

int main(int argc, char** argv) {
    doctest::Context context(argc, argv); // initialize

    // overrides
    context.addFilter("no-run", !context.flag("run")); // control --no-run from --run
    ...
}

Note that currently configuring the context can only occur after processing the commandline options in Context's constructor, hence the comment mentioning overrides. The lack of being able to set defaults before commandline processing leads to the convoluted example above.

Supporting configuring defaults, e.g.:

int main(int argc, char** argv) {
    doctest::Context context;
    context.addFilter("no-run", true);  // default

    // --run (--unittest) on the commandline now overrides the --no-run default
    context(argc, argv);  // or context.apply(argc, argv);, or ...
    ...
}

[onqtam]

Indeed there should be an option to omit all output in the case of only success - this will be added.

Also it really is important for the user to be able to set defaults and not just overrides.

perhaps I should eliminate the --no-overrides option altogether? and allow the user to construct a Context without passing argc/argv, then to set some options and then to apply argc/argv by calling context.applyCommandLine(argc, argv);?

And in this way the user will still be able to set real and final overrides after the command line has been applied - again with setOption - but there will be no way to override them.

This seems like the cleanest solution - keeping the --no-overrides option will force the addition of things like setDefaultOption() which I don't want - and I'm almost sure removing the --no-overrides option is the right way to go.

What do you think about this?

[martinmoene]

Conforming to Catch's Amending the config as you describe above seems like a good idea.