Failed test cases can be hard to find

[sjakobi]

In dhall we have a large testsuite. When it fails in CI, the failing test cases can be hard to find in the test log. Searching for "fail" is tedious, since "failure" is a common word in the test names and the test suite also contains expected test failures (using tasty-expected-failure). For an example, see this CI log: https://ci.appveyor.com/project/Gabriel439/dhall-haskell/builds/34014567

I'm wondering how this could be made more ergonomic.

• Could tasty simply collect the failing test cases at the end of the log?

• We could probably use --hide-successes, although it's sometimes also nice to see which test cases were run.

  How do we best enable --hide-successes by default though? (localOption (HideSuccesses True) doesn't appear to work #231 looks related.) And how could we then disable it on the CLI?

[UnkindPartition]

Could tasty simply collect the failing test cases at the end of the log?

You could certainly write a test reporter that would work this way if you prefer.

How do we best enable --hide-successes by default though? (#231 looks related.) And how could we then disable it on the CLI?

The easiest way is through the environment variable: either in the code (https://github.com/feuerbach/tasty#compile-time) or in your CI script. It's true that the CLI argument currently doesn't allow to un-set --hide-successes, only set it; the environment variable allows both.

[sjakobi]

Thanks for the tip to use env vars! :) I'm giving this a try in dhall-lang/dhall-haskell#1914.

I'll probably report back if this doesn't work out well for us.

[hdgarrood]

Could tasty simply collect the failing test cases at the end of the log?

We are also running into the exact same issue in the purescript compiler, and while I expect it is possible to write a reporter that reports results at the end, I feel like it is worth considering making this behaviour the default. For any project with more than, say, 100 tests I think the current behaviour is not a good default, because you’re having to do a lot of hunting to find the tests that failed. Moreover, I don’t think there’s much point in printing the full failure message while the test is running, because chances are it will have scrolled off the terminal before you get a chance to read it properly.

[UnkindPartition]

@hdgarrood did --hide-successes not work for you for some reason?

I think you're right that the current default works better for smaller test suites. I think that's exactly what the default should target: a beginner starting to compose their tests. Once you've written hundreds of tests, I think it's reasonable to spend some time to find the settings that maximize your efficiency.

[hdgarrood]

That’s understandable. There’s no problem with --hide-successes - it’s just that my preference would be to see the successes as well as the failures, but just have the failure messages shown at the end. But yeah, I’m sure I can manage writing a test reporter to handle this.

[sjakobi]

But yeah, I’m sure I can manage writing a test reporter to handle this.

If you end up implementing this, would you kindly ping me? I'd be interested in using it too! :)