Should we continue to allow for multiple examples?

[sshine]

Since Exercism v3 necessitates a different directory structure (as evidenced in exercism/v3#200), I thought that this would be a good time to review which decisions related to directory structure were made in the past and if their reasons still apply.

It was decided in #395 (and implemented in #396) that we should support multiple examples per exercise and extended in #397 (and implemented in #477) with the possibility of negative examples. One consequence of these discussions is that multiple examples are provided in a directory structure examples/{success,fail}-{foo,bar}/{package.yaml,src/<SLUG>.hs}. The merits of this particular directory structure for handling multiple files won't be elaborated here, but can be read in the referenced issues and pull requests.

Re-iterating some of the advantages and disadvantages of multiple examples:

• 👍 we can test that multiple signatures are accepted by the test suite
• 👍 we know that some recommended solution is possible
• 👎 running the test suite takes longer (18/101 ≈ 20% of exercises have 2 examples)

Re-iterating advantages and disadvantages of negative examples:

• 👍 we can test the quality of our test suite, to see it catches various cases we want it to catch
• 👎 examples that intentionally fail currently clutter the Travis CI logs (Make fail-* examples not show up in red in Travis logs #783)

We have three negative examples currently: the test suites of accumulate and strain are verified to fail when solutions are not lazy enough, and test suite of of robot-name is verified to fail when solutions are not collision-aversive.

I'm okay with leaving things as they are, since both of these features were at one point favored by both @petertseng and @rbasso. I'm also okay with simplifying things and have just one (positive) example at the cost of quality. This is slightly more maintainable and provides a slightly lower bar for new contributors.

This issue was created as a reaction to exercism/v3#200 in which it was briefly assumed that we distribute only a single example solution in a file called Example.hs (as is the norm on many other tracks), when in fact our current setup is more elaborate. Rather than to insist on continuing to do so, this issue can be an opening for having a discussion of the subject if anyone is inclined to do so. If not, this issue can be closed briefly and be resumed at any point later in time.

[petertseng]

In which I explain how I would evaluate this decision (what assurances does the project want to make about the product it provides?)

In e.g. an open source project that provides a library, they test the library to make sure that e.g. it compiles, or that it can be used in some specified way to get some specified result. Since one of the things Exercism provides is the tests for a problem (along with its specification and the stub file), then we ask what properties we want to ensure are true about the tests.

For example, "it is possible to write a solution that passes these tests" seems to be something all tracks agree that we want to test, and providing one solution is one way to achieve that goal.

So, the current structure implied that this track additionally wanted to test "it is possible to write a solution with this other type signature" and "if a solution has X incorrect behaviour, the tests reject that solution".

So, I think the first question to ask is what exactly this track wants to ensure is true about the tests it provides. Then it is possible to think of ways to ensure those things.

[sshine]

I'm closing this issue as its main purpose is to highlight that this discussion was being made in exercism/v3#200. Since nobody is interested in having it, this issue can be opened again if the discussion should once again arise.