chore(ci): Test run examples as well as the regression suite #349
Conversation
|
In case it wasn't clear, the tests on this PR are expected to fail and reflect the fact that the 1.9.1 release and master branch are in fact broken. I believe this can be merged as-is and the fix for #348 should bring things back into the green. |
| @@ -24,6 +24,7 @@ install: | |||
| script: | |||
| - luacheck . | |||
| - lua run.lua tests --luacov | |||
| - lua run.lua examples | |||
There was a problem hiding this comment.
I think we should have a single line here;
lua run.lua --luacov
this will run everything, and add coverage reporting for everything as well.
There was a problem hiding this comment.
I deliberately ran it in two steps because then it is clear from the Travis output log what failed. The output isn't very clear when they are both run together. Also it allowed for coverage testing to be separate.
I did consider adding coverage reporting to the examples, but my experience with testing on other projects has led me to believe that its better to measure test coverage via the most specific unit testing possible, big monolithic examples that cover lots of ground can be very useful to end users and making sure they work can catch some kinds of issues but they don't do a great job at identifying specific expectations of each interface along the way. It's surprising what can slip though as 'working overall' but actually be broken inside. I would rather specific unit tests be the baseline for coverage reporting. Examples should work too (hence adding the to the CI run) but that shouldn't count as really testing the expectations on each function.
Related to but does not solve №348, see my comment there.