Syntest reports

[trishume]

This adds syntax tests to CI and makes it easier to check for regressions locally using them!

I was about to spend some time debugging syntax test regressions in #166 but then I realized I still hadn't gotten around to making regressions be caught automatically and be easy to digest.

I switched syntest to use getopts since syncat was already using it, and then added a --summary option which only prints failures and failure counts. I then added a make syntest command to diff against known issues and fail if there's a difference, and a make update-known-failures to easily update the counts on submodule bump PRs and things, or when we fix some.

I don't print the total number of assertions in summaries so that adding new succeeding tests doesn't cause CI failures.

cc @sharkdp

[sharkdp]

@trishume I just tested this, it works great. I was looking for something like this and ended up writing a small script myself for the output in #166.

Just one small thing: you should probably sort the output before building the diff because directory-walking is not deterministic:

--- testdata/known_syntest_failures.txt	2018-05-27 17:17:45.865872515 +0200
+++ -	2018-05-27 17:19:31.194805415 +0200
@@ -1,6 +1,6 @@
 loading syntax definitions from testdata/Packages
-FAILED testdata/Packages/ASP/syntax_test_asp.asp: 162
 FAILED testdata/Packages/C#/tests/syntax_test_Strings.cs: 38
 FAILED testdata/Packages/LaTeX/syntax_test_latex.tex: 1
 FAILED testdata/Packages/Makefile/syntax_test_makefile.mak: 7
+FAILED testdata/Packages/ASP/syntax_test_asp.asp: 162
 exiting with code 1

[trishume]

@sharkdp As soon as I submitted the CI run I guessed that might be a problem. Walkdir iterates in sorted order only on macOS apparently.

I fixed it by making syntest output in sorted order instead, which properly preserves messages at the start and end and also leads to consistent syntest output for other scripts.

[keith-hall]

looks great, nice work - I was also thinking something like this would be useful

[sharkdp]

I can rebase my branch in #166 on master after this is merged - to test if CI fails.

[robinst]

Great idea :)! Changes look good.