[CSS] Missing tokens on invalid input

[ris58h]

No description provided.

[ris58h]

I would like to add a test but it seems like antlr4test-maven-plugin can't check output tokens, only trees.

I've created corresponding issue antlr/antlr4test-maven-plugin#88

[kaby76]

antlr4test-maven-plugin isn't used in CI testing because it does not test across the other seven targets besides Java.

But, this can be added to CI testing, with not much effort.

• Update the templates, specifically */test.{sh,ps1}. Add a -tokens option. All drivers have this option, but the code does not output to a .tokens file. In addition, some of the targets don't have a Reset() API in the runtime. (I keep telling Antlr folks that the runtimes are incomplete, but no one listens.)
• Update the driver programs to output to a .tokens file. The code is there for outputting to .tree, and only needs a similar adjustment for tokens. "Diffing" is already performed, so there's nothing more to do to add the feature.

I'll create an issue to track the "to do" task, and implement it next week.

[teverett]

@ris58h thanks!