Issue 5 - Move HIE Tests and convert to Tasty

[jeffwindsor]

Tests are converted to Tasty format. The tests compile and run from console, but most fail / timeout.
I want to check in to see if this is going the right direction, and maybe get some help wiring up the CI / Runner / Config.

[jeffwindsor]

helps resolve #5

[alanz]

I'm a bit embarrased to say this, but we decided to stick with hspec, because it has better rerun options.

[Bodigrim]

What's missing in tasty-rerun?

[alanz]

@Bodigrim possibly just my inability to look up documentation.

But I do know that if an hspec test fails, it lists a way to rerun just that test, with the appropriate command line invocation, as part of the test result. I find this invaluable.

[alanz]

And, full disclosure, I have a personal bias, as all my other OSS projects use hspec.

[Bodigrim]

tasty-rerun provides --rerun and --rerun-all-on-success options, pretty much similar to hspec.

Full disclosure: I also have a personal bias, being a maintainer of tasty-rerun :) I do not mean to interfere with your project's decisions, but I'm genuinely interested what can be improved. If something is missing from tasty-rerun, I'll be happy to fix it.

[jeffwindsor]

closing to move to hspec

[fendor]

More discussion needed. As I look at this PR, I dont have anything to complain about on the first sight!

@jeffwindsor Please dont undo your work, yet.

[alanz]

@Bodigrim glad to have you take an interest. The main thing I care about is if a test fails, I get something like

  To rerun use: --match "/Plugin/composes code actions/provides 3.8 code actions/"

And then I can do

stack --stack-yaml=stack.yaml test haskell-language-server:func-test --test-arguments "--match \"/Plugin/composes code actions/provides 3.8 code actions/\""

To run just that test

[alanz]

One other thing that would be useful, is junit.xml output. as per https://github.com/haskell/haskell-ide-engine/blob/master/test/utils/TestUtils.hs#L306

[Bodigrim]

One other thing that would be useful, is junit.xml output

http://hackage.haskell.org/package/tasty-ant-xml works well for me.

[alanz]

To summarize the discussion on IRC, we will continue with tasty. For the avoidance of future doubt. Sorry to have caused so much trauma for all.

[fendor]

Looking good so far. Thank you for this huge contribution!
What I think is missing:

• Marking most of the test-cases as broken
• Add junit-xml runner

[jeffwindsor]

@fendor:

• made the changes you requested
• broken tests are now marked and "pass" (https://github.com/nomeata/tasty-expected-failure/blob/master/Test/Tasty/ExpectedFailure.hs)
• added xml output (https://github.com/ocharles/tasty-ant-xml/blob/master/Test/Tasty/Runners/AntXML.hs)
• added rerun (https://github.com/ocharles/tasty-rerun/blob/master/src/Test/Tasty/Ingredients/Rerun.hs)

As of this commit I have not been able to test the re-run or xml output successfully, but I am sure it is just a matter of getting the cli parameters to get through with the cradle

Also I am not able to the see the CI results so not sure why the ghc version failed.

[fendor]

Awesome, I will review it in the next days in depth!

[fendor]

LGTM!

[Bodigrim]

@Bodigrim glad to have you take an interest. The main thing I care about is if a test fails, I get something like

  To rerun use: --match "/Plugin/composes code actions/provides 3.8 code actions/"

Your wish is granted :)