Stop using Cabal's test (yay!)

[snoyberg]

Cabal's test runner just causes huge amounts of problems:

• GHC_PACKAGE_PATH screws up doctest and others
• I'm now tripping up on streaming data bug again (cabal test freezes but cabal test --show-details=streaming works haskell/cabal#1810)

So instead: let's just run tests ourselves directly. We can even have the output go to stdout if desired. Downside: we won't get automatic support for the new test types like detailed. I'm OK with that.

[bitemyapp]

👍

[snoyberg]

I should have gone to bed already, but this was annoying me enough that I just implemented it. I don't think it's close to perfect yet, but should hopefully be an improvement already. For starters, when you're just running a single test suite, it will print the output to the console. Unfortunately, I still seem to be seeing the freezing behavior... very weird. Will investigate tomorrow.

[snoyberg]

I'm able to reproduce the freezing behavior by just calling rawSystem on the executable all by itself, so this is not a stack-specific bug. Using GHC 7.10 does not demonstrate that behavior, so either the RTS has some fixes or process got a fix. I'm trying to build stack with a newer process on GHC 7.8, should be interesting.

[snoyberg]

Nope, problem persists with GHC 7.8 and newer process.

[snoyberg]

I've resolved this problem, but I'm walking away unsatisfied. tl;dr: If I add -threaded to the test suite section (which should have been there anyway actually), everything works, both with this change and with runghc Setup.hs test. It seems like there's a bad interaction between a the single-threaded RTS being used in a process being run, and GHC 7.8 running that process.

I still don't know why this test suite triggered the problem when others didn't, but I've done all I can on this tonight.

@bitemyapp Since we've already been working together on improving the testing, would you mind taking the latest stack and see how its stack test plays with your doctests? Hopefully the result should be nicer that what you were getting previously.

[bitemyapp]

@snoyberg I'll give it a shot tonight (US Central), thank you!

[bitemyapp]

@snoyberg

Seems like it got pretty far.

Running 2 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to:
dist-stack/Cabal-1.18.1.5/test/bloodhound-0.5.0.1-tests.log
Test suite doctests: RUNNING...
doctests: While parsing "/home/callen/Work/bloodhound/.cabal-sandbox/x86_64-linux-ghc-7.10.1-packages.conf.d/package.cache": Prelude.chr: bad argument: 3703282
Test suite doctests: FAIL
Test suite logged to:
dist-stack/Cabal-1.18.1.5/test/bloodhound-0.5.0.1-doctests.log
1 of 2 test suites (1 of 2 test cases) passed.
stack: Error when running Shake build system:
* /home/callen/Work/bloodhound/dist-stack/Cabal-1.18.1.5/stack.gen
Process exited with ExitFailure 1: /home/callen/.stack/ghc/x86_64-linux/ghc-7.8.4/bin/runghc-7.8.4 -package=Cabal-1.18.1.5 -clear-package-db -global-package-db /home/callen/Work/bloodhound/Setup.hs --builddir=dist-stack/Cabal-1.18.1.5/ test

[callen@atlantis ~/Work/bloodhound]$ cat dist-stack/Cabal-1.18.1.5/test/bloodhound-0.5.0.1-doctests.log
Test suite doctests: RUNNING...
doctests: While parsing "/home/callen/Work/bloodhound/.cabal-sandbox/x86_64-linux-ghc-7.10.1-packages.conf.d/package.cache": Prelude.chr: bad argument: 3703282
Test suite doctests: FAIL
Test suite logged to: dist-stack/Cabal-1.18.1.5/test/bloodhound-0.5.0.1-doctests.log

[snoyberg]

Interesting. doctest is finding your .cabal-sandbox and allowing that to override the package sandbox information that stack is setting. This results in it trying to use a GHC 7.10 package database, which GHC 7.8 doesn't understand. Presumably you'd get the same kind of error message if you had GHC 7.8 on your PATH and ran the test executable manually.

Can you try renaming .cabal-sandbox to something else and trying again? For the record, in my testing, I got the following output (I don't have a local server running):

Linking dist-stack/Cabal-1.18.1.5/build/doctests/doctests ...
### Failure in src/Database/Bloodhound/Client.hs:76: expression `_ <- deleteIndex testServer testIndex'
expected:
 but got: *** Exception: FailedConnectionException2 "localhost" 9200 False connect: does not exist (Connection refused)
Examples: 77  Tried: 16  Errors: 0  Failures: 1
stack: Error when running Shake build system:
* /home/vagrant/Desktop/bloodhound/dist-stack/Cabal-1.18.1.5/stack.gen
TestSuiteFailure "/home/vagrant/Desktop/bloodhound/dist-stack/Cabal-1.18.1.5/build/doctests/doctests" Nothing (ExitFailure 1)

[bitemyapp]

@snoyberg pulled latest stack, worked great until doctests (nice test output too)

Finished in 20.7952 seconds
44 examples, 0 failures
stack: ExecutionFailure [TestSuiteFailure2 "bloodhound-0.6.0.0" (fromList [("doctests",Just (ExitFailure 1))]) Nothing]

I had to scroll up (the error output here is poor, suggest fixing error locality/reporting), but I found:

doctests: doctests: can't find a package database at /home/callen/Work/.cabal-sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d

I ran cabal clean, nuked sandbox and sandbox config. Don't know what's causing this.

bad news 🐻 🐻 🐻 😢

[snoyberg]

Are you sure you nuked the sandbox config? There isn't one in the current directory or a parent?

I already sent a PR (which is merged) to doctest that addresses this problem in how it works, but it's only available in doctest 0.10.

[bitemyapp]

@snoyberg ^^ no sandbox, yeah? I don't think I missed anything there. It'd been cabal clean'd too.

I have a knack for hitting bad states with tools. Kind of a curse really because repro is hard.

[snoyberg]

Only other thing I can think of: is there a cabal.sandbox.config file in any of the ancestor directories? Looking through the doctest source code, that seems to be the only way this code path would get triggered.

[bitemyapp]

@snoyberg

Works fine now. YEESH.

I know this seems a bit one off, but if we detect that doctests failed, could we toss a warning out about this or are the doctests fixes going to make this go away?

[snoyberg]

The doctest fixes completely resolve this issue. I made it so that the environment variables that stack sets override the presence of a cabal.sandbox.config.

[bitemyapp]

@snoyberg awesome, thank you! This is so exciting 💃 🐻