How to run the tests?

[gelisam]

Running cabal install --enable-tests used to be enough to run the tests in the background, but it doesn't seem to do so anymore; maybe cabal-install changed? Also, running cabal test afterwards complains that I haven't run cabal configure; shouldn't cabal install perform this step implicitly?

[gelisam]

For now, the workaround is to first build with --enable-tests, and then to manually run ./dist/dist-sandbox-*/build/reference/reference.

[gelisam]

I think the configure step has been performed by cabal install, but that it has been performed on a different version of Cabal (exactly like the error message says, actually). I don't know why it's doing that.

When running the tests with ghc 7.10 (see #140), the error message is much worse:

cabal: Prelude.chr: bad argument: 2956230

But the explanation of this error message is compatible with the hypothesis that cabal install and cabal test are using different versions of Cabal for some reason. I suspect this must be somehow due to the fact that hawk is using a custom Setup.hs?

[gelisam]

I have a note in Setup.hs which says that when you call cabal install --enable-tests, cabal runs Setup.hs a few times with different arguments:

["configure","--verbose=1","--builddir=dist/dist-sandbox-28d8356a","--ghc","--prefix=...",...]
["build","--verbose=1","--builddir=dist/dist-sandbox-28d8356a"]
["test","--builddir=dist/dist-sandbox-28d8356a"]
["install","--verbose=1","--builddir=dist/dist-sandbox-28d8356a"]

However, when I tried it today with cabal-install-1.21.1.0 and Cabal-1.21.1.0, cabal ran Setup.hs a different number of times with different arguments:

["configure","--verbose=1","--builddir=dist/dist-sandbox-f1c53858","--ghc","--prefix=...",...]
["build","--verbose=1","--builddir=dist/dist-sandbox-f1c53858"]
["register","--verbose=1","--builddir=dist/dist-sandbox-f1c53858","--gen-pkg-config=..."]
["copy","--verbose=1","--builddir=dist/dist-sandbox-f1c53858"]
["register","--verbose=1","--builddir=dist/dist-sandbox-f1c53858"]

I guess that explains why cabal install --enable-tests no longer runs the tests automatically.

[gelisam]

Turns out the fact that cabal install --enable-tests no longer runs the tests is actually a new feature, and that the fact that cabal test uses a different version of Cabal is a known bug.

[gelisam]

Can I get rid of the Setup.hs file? All I want is to make sure the library is installed before I run the tests. Maybe I can detect that and print a message to the user? The doctest library prints the following message if Cabal is not installed, but clearly doesn't simply list Cabal.1.22.2.0 in its .cabal file. Maybe investigate how they do that?

"user error (The package 'doctest' requires Cabal library version 1.22.2.0 but
no suitable version is installed.)"

[orome]

What is the status of this issue?

Currently if try:

cabal install --reinstall --enable-tests --run-tests the-package

and its various combinations and permutations, no tests seem to run at all.

[gelisam]

I've never heard of the "--run-tests" flag, where is it documented?

Also, are you trying to run hawk's tests or crypto-enigma's? You refer to both packages in your StackOverflow comments.

As for the status of this bug, I see that the Cabal bug on which this is depending has been closed, and indeed, with cabal-install-1.22.6.0, instead of complaining that I haven't ran configure...

$ cabal sandbox init
$ cabal update
$ cabal install cabal-install
$ cabal install happy
$ .cabal-sandbox/bin/cabal install --enable-tests
$ .cabal-sandbox/bin/cabal test
[...]
setup: Unrecognised flags: reference

... my custom Setup.hs complains about an unrecognized flag, "reference", because cabal likes to switch up the way in which it talks with Setup.hs from version to version. So the bug stands, and the best way to run hawk's tests is still to install them with cabal install --enable-tests and then to run them manually with ./dist/dist-sandbox-*/build/reference/reference.

$ cabal install aeson
$ ./dist/dist-sandbox-*/build/reference/reference
[...]
13 examples, 0 failures

[gelisam]

I've never heard of the "--run-tests" flag, where is it documented?

Never mind, I see it's the replacement for the old behaviour of also running the tests when "--enable-tests" is passed.

[gelisam]

Tests are failing again (#217), but at least the question of how the tests are supposed to be run is now clear, since they run on CI: either using cabal with cabal v1-update && cabal v1-sandbox init && cabal v1-install happy && cabal v1-install --enable-tests && cabal v1-test (these pass on my machine and on CI), or using stack with stack test --no-run-tests && stack test (these fail both on CI and on my machine, but with different error messages).

hawk/.travis.yml

Lines 55 to 67 in 7213655

	# Build the package, and its tests and its docs (stack)
	- if [ "$BUILD" == "stack" ]; then stack --resolver="$RESOLVER" --no-terminal --install-ghc test --no-run-tests --haddock --no-haddock-deps; fi
	# Build the package and its tests (cabal)
	- if [ "$BUILD" == "cabal" ]; then ~/.local/bin/cabal update; fi
	- if [ "$BUILD" == "cabal" ]; then ~/.local/bin/cabal sandbox init; fi
	- if [ "$BUILD" == "cabal" ]; then ~/.local/bin/cabal install happy; fi
	- if [ "$BUILD" == "cabal" ]; then ~/.local/bin/cabal install --enable-tests; fi
	script:
	# Run the tests (stack)
	- if [ "$BUILD" == "stack" ]; then stack --resolver="$RESOLVER" --no-terminal test; fi
	# Run the tests (cabal)
	- if [ "$BUILD" == "cabal" ]; then ~/.local/bin/cabal test; fi

[gelisam]

I was able to get the tests to pass locally with stack as well by following the instructions at #211