Exceptions cause entire test suite to fail early

[ocharles]

I'm trying to implement a combination of a lexer/parser for SQL, and as such was hoping to just stub out swaths of SQL that my parser can't currently handle, and gradually implement them all to green and then move on. As such, I have something like the following:

import Prelude hiding (lex)

import Test.Tasty
import Test.Tasty.HUnit

import Lexer (lex)
import Parser

tests = testGroup "PostgreSQL regression tests"
  [ testGroup "ALTER TABLE" $ statements
      [ ("ALTER TABLE tmp ADD COLUMN xmin integer;", AlterTable "tmp" (AddColumn (ColumnDefinition "xmin" "integer" [])))
      , ("ALTER TABLE tmp ADD COLUMN a int4 default 3;", AlterTable "tmp" (AddColumn (ColumnDefinition "a" "int4" [Default "3"])))
      , ("ALTER TABLE tmp ADD COLUMN b name;", AlterTable "tmp" (AddColumn (ColumnDefinition "b" "name" [])))
      , ("ALTER TABLE tmp RENAME TO tmp_new;", AlterTable "tmp" (RenameTo "tmp_new"))
      , ("ALTER TABLE onek ADD CONSTRAINT onek_unique1_constraint UNIQUE (unique1);", undefined)
      , ("ALTER TABLE onek DROP CONSTRAINT onek_unique1_constraint_foo;", undefined)
      , ("ALTER TABLE onek ADD CONSTRAINT onek_check_constraint CHECK (unique1 >= 0);", undefined)
      , ("ALTER TABLE onek RENAME CONSTRAINT onek_unique1_constraint TO onek_unique1_constraint_foo;", undefined)
      , ("ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK (b > 0) NO INHERIT;", undefined)
      , ("ALTER TABLE IF EXISTS constraint_rename_test ADD CONSTRAINT con4 UNIQUE (a);", undefined)
      , ("ALTER TABLE tmp3 add constraint tmpconstr foreign key(c) references tmp2 match full;", undefined)
      , ("ALTER TABLE tmp3 add constraint tmpconstr foreign key (a) references tmp2 match full NOT VALID;", undefined)
      , ("ALTER TABLE tmp3 validate constraint tmpconstr;", undefined)
      , ("ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references pktable;", undefined)
      , ("ALTER TABLE FKTABLE ADD FOREIGN KEY(ftest1) references pktable(ptest1);", undefined)
      ]
  ]

statements = map (\(q, expected) -> testCase q (parse q @?= [expected]))

parse = sql . lex

main = defaultMain tests

However, the first test that fails:

    ALTER TABLE onek ADD CONSTRAINT onek_unique1_constraint UNIQUE (unique1);:                       FAIL
      Tests2: lexical error

causes the entire test executable to stop - no summary, no further tests.

I would expect each test to be bracketed for exceptions, so that one exception doesn't prevent everything from running.

[UnkindPartition]

This looks like a bug. Can you create a self-contained example?

[ocharles]

Oddly, my initial attempts to reproduce this on a smaller test case have the correct behavior... I'll try and give this another stab tomorrow.

[UnkindPartition]

Closing for now. Feel free to comment/reopen with more info.

[ocharles]

I can't re-open it, but I don't see why this should be closed - the problem has not been fixed. While I didn't produce a minimal test case, I have code that will reproduce it. I'll point to that commit shortly.

[UnkindPartition]

Because there's no way for anyone to diagnose and fix this without being able to reproduce it.

Please do point to your code, and I'll reopen.

[ocharles]

Grab https://github.com/ocharles/language-postgresql/tree/tasty

You will need happy and alex installed. Then:

alex Lexer.x && happy Parser.y && ghc --make Tests2.hs && ./Tests2

Should produce:

[nix-shell:~/work/language-postgresql]$ alex Lexer.x && happy Parser.y && ghc --make Tests2.hs && ./Tests2 --num-threads 20                                                      
[1 of 3] Compiling Lexer            ( Lexer.hs, Lexer.o )
[2 of 3] Compiling Parser           ( Parser.hs, Parser.o )
Linking Tests2 ...
PostgreSQL regression tests
  ALTER TABLE
    ALTER TABLE tmp ADD COLUMN xmin integer;:                                                        OK
    ALTER TABLE tmp ADD COLUMN a int4 default 3;:                                                    FAIL
      lexical error
    ALTER TABLE tmp ADD COLUMN b name;:                                                              OK
    ALTER TABLE tmp RENAME TO tmp_new;:                                                              FAIL
      Tests2: lexical error

[ocharles]

I should also mention these are the libraries I have installed:

[nix-shell:~/work/language-postgresql]$ ghc-pkg list
/nix/store/vn7g0xb0n470cwgc42cswd2jjl3q7k0n-haskell-tasty-ghc7.6.3-0.3.1/bin/../lib/ghc-7.6.3/package.conf.d/tasty-0.3.1.installedconf:
    tasty-0.3.1

/nix/store/fafbsj67g6lphc0a2p7h5qhya606il77-haskell-ansi-terminal-ghc7.6.3-0.6/bin/../lib/ghc-7.6.3/package.conf.d/ansi-terminal-0.6.installedconf:
    ansi-terminal-0.6

/nix/store/b9hr9slk217qf4cygjnpjjx5qacg20bs-haskell-mtl-ghc7.6.3-2.1.2/bin/../lib/ghc-7.6.3/package.conf.d/mtl-2.1.2.installedconf:
    mtl-2.1.2

/nix/store/0cdzm4liawz5qlscv8bqyk6h8z8a1yr8-haskell-transformers-ghc7.6.3-0.3.0.0/bin/../lib/ghc-7.6.3/package.conf.d/transformers-0.3.0.0.installedconf:
    transformers-0.3.0.0

/nix/store/3prmqbkyaprn0ys5byk75knl09lf9ccq-haskell-optparse-applicative-ghc7.6.3-0.7.0.2/bin/../lib/ghc-7.6.3/package.conf.d/optparse-applicative-0.7.0.2.installedconf:
    optparse-applicative-0.7.0.2

/nix/store/crk6m28fs92kp0wqd5dlcraqh0qjlv73-haskell-regex-posix-ghc7.6.3-0.95.2/bin/../lib/ghc-7.6.3/package.conf.d/regex-posix-0.95.2.installedconf:
    regex-posix-0.95.2

/nix/store/nhspk3fc1pnwag57cf1b86lzp9ayxfmd-haskell-regex-base-ghc7.6.3-0.93.2/bin/../lib/ghc-7.6.3/package.conf.d/regex-base-0.93.2.installedconf:
    regex-base-0.93.2

/nix/store/qzg65asdhgxx6gr1xn9qnpw35bhlifqv-haskell-stm-ghc7.6.3-2.4.2/bin/../lib/ghc-7.6.3/package.conf.d/stm-2.4.2.installedconf:
    stm-2.4.2

/nix/store/gzhxjwhwm9402p0mnlnfxnljnnsaphjn-haskell-tagged-ghc7.6.3-0.7/bin/../lib/ghc-7.6.3/package.conf.d/tagged-0.7.installedconf:
    tagged-0.7

/nix/store/7kj70dm2rf7czgcpibzncvc1g4v8jydc-haskell-tasty-hunit-ghc7.6.3-0.2/bin/../lib/ghc-7.6.3/package.conf.d/tasty-hunit-0.2.installedconf:
    tasty-hunit-0.2

/nix/store/xy1nhclic30xrixrg50dch0sjvn5izcp-haskell-HUnit-ghc7.6.3-1.2.5.2/bin/../lib/ghc-7.6.3/package.conf.d/HUnit-1.2.5.2.installedconf:
    HUnit-1.2.5.2

/nix/store/dvglca2lzxs1ccgpnrci1v64hbsc310q-haskell-uu-parsinglib-ghc7.6.3-2.8.1/bin/../lib/ghc-7.6.3/package.conf.d/uu-parsinglib-2.8.1.installedconf:
    uu-parsinglib-2.8.1

/nix/store/a32khkfy2q4ih03r0l9iiswimfmz6zip-haskell-ListLike-ghc7.6.3-4.0.0/bin/../lib/ghc-7.6.3/package.conf.d/ListLike-4.0.0.installedconf:
    ListLike-4.0.0

/nix/store/jg6gv194l3x7iqb2da7rz2w7ai3ickkf-haskell-text-ghc7.6.3-0.11.3.1/bin/../lib/ghc-7.6.3/package.conf.d/text-0.11.3.1.installedconf:
    text-0.11.3.1

/nix/store/05z7c8gy3s8dcfl7h8jkh1l4497c99ik-haskell-vector-ghc7.6.3-0.10.0.1/bin/../lib/ghc-7.6.3/package.conf.d/vector-0.10.0.1.installedconf:
    vector-0.10.0.1

/nix/store/m1rq0r68jshwfa73iz83m2i8wrv3m2db-haskell-primitive-ghc7.6.3-0.5.0.1/bin/../lib/ghc-7.6.3/package.conf.d/primitive-0.5.0.1.installedconf:
    primitive-0.5.0.1

/nix/store/29mh7v87r4bwih0gmm1lszrqmsi4d5hi-haskell-uu-interleaved-ghc7.6.3-0.1.0.0/bin/../lib/ghc-7.6.3/package.conf.d/uu-interleaved-0.1.0.0.installedconf:
    uu-interleaved-0.1.0.0

/nix/store/6vrrb42wk7vca94m0a3fzpqkrf1h7nx8-haskell-parsers-ghc7.6.3-0.10/bin/../lib/ghc-7.6.3/package.conf.d/parsers-0.10.installedconf:
    parsers-0.10

/nix/store/q8fhiccrkrwmjqb672i4lryzryai26ds-haskell-charset-ghc7.6.3-0.3.5.1/bin/../lib/ghc-7.6.3/package.conf.d/charset-0.3.5.1.installedconf:
    charset-0.3.5.1

/nix/store/dyzp4pc47nb4zh3hjffv1yw0hxsabfqk-haskell-semigroups-ghc7.6.3-0.11/bin/../lib/ghc-7.6.3/package.conf.d/semigroups-0.11.installedconf:
    semigroups-0.11

/nix/store/mgisrsm504sbdkzks0xancsafcq4jpjv-haskell-hashable-ghc7.6.3-1.1.2.5/bin/../lib/ghc-7.6.3/package.conf.d/hashable-1.1.2.5.installedconf:
    hashable-1.1.2.5

/nix/store/jhxl424z0hdsnswj7x2wdkrmyswd945w-haskell-nats-ghc7.6.3-0.1.2/bin/../lib/ghc-7.6.3/package.conf.d/nats-0.1.2.installedconf:
    nats-0.1.2

/nix/store/3qf8km2r9ibhz6kqpvb5rpb715v5jf5l-haskell-unordered-containers-ghc7.6.3-0.2.3.0/bin/../lib/ghc-7.6.3/package.conf.d/unordered-containers-0.2.3.0.installedconf:
    unordered-containers-0.2.3.0

/nix/store/bnnjc7v84p87xf4kizqg7xzzknxvn2nq-haskell-parsec-ghc7.6.3-3.1.3/bin/../lib/ghc-7.6.3/package.conf.d/parsec-3.1.3.installedconf:
    parsec-3.1.3

[UnkindPartition]

This should be fixed now.

The problem was that the failure message threw an exception when printed.

The message could be better (it could print some initial part of the message before the exception is thrown), but I'll leave it to someone else (or the future myself) to do.

[ocharles]

Ha, what a fun little bug! Thanks for getting this fixed, sorry for not throwing the code up sooner :)

[larsrh]

This also affects tasty-html: UnkindPartition/tasty-html#15. It might be worth to expose a general mechanism how reporters can deal with it.