Some integration tests not being run in CI

[robdockins]

It appears the integration test suite is not being run in CI jobs, which allows errors like #994 to occur.

[RyanGlScott]

CI is running the integration tests—see here. The issue is that CI is excluding certain tests, including test_examples (the subject of #994) here.

[kquick]

If test suite invoked SAW directly instead of as a sub-process, this might enable better coverage reporting. Still probably need at least a couple of tests that invoke the executable to ensure that part works correctly.

[brianhuffman]

I made this comment in #1013, but I thought I should repeat it here. This CI script contains this list of excluded tests:

saw-script/.github/ci.sh

Line 173 in 6789b1a

for t in test0001 test0019_jss_switch_statement test_crucible_jvm test_ecdsa test_examples test_issue108 test_tutorial1 test_tutorial2 test_tutorial_w4; do echo $t >> disabled_tests.txt; done

It would be nice to add comments explaining why each of these tests is excluded, to make it easier to know if/when it is appropriate to add each one back in. And any test that is actually broken should have a ticket.

[brianhuffman]

Just for historical context, here are the commits that introduced the lists of excluded tests: e905271, 4acafb3.

The file intTests/disabled_tests.txt actually does contain comments explaining why certain tests are excluded. It appears CI excludes the tests named in the CI scripts themselves in addition to the ones listed in this file.

[brianhuffman]

As of #1070, .github/ci.sh no longer excludes any additional tests (besides those explicitly excluded in intTests/disabled_tests.txt). Is this sufficient to close this ticket?

[kquick]

I believe it can, if @rdockins and @RyanGlScott don't have any objections. Thanks @brianhuffman !

[RyanGlScott]

Sounds good to me. I agree with this part of #1070:

If there is a specific reason for any integration test to be
excluded from CI, the rationale for exclusion should be given
in a comment inside disabled_tests.txt.