Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests intermittently fail without causing the build to fail #122

Closed
chrispcampbell opened this issue Sep 27, 2021 · 1 comment · Fixed by #123 or #190
Closed

Tests intermittently fail without causing the build to fail #122

chrispcampbell opened this issue Sep 27, 2021 · 1 comment · Fixed by #123 or #190
Assignees
Labels
Milestone

Comments

@chrispcampbell
Copy link
Contributor

I just noticed that sometimes one or more tests will fail (sporadically) when running on GitHub Actions, for example:
https://github.com/climateinteractive/SDEverywhere/runs/3677720774?check_suite_focus=true#step:4:65

Example error message:

Testing the initial model
../../models/initial/output/initial.dat not found

This has apparently been happening for a while now. I went back through the build logs and found examples of this going back as far as this build on July 24:
https://github.com/climateinteractive/SDEverywhere/actions/runs/1063070071

I didn't find examples of the issue in builds before that one, but it doesn't necessarily mean that change is to blame (and it probably isn't).

There are two issues at play here:

  1. A test is failing when it shouldn't.
  2. The build is not failing when sde test encounters an error; the build should fail in this case.
@chrispcampbell chrispcampbell self-assigned this Sep 27, 2021
@chrispcampbell
Copy link
Contributor Author

For the first issue (sde test fails to output DAT file intermittently), I found an issue in sde-log.js in the writeDat function. It was calling stream.end then immediately resolving the promise. The problem is that the OS may not have flushed the write buffer to the output file yet, which is why we'd intermittently get an error message saying that the output DAT file is not found. That code needs to be changed to wait for the finish event, which is called only after the buffer is flushed.

For the second issue, we need to improve the error handling in sde-test and sde-compare so that it exits with a non-zero exit code if some step fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment