-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Marks failing commands as having succeeded if they have too much output (SIGPIPE) #76
Comments
Besides test retries being bad (they're there for the case that the test intermittently flakes, but mask the case that the code-under-test only intermittently works), this is important in order to remove the `nick-invision/[email protected]` GitHub action. It has a bug where if there is too much output it marks the test as passing even if it fails. That is very bad, and we cannot permit that type of failure mode in our CI. nick-fields/retry#76 Signed-off-by: Luke Shumaker <[email protected]>
Besides test retries being bad (they're there for the case that the test intermittently flakes, but mask the case that the code-under-test only intermittently works), this is important in order to remove the `nick-invision/[email protected]` GitHub action. It has a bug where if there is too much output it marks the test as passing even if it fails. That is very bad, and we cannot permit that type of failure mode in our CI. nick-fields/retry#76 Signed-off-by: Luke Shumaker <[email protected]> Signed-off-by: Sirish Bathina <[email protected]>
Doing some research on this, it looks like this could be connected with the Doing a bit more research, is there a reason to not use |
Thanks for this report. I just published v2.8.0 that will resolve this. Thanks to your repo, I was able to reproduce the issue before the fix and then confirm it was fixed after by testing with upto 1000MiB of output. |
Describe the bug
If a command has too much output, then even though the command might fail, the step gets marked as passing.
I measured "too much" as ≥1026 KiB, but I wouldn't be surprised if it's timing related and varies at any value much over 1024.
I've created a reproducer over at https://github.com/LukeShu/gha-test where each step
always()
runs and dumps an increasing amount of output to stdout and then exits with a failure. As seen in the screenshot below, everything <=1MiB correctly fails, but then everything >=1⅛MiB starts passing.This can be replicated locally, as well:
(All you need to know about my
gha-test
reproducer is thatmake -C ../gha-test bytes-1152
spits out 1152KiB of <=80-character lines on stdout, and then exits with code 1.)I originally encountered this with v2.4.0, but the local-reproduction above is with the latest version (v2.7.0).
Expected behavior
I expected that if my tests fail, then my CI will fail.
Screenshots
Logs
Enable debug logging then attach the raw logs (specifically the raw output of this action).
The text was updated successfully, but these errors were encountered: