ci: keep waiting on failing workloads for sending slack alerts #9371
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for determined-ui canceled.
|
NicholasBlaskey
changed the title
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #9371 +/- ##
=======================================
Coverage 45.28% 45.29%
=======================================
Files 1227 1227
Lines 154048 154048
Branches 2404 2403 -1
=======================================
+ Hits 69767 69773 +6
+ Misses 84089 84083 -6
Partials 192 192
Flags with carried forward coverage won't be shown. Click here to find out more. |
This reverts commit 6b99a16.
stoksc
approved these changes
May 15, 2024
JComins000
reviewed
May 15, 2024
| @@ -41,7 +41,7 @@ def send_alerts_for_failed_jobs(sent_alerts: Set[str]) -> bool: | |||
| continue | |||
|
|
|||
| workflow_id = w["id"] | |||
| if not workflows_are_running and w["status"] == "running": | |||
| if not workflows_are_running and w["status"] in ["running", "failing"]: | |||
There was a problem hiding this comment.
maybe w["stopped_at"] might be better to use?
JComins000
approved these changes
May 15, 2024
This reverts commit 0c6d970.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
Description
Sometimes we could miss alerts due to the alert job stopping before all jobs are completed.
https://hpe-aiatscale.slack.com/archives/C9LFPNA3Y/p1715639036613519
Update the script to wait for failing state too.
Test Plan
see if we keep having this issue
Checklist
docs/release-notes/.See Release Note for details.