Skip to content

Commit

Permalink
test: ensure make unslurmcluster always runs in CI (#9420)
Browse files Browse the repository at this point in the history
When cluster creation fails halfway, pkill determined-mast will
exit with 1. CircleCI runs these inline script with set -e, so that
failure causes make unslurmcluster to no run, leaving stray
resources behind.
  • Loading branch information
stoksc authored May 24, 2024
1 parent ba31f03 commit cdd7a82
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion .circleci/real_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2800,7 +2800,9 @@ jobs:
name: Make unslurmcluster
when: always
command: |
pkill determined-mast
# These scripts always run with `set -e`. Be careful that a failure doesn't stop
# make unslurmcluster from running, otherwise resources are left around.
pkill determined-mast || echo "failed to kill master (exit code $?)"
(yes yes || true) | make unslurmcluster TF_LOCK=false
# For some reason, even when `make unslurmcluster` is successful, CircleCI
# receives exit code 141, so ignore that.
Expand Down

0 comments on commit cdd7a82

Please sign in to comment.