Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cancels entire series if there's a scheduler error #399

Merged
merged 7 commits into from
Apr 6, 2021
Merged

cancels entire series if there's a scheduler error #399

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5bf019d
cancels entire series if there's a scheduler error
francinelapid Mar 16, 2021
0c6102a
includes test id in output error
francinelapid Mar 17, 2021
f4f9c02
Merge branch 'master' into sched-error-398
Paul-Ferrell Mar 23, 2021
8df2465
Merge branch 'master' of https://github.com/hpc/pavilion2 into sched-…
francinelapid Mar 23, 2021
16fa01a
Merge branch 'sched-error-398' of https://github.com/hpc/pavilion2 in…
francinelapid Mar 23, 2021
d35f2cd
replaced by_sigterm parameter with message
francinelapid Mar 23, 2021
fa75413
Merge branch 'master' into sched-error-398
Paul-Ferrell Apr 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions lib/pavilion/series.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,12 +539,12 @@ def run_tests(self, wait: Union[None, int] = None,
try:
sched.schedule_tests(self.pav_cfg, [test])
except schedulers.SchedulerPluginError as err:
fprint('Error scheduling test: ', file=self.errfile,
fprint('Error scheduling test: ', test.id, file=self.errfile,
color=output.RED)
fprint(err, bullet=' ', file=self.errfile)
fprint('Cancelling already kicked off tests.',
file=self.errfile)
sched.cancel_job(test)
self.cancel_series(message="Killed because of scheduler error.")
return errno.EINVAL

# Tests should all be scheduled now, and have the SCHEDULED state
Expand Down Expand Up @@ -684,15 +684,15 @@ def create_set_graph(self):

return

def cancel_series(self):
def cancel_series(self, message=None):
"""Goes through all test objects assigned to series and cancels tests
that haven't been completed. """

for test_id, test_obj in self.tests.items():
if not (test_obj.path/'RUN_COMPLETE').exists():
sched = schedulers.get_plugin(test_obj.scheduler)
sched.cancel_job(test_obj)
test_obj.status.set(STATES.COMPLETE, "Killed by SIGTERM.")
test_obj.status.set(STATES.COMPLETE, message)
test_obj.set_run_complete()

def run_series(self):
Expand All @@ -702,7 +702,7 @@ def run_series(self):
def sigterm_handler(_signals, _frame_type):
"""Calls cancel_series and exists."""

self.cancel_series()
self.cancel_series(message="Series killed by SIGTERM.")
sys.exit()

signal.signal(signal.SIGTERM, sigterm_handler)
Expand Down