Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop evaluations that take too long #408

Open
fricklerhandwerk opened this issue Nov 30, 2024 · 0 comments
Open

Stop evaluations that take too long #408

fricklerhandwerk opened this issue Nov 30, 2024 · 0 comments

Comments

@fricklerhandwerk
Copy link
Collaborator

Since #405, starting the worker service will stop all ongoing evaluations (or mark stuck ones) by labeling them as crashed. We want to do this periodically in case evaluations go on longer than some reasonable time, such as 24h since that's our update period.

Implementation note:

add a supervisor timer unit that will check if there's any evaluation taking far too long (> 24 hours) and will forcibly restart the worker.
#405 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

1 participant