Reschedule or restart a task on health check failure

[samart]

It would nice if a failing check restarted a task or re-scheduled it elsewhere.

Consul provides a service discovery healthcheck, but when a service is unresponsive we'd like to restart it. Our mesos clusters do this with marathon keep-alive healthchecks and it works well to keep applications responsive.

We should be able to specify at least:

• gracePeriod to wait before starting to healthcheck, after the task has started
• number of check failures before a restart/re-schedule
• interval between checks
• timeout on the check attempt

Including the current nomad restart options would be nice as well. Max restart attempts, mode, etc.

[epetrovich]

#876
This is an essential feature, but I've did it using my own script that is launched in the task group of the main app. Restarts of individual tasks/allocation is possible via Consul-kv and template engine. I just wrote a simple script that checks the status of consul healthcheck and implements my own restart logic via Consul KV.
Here's my working script for that.
It's designed to be launched in the allocation with the main app. All necessary parameters should be passed from jobs manifest

watcher.py.gz

[dadgar]

Hey @samart Going to close as duplicate of #876 but I captured your comment there. We will be working on this soon!

[samart]

i will watch the other issue, thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.