Service jobs should support the ability to be restarted after a failure

[kelseyhightower]

While testing the example redis job, I stopped the agent and the job was marked dead. When I restarted the agent, the job did not get rescheduled. This maybe by design, but it was unexpected, and I assumed since the job was a service nomad would just reschedule the job to an health node.

[armon]

@kelseyhightower That is odd, it should have done that automatically... Was there another healthy client available at the time of failure?

[kelseyhightower]

No, which has me thinking that the scheduler attempts to make a decision and fails the job when there are no other healthy nodes. I was expecting the job to be retried until success.

[armon]

@kelseyhightower Yeah, there is no good story for retrying a job when the scheduler fails in 0.1. We probably need to create a "retry" evaluation in a wait state.

[dadgar]

PR #393 fixes this.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.