Nomad doesn't re-schedule jobs if Docker daemon is restarted

[c4milo]

I've been testing the ability of a cluster to recover its state, especially when the Docker daemon is restarted. But, every time this happens, the allocations moved to failed status and Nomad didn't restart them ever again, not even if I re-run the job. In order to fix it, I had to stop the job and start over.

[dadgar]

Yeah this will hopefully be fixed by Nomad 0.3 and in a generic way too. We currently don't migrate failed allocs to new nodes and once we do that, this problem should be fixed

[c4milo]

@dadgar is this still on track for 0.3?

[dadgar]

No this will not be fixed in 0.3. We decided to not tackle server side restarts in this release.

[c4milo]

@dadgar, Nomad making sure the number of instances is always honored is the only thing right now stopping me from using Nomad in production, instead of Kubernetes. Are there any plans to fix this in the short/mid term?

[dadgar]

@c4milo So I just restarted a docker daemon that was running containers started by Nomad. Nomad detected it and restarted the task locally

[ketzacoatl]

There are two separate issues:

1. if the docker daemon restarts, and the docker container dies, nomad should restart that - it does for me too, in my tests now.
2. if the node fails, nomad needs to have the task moved to a different node - in my tests, this has not worked

[dadgar]

@ketzacoatl: Nomad actually does both of these. There is a node failure detection window and once the node hasn't talked to the servers for a period of time, Nomad will reschedule what it is running

[dadgar]

@c4milo can you reproduce the behavior you were experiencing? If not we should probably close this

[c4milo]

@dadgar I haven't tested but feel free to close it. I will re-open it if I'm able to reproduce.

[ketzacoatl]

when I last tested, nomad did not reschedule the job on a node that failed,. Maybe that has changed, I will squeeze in a test in the time to come.. but I agree that is not what this issue intends to be about.
+1 on closing

[dadgar]

Cool! Thanks guys!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.