All nomad containers receive SIGTERM after ~10 minutes

[ColtonHerrod]

I've been troubleshooting a strange issue off and on for a while. All nomad containers I run receive a SIGTERM after about 10 minutes of runtime and reboot as appropriate for their configuration. This appears to be the case regardless of the image being run or if health checks are enabled. None of the containers approach their memory or CPU limits and the single node I'm running is nowhere near its resource limits. This pattern seems to persist as long as the job runs.

There is nothing seemingly obvious in the logs that points to a specific issue. Is this something that could be an issue with the podman driver or with my local configuration? Happy to provide additional information as needed.

System Information

• OS: RHEL 9.4
• Nomad Version: 1.8.0
• Podman Driver: 4.9.4-rhel

[thr27]

If you use podman, make sure to disable docker plugin. I had a similar issue. It seems dangling controler gc of docker plugin kills podman containers. As docker plugin detects podman api as a installed docker environment