Nomad 0.9.0 Client deadlocks when starting after reboot

[schmichael]

Originally reported by @vkiranananda here: #2560 (comment)

Nomad version

Nomad v0.9.0 (18dd59056ee1d7b2df51256fe900a98460d3d6b9)

Operating system and Environment details

dev vagrant box

Issue

When a node (the actual VM or server) is reboot Nomad deadlocks on startup trying to restore Docker tasks.

Reproduction steps

On the dev vagrant box with Nomad 0.9.0 installed with this config file.

Terminal 1:

curl https://gist.githubusercontent.com/schmichael/579e22c5bd64cd9a3b05e81c3a0964e9/raw/f3bb5ed8a2aeebc1aeccf24e86f508472418592e/devagent.hcl > devagent.hcl
sudo nomad agent -config devagent.hcl

Terminal 2:

nomad init 
nomad run example.nomad
# wait for job to be running
^D # logout of vagrant box
vagrant halt
vagrant up
vagrant ssh
sudo nomad agent -config devagent.hcl
# deadlocks

Job file (if appropriate)

Example job (w/o service stanza)

Nomad Client logs (if appropriate)

Full logs + goroutine dump here: https://gist.github.com/schmichael/b0f663b293c9f2c47e2790c4f9f8fb70

[schmichael]

Killing the docker_logger unblocks the client and causes it to start a new alloc. Task events from original alloc:

2019-04-15T22:33:46Z  Killing           Sent interrupt
2019-04-15T22:33:46Z  Restarting        Task restarting in 15.954383114s
2019-04-15T22:33:46Z  Task hook failed  logmon: Reattachment process not found

[MorphBonehunter]

maybe this is related/duplicate to #5561 ?

[schmichael]

@MorphBonehunter Oh interesting! Yes! The only difference is that you kill all Nomad processes but the container is still running. I actually like your repro much better.

Going to close #5561 just because we've already started referencing this issue in PRs and internal comms. Sorry for missing yours and thanks for commenting. Fix is landing soon.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.