Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker: kill signal API should include timeout context #9502

Merged
merged 1 commit into from
Dec 2, 2020

Conversation

tgross
Copy link
Member

@tgross tgross commented Dec 2, 2020

Possible fix for #9375 and #9376, based on the discovery of open Docker bugs
that might trigger this behavior (as described in #9375 (comment)). I've smoke-tested
this for safety but haven't had a chance yet to see if it really improves the situation from
those issues.

When the Docker driver kills as task, we send a request via the Docker API for
dockerd to fire the signal. We send that signal and then block for the
kill_timeout waiting for the container to exit. But if the Docker API
blocks, we will block indefinitely because we haven't configured the API call
with the same timeout.

This changeset is a minimal intervention to add the timeout to the Docker API
call only when we have the kill_timeout set. Future work should examine
whether we should be threading contexts through other go-dockerclient API
calls.

@vercel
Copy link

vercel bot commented Dec 2, 2020

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployments, click below or on the icon next to each commit.

nomad-storybook-and-ui – ./ui

🔍 Inspect: https://vercel.com/hashicorp/nomad-storybook-and-ui/npcav2w35
✅ Preview: Canceled

[Deployment for a1bfc03 canceled]

@tgross tgross requested review from notnoop and schmichael December 2, 2020 21:30
Copy link
Member

@schmichael schmichael left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Don't forget a changelog line under bug fixes like:

driver/docker: Fixed a bug where the Docker daemon could block longer than the `kill_timeout`  [[GH-9502](https://github.com/hashicorp/nomad/issues/9502)

drivers/docker/driver.go Show resolved Hide resolved
When the Docker driver kills as task, we send a request via the Docker API for
dockerd to fire the signal. We send that signal and then block for the
`kill_timeout` waiting for the container to exit. But if the Docker API
blocks, we will block indefinitely because we haven't configured the API call
with the same timeout.

This changeset is a minimal intervention to add the timeout to the Docker API
call _only_ when we have the `kill_timeout` set. Future work should examine
whether we should be threading contexts through other `go-dockerclient` API
calls.
@tgross tgross force-pushed the b-docker-signal-timeout branch from 3f9cf73 to a1bfc03 Compare December 2, 2020 21:52
@vercel vercel bot temporarily deployed to Preview – nomad December 2, 2020 21:52 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui December 2, 2020 21:52 Inactive
@tgross tgross merged commit dc2bec7 into master Dec 2, 2020
@tgross tgross deleted the b-docker-signal-timeout branch December 2, 2020 21:53
Copy link
Contributor

@notnoop notnoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Good catch!

@github-actions
Copy link

github-actions bot commented Dec 7, 2022

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 7, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants