drainer: defer CSI plugins until last #12324
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a node is drained, system jobs are left until last so that operators can rely on things like log shippers running even as their applications are getting drained off. Include CSI plugins in this set so that Controller plugins deployed as services can be handled as gracefully as Node plugins that are running as system jobs.
tgross
commented
Mar 21, 2022
| last. You should always use this flag when draining a node running | ||
| [CSI node plugins][internals-csi]. | ||
| stopping system job allocations. By default system jobs (and CSI | ||
| plugins) are stopped last, after the `deadline` time has expired. |
There was a problem hiding this comment.
Note to reviewers: this "after the deadline time has expired" bit is existing behavior that didn't seem to be explicitly documented.
tgross
commented
Mar 21, 2022
| last. You should always use this flag when draining a node running | ||
| [CSI node plugins][internals-csi]. | ||
| stopping system job allocations. By default system jobs (and CSI | ||
| plugins) are stopped last, after the `deadline` time has expired. |
There was a problem hiding this comment.
Note to reviewers: this "after the deadline time has expired" bit is existing behavior that didn't seem to be explicitly documented.
tgross
added a commit
that referenced
this pull request
May 2, 2022
In #12324 we made it so that plugins wait until the node drain is complete, as we do for system jobs. But we neglected to mark the node drain as complete once only plugins (or system jobs) remaining, which means that the node drain is left in a draining state until the `deadline` time expires. This was incorrectly documented as expected behavior in #12324.
tgross
added a commit
that referenced
this pull request
May 3, 2022
In #12324 we made it so that plugins wait until the node drain is complete, as we do for system jobs. But we neglected to mark the node drain as complete once only plugins (or system jobs) remaining, which means that the node drain is left in a draining state until the `deadline` time expires. This was incorrectly documented as expected behavior in #12324.
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #11614 (when combined with #11892)
When a node is drained, system jobs are left until last so that
operators can rely on things like log shippers running even as their
applications are getting drained off. Include CSI plugins in this set
so that Controller plugins deployed as services can be handled as
gracefully as Node plugins that are running as system jobs.
The changeset here is fairly trivial and I've tested it by hand; I don't know that a unit test is going to tell us anything interesting here. But I'm going to follow up this PR next week with an E2E test for CSI node drain behavior more generally.