Account for updated pods when waiting on DaemonSet #102
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #102 +/- ##
=======================================
Coverage 24.02% 24.02%
=======================================
Files 1 1
Lines 154 154
Branches 29 29
=======================================
Hits 37 37
Misses 112 112
Partials 5 5 Continue to review full report at Codecov.
|
The exising logic that's used to determine when a DaemonSet is ready fails to account for the fact that a RollingUpdate first kills the pod and then creates a new one. Simply checking if the desiredNumberScheduled equals the numberReady will succeed in cases when the old pod takes time to shut down, and would report that the new Deployment is ready despite the fact that the old pod has not been replaced, yet.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
We have been seeing infrequent CI failures recently during the DaemonSet tests in the waiter test suite. The problem can be reliably reproduced by using a container that ignores SIGTERM and forces the replacement cycle to run out the grace period for termination. The logic we're using to determine when a DaemonSet is ready checks whether the number of nodes that should be running the daemon pod is the same as the number nodes currently running a daemon pod. In this case, however, the pod is the old pod that just hasn't been replaced, yet. This change makes sure the number of nodes that should be running the daemon pod is the same as the number of nodes currently running the updated daemon pod.
From what I can tell, this follows the same logic used by kubectl when waiting on a rollout:
https://github.com/kubernetes/kubectl/blob/ac49920c0ccb0dd0899d5300fc43713ee2dfcdc9/pkg/polymorphichelpers/rollout_status.go#L107-L115.
ISSUE TYPE
COMPONENT NAME
k8s
ADDITIONAL INFORMATION