Implement dynamically adjustment of NodeDeleteDelayAfterTaint based on round trip time between CA and api-server #6019
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
kind/bug
x13n
requested changes
Aug 18, 2023
damikag
force-pushed
the
fix-rc-ca-sched
branch
from
83ef332 to
e472bb1
Compare
x13n
requested changes
Aug 31, 2023
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
damikag
force-pushed
the
fix-rc-ca-sched
branch
from
e472bb1 to
dd3258a
Compare
k8s-ci-robot
added
size/L
x13n
requested changes
Sep 1, 2023
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
damikag
force-pushed
the
fix-rc-ca-sched
branch
from
dd3258a to
f20b65c
Compare
|
/assign |
x13n
requested changes
Sep 4, 2023
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
damikag
force-pushed
the
fix-rc-ca-sched
branch
2 times, most recently
from
10577b4 to
e76699e
Compare
x13n
requested changes
Sep 7, 2023
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker_test.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker_test.go
Show resolved
Hide resolved
damikag
force-pushed
the
fix-rc-ca-sched
branch
2 times, most recently
from
8a8dfac to
e3bb2ce
Compare
x13n
reviewed
Sep 8, 2023
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker_test.go
Outdated
Show resolved
Hide resolved
cluster-autoscaler/core/scaledown/actuation/update_latency_tracker_test.go
Outdated
Show resolved
Hide resolved
damikag
force-pushed
the
fix-rc-ca-sched
branch
2 times, most recently
from
732a966 to
7c0bdec
Compare
Implement dynamically adjustment of NodeDeleteDelayAfterTaint based on round trip time between ca and apiserver
damikag
force-pushed
the
fix-rc-ca-sched
branch
from
7c0bdec to
42691d3
Compare
|
Come to think about it, the observed latency is something that may be worth having a metric for. But maybe that can be done as a follow-up change. /lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: damikag, x13n The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement dynamically adjustment of NodeDeleteDelayAfterTaint based on round trip time between CA and api-server
What type of PR is this?
/kind bug
What this PR does / why we need it:
Since CA and Scheduler are not synced, there could be race condition because scheduler might put pods on tainted nodes for deletion on scale down. Currently the probability of a race condition is reduced by waiting
NodeDeleteDelayAfterTaintamount of time before deleting a node after tainting. The value ofNodeDeleteDelayAfterTaintaffects scale down throughput so this PR makes it dynamic based on the round trip time between CA and api-server.Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?
When
--dynamic-node-delete-delay-after-taint-enabledset to true CA will adjustnodeDeleteDelayAfterTaintbased on the round-trip time between CA and api-server.nodeDeleteDelayAfterTaintwill be set to 3 times the round-trip time between CA and api-server.release-note
A new flag(
--dynamic-node-delete-delay-after-taint-enabled) is introduced which enables dynamic adjustment ofnodeDeleteDelayAfterTaintbased on the round-trip time between CA and api-server.Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: