CA fails to scale-up or cancel in-progress scale down when there are un-schedulable pods #4456

varkey · 2021-11-10T09:35:18Z

Which component are you using?: cluster-autoscaler

What version of the component are you using?: cluster-autoscaler

Component version: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.20.0

What k8s version are you using (kubectl version)?: v1.20.10

kubectl version Output

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.10", GitCommit:"8152330a2b6ca3621196e62966ef761b8f5a61bb", GitTreeState:"clean", BuildDate:"2021-08-11T18:00:37Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?:

KOPS (v1.20.2) based cluster deployed on AWS

CA is run with the following arguments

- ./cluster-autoscaler
- --v=2
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --skip-nodes-with-system-pods=false
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/env-name
- --scale-down-utilization-threshold=0.6
- --balance-similar-node-groups
- --ignore-taint=node-upgrading
- --max-graceful-termination-sec=10800
- --scale-down-delay-after-delete=5m
- --max-empty-bulk-delete=2
- --scale-down-delay-after-add=5m

The nodes span multiple AZs, but we use separate ASGs for each AZ. Also there are several node groups with different taints so that we can place applications in dedicated node groups.

node-group-1-az1
node-group-1-az2
node-group-1-az3
node-group-2-az1
node-group-2-az2
node-group-2-az3
...
...
...

In the above case, all node-group-1-xx node groups have the same taints, instance types etc.

We use HPA for many of our applications, and can dynamically scale up and down the number of replicas. As can be seen below, the scale up happen every hour and then the load subsides, and the HPA scales down the number of replicas.

Due to the nature of our applications, some of them are configured with a termination grace period of up to 3 hours. Since we do not want CA to prematurely terminate a node, we have configured CA with --max-graceful-termination-sec=10800 which is the maximum termination grace period across all pods in the cluster.

What did you expect to happen?:

When an HPA scales up a deployment, CA should scale up the appropriate node group without any delay if there is no available capacity. The pods will fit any of the nodes coming under node-group-x-az1 or node-group-x-az2 or node-group-x-az3

What happened instead?:

We noticed that CA does not trigger a scale up always and the pod remains in a Pending state for 20-30 min or at times even more.

I1105 11:06:16.999039       1 event.go:291] "Event occurred" object="default/app-1-cf767bfc7" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: app-1-cf767bfc7-jtcc4"
I1105 11:26:37.889440       1 scheduler.go:604] "Successfully bound pod to node" pod="default/app-1-cf767bfc7-jtcc4" node="ip-10-10-72-247.ec2.internal" evaluatedNodes=93 feasibleNodes=1

In the above case, the pod remained in Pending state for 20 minutes and CA was logging messages such as below repeatedly.

I1105 11:07:58.593642 1 filter_out_schedulable.go:157] Pod default.app-1-cf767bfc7-m5672 marked as unschedulable can be scheduled on node template-node-for-node-group-2-az2-6211936831103862438-0. Ignoring in scale up.
I1105 11:07:58.593798 1 filter_out_schedulable.go:157] Pod default.app-1-cf767bfc7-jtcc4 marked as unschedulable can be scheduled on node template-node-for-node-group-2-az2-6211936831103862438-0. Ignoring in scale up.
I1105 11:07:58.593818 1 filter_out_schedulable.go:170] 0 pods were kept as unschedulable based on caching
I1105 11:07:58.593825 1 filter_out_schedulable.go:171] 2 pods marked as unschedulable can be scheduled.

From a kube-scheduler perspective, the log said

I1105 11:06:17.027552 1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="default/app-1-cf767bfc7-jtcc4" err="
0/93 nodes are available: 
 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1636105837}, that the pod didn't tolerate, 
 1 node(s) had taint {dedicated: app-group-2}, that the pod didn't tolerate, 
 1 node(s) had taint {dedicated: app-group-3}, that the pod didn't tolerate, 
 1 node(s) had taint {dedicated: app-group-4}, that the pod didn't tolerate, 
 1 node(s) had taint {dedicated: app-group-5}, that the pod didn't tolerate, 
 10 node(s) had taint {dedicated: app-group-6}, that the pod didn't tolerate, 
 17 Insufficient cpu, 
 2 node(s) had taint {dedicated: app-group-7}, that the pod didn't tolerate, 
 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 
 33 node(s) had taint {dedicated: app-group-8}, that the pod didn't tolerate, 
 6 node(s) had taint {dedicated: app-group-9}, that the pod didn't tolerate, 
 8 node(s) didn't match Pod's node affinity, 
 9 node(s) had taint {dedicated: app-group-10}, that the pod didn't tolerate."

Investigating further on the 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1636105837} message, we noticed there was a scale down in progress which had been initiated around 76 min prior to the new un-schedulable pod being created. The node in question was ip-10-10-77-101.ec2.internal which was the same node that had the ToBeDeletedByClusterAutoscaler 1636105837 taint. The node also belonged to one of the node-groups where the pod could have been scheduled.

I1105 09:50:37.035315       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"3015546a-9c9e-4c53-b568-efba24823b6d", APIVersion:"v1", ResourceVersion:"1388297360", FieldPath:""}): type: 'Normal' reason: 'ScaleDown' Scale-down: removing node ip-10-10-77-101.ec2.internal, utilization: {0.33875 0.17038226983488522 0 cpu 0.33875}, pods to reschedule: default/app-2-848fdf7fcc-jmf4z,default/app-3-7768795d75-skgtk,default/app-4-66fc8d76dc-58vkv,default/app-5-7d95497c9c-8zzl4
I1105 09:50:37.047558       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-10-77-101.ec2.internal", UID:"da45409f-0ea3-4e5e-b63e-1c657d30e473", APIVersion:"v1", ResourceVersion:"1388297432", FieldPath:""}): type: 'Normal' reason: 'ScaleDown' marked the node as toBeDeleted/unschedulable
I1105 12:50:41.577858       1 scale_down.go:1266] All pods removed from ip-10-10-77-101.ec2.internal
I1105 12:50:43.216890       1 scale_down.go:1054] Scale-down: removing empty node ip-10-10-77-101.ec2.internal
I1105 12:50:43.216947       1 delete.go:111] ToBeDeletedByClusterAutoscaler already present on node ip-10-10-77-101.ec2.internal, taint: {ToBeDeletedByClusterAutoscaler 1636105837 NoSchedule <nil>}
I1105 12:50:43.217131       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"3015546a-9c9e-4c53-b568-efba24823b6d", APIVersion:"v1", ResourceVersion:"1388523279", FieldPath:""}): type: 'Normal' reason: 'ScaleDownEmpty' Scale-down: removing empty node ip-10-10-77-101.ec2.internal

CA seem to be thinking that the above node can be used to schedule the un-schedulable pod but it is actually being deleted due to under-utilisation. Due to a max termination grace period of 3 hours (and cause the pods already running on the node failed to grace-fully terminate before that), CA had to wait 3 hours for all pods to be removed from the affected node. During this interval, there was no scale up triggered by CA even though there were pods that needed to be scheduled.

To summarise, it appears CA will not trigger a scale up in node groups where a scale down is in progress, even if there are Pending pods waiting to be scheduled. This may not be a real issue if termination grace period isn't this long.

Anyway, the current behaviour doesn't seem right as CA thinks that the pod is schedulable and does nothing, but there is no capacity and should have triggered a scale-up. I am not sure if that is efficient though, perhaps CA should just cancel the in-progress scale down by removing the ToBeDeletedByClusterAutoscaler taint allowing the Pending pod to be scheduled immediately? I would think this is more efficient than allowing a new node to launch.

We see this behaviour happen multiple times a day in our prod environments. At times, capacity is made available after couple of minutes without a scale up possibly due to reduction in number of replicas of other services/pods running on the same node groups. But the fundamental issue of CA failing to scale up or cancel the in-progress scale down is a problem.

How to reproduce it (as minimally and precisely as possible):

This should be re-produceable whenever there are Pending pods to be scheduled and there is a scale down in progress due to under-utilisation needing eviction of existing pods. If CA and the pods are configured with a high max termination grace period and the pods take a while to gracefully terminate, CA will not trigger a scale up or cancel the in-progress scale down.

Anything else we need to know?:

The text was updated successfully, but these errors were encountered:

naveenb29 · 2021-12-27T23:55:24Z

I am facing the same issue as @varkey . Also observed that when the existing CA pod is deleted manually , forcing a new CA pod comes up the pending pods get scheduled.

Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.7-eks-d88609", GitCommit:"d886092805d5cc3a47ed5cf0c43de38ce442dfcb", GitTreeState:"clean", BuildDate:"2021-07-31T00:29:12Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

varkey · 2022-01-10T05:57:51Z

Also observed that when the existing CA pod is deleted manually , forcing a new CA pod comes up the pending pods get scheduled.

We've noticed this as well, re-creating a new CA pod seem to reset the state somehow. It seem to forget about the node which is being drained or at least it no longer waits for the remaining termination grace period.

k8s-triage-robot · 2022-04-10T06:48:00Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

fookenc · 2022-04-12T23:11:06Z

I believe that this issue is similar to #4051 and #3949. There is a PR #4211 which removes ToBeDeletedByClusterAutoscaler tainted nodes from being considered upcoming nodes. The change allows scaling operations to proceed as expected.

varkey · 2022-05-11T14:27:12Z

/remove-lifecycle stale

k8s-triage-robot · 2022-08-09T14:56:23Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2022-09-08T15:19:04Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2022-10-08T15:58:12Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot · 2022-10-08T15:58:16Z

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

x13n · 2022-10-14T08:01:34Z

/reopen

k8s-ci-robot · 2022-10-14T08:01:39Z

@x13n: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2022-11-13T08:15:42Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot · 2022-11-13T08:15:48Z

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

varkey · 2022-11-14T07:21:20Z

/remove-lifecycle rotten
/reopen

k8s-ci-robot · 2022-11-14T07:21:25Z

@varkey: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2023-02-12T07:25:43Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2023-03-14T07:58:58Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

vadasambar · 2023-03-21T05:21:43Z

/remove-lifecycle rotten

- this is a follow-up to kubernetes#5054 - this might fix kubernetes#4456 Signed-off-by: vadasambar <[email protected]>

- this is a follow-up to kubernetes#5054 - this might fix kubernetes#4456 Signed-off-by: vadasambar <[email protected]> fix: make `HasInstance` in aws provider thread-safe Signed-off-by: vadasambar <[email protected]>

- this is a follow-up to kubernetes#5054 - this might fix kubernetes#4456 fix: make `HasInstance` in aws provider thread-safe Signed-off-by: vadasambar <[email protected]> (cherry picked from commit 1cb55fe)

- this is a follow-up to kubernetes#5054 - this might fix kubernetes#4456 Signed-off-by: vadasambar <[email protected]> fix: make `HasInstance` in aws provider thread-safe Signed-off-by: vadasambar <[email protected]> (cherry picked from commit 1cb55fe)

- this is a follow-up to kubernetes#5054 - this might fix kubernetes#4456 Signed-off-by: vadasambar <[email protected]> fix: make `HasInstance` in aws provider thread-safe Signed-off-by: vadasambar <[email protected]>

chrisyao2022 · 2023-11-07T22:16:36Z

/remove-lifecycle rotten

chrisyao2022 · 2023-11-07T22:17:44Z

/reopen

k8s-ci-robot · 2023-11-07T22:17:49Z

@chrisyao2022: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

- this is a follow-up to kubernetes#5054 - this might fix kubernetes#4456 Signed-off-by: vadasambar <[email protected]> fix: make `HasInstance` in aws provider thread-safe Signed-off-by: vadasambar <[email protected]> (cherry picked from commit 1cb55fe)

varkey added the kind/bug Categorizes issue or PR as related to a bug. label Nov 10, 2021

jbartosik added the area/cluster-autoscaler label Nov 10, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 10, 2022

fookenc mentioned this issue Apr 27, 2022

Subtract toBeDeleted nodes from number of upcoming nodes; cleanup toBeDeleted taints from all nodes, not only ready ones #4211

Closed

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 11, 2022

fookenc mentioned this issue May 17, 2022

Adding support for identifying nodes that have been deleted from cloud provider that are still registered within Kubernetes #4896

Merged

fookenc mentioned this issue Jul 27, 2022

Identifying cloud provider deleted nodes #5054

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 9, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 8, 2022

k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 8, 2022

k8s-ci-robot reopened this Oct 14, 2022

k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 13, 2022

k8s-ci-robot reopened this Nov 14, 2022

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Nov 14, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 12, 2023

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 14, 2023

vadasambar mentioned this issue Mar 14, 2023

Cluster Autoscaler Forgets Nodes Scheduled for Deletion during Restart #5048

Open

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Mar 21, 2023

vadasambar added a commit to vadasambar/autoscaler that referenced this issue Mar 29, 2023

fix: implement function to identify if node is present in aws

4dcbc14

- this is a follow-up to kubernetes#5054 - this might fix kubernetes#4456 Signed-off-by: vadasambar <[email protected]>

vadasambar mentioned this issue Mar 29, 2023

fix: implement function to identify if node is present in aws #5632

Merged

mimmus mentioned this issue Apr 5, 2023

Nodes tainted with ToBeDeletedByClusterAutoscaler remain in the cluster blocking cluster-autoscaler normal operations #5657

Closed

k8s-ci-robot closed this as completed in #5632 Apr 10, 2023

MaxFedotov mentioned this issue Apr 15, 2024

fix: implement function to identify if node is present in cluster-api #6708

Merged

claassen mentioned this issue May 14, 2024

[BUG] Cluster autoscaler bug requires Azure specific implementation to resolve Azure/AKS#4286

Open

Bryce-Soghigian mentioned this issue Jun 21, 2024

feat: Azure Provider HasInstance implementation #6956

Merged

alfredkrohmer mentioned this issue Aug 12, 2024

fix: implement HasInstance() for OCI providers #7154

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CA fails to scale-up or cancel in-progress scale down when there are un-schedulable pods #4456

CA fails to scale-up or cancel in-progress scale down when there are un-schedulable pods #4456

varkey commented Nov 10, 2021

naveenb29 commented Dec 27, 2021 •

edited

Loading

varkey commented Jan 10, 2022

k8s-triage-robot commented Apr 10, 2022

fookenc commented Apr 12, 2022

varkey commented May 11, 2022

k8s-triage-robot commented Aug 9, 2022

k8s-triage-robot commented Sep 8, 2022

k8s-triage-robot commented Oct 8, 2022

k8s-ci-robot commented Oct 8, 2022

x13n commented Oct 14, 2022

k8s-ci-robot commented Oct 14, 2022

k8s-triage-robot commented Nov 13, 2022

k8s-ci-robot commented Nov 13, 2022

varkey commented Nov 14, 2022

k8s-ci-robot commented Nov 14, 2022

k8s-triage-robot commented Feb 12, 2023

k8s-triage-robot commented Mar 14, 2023

vadasambar commented Mar 21, 2023

chrisyao2022 commented Nov 7, 2023

chrisyao2022 commented Nov 7, 2023

k8s-ci-robot commented Nov 7, 2023

CA fails to scale-up or cancel in-progress scale down when there are un-schedulable pods #4456

CA fails to scale-up or cancel in-progress scale down when there are un-schedulable pods #4456

Comments

varkey commented Nov 10, 2021

naveenb29 commented Dec 27, 2021 • edited Loading

varkey commented Jan 10, 2022

k8s-triage-robot commented Apr 10, 2022

fookenc commented Apr 12, 2022

varkey commented May 11, 2022

k8s-triage-robot commented Aug 9, 2022

k8s-triage-robot commented Sep 8, 2022

k8s-triage-robot commented Oct 8, 2022

k8s-ci-robot commented Oct 8, 2022

x13n commented Oct 14, 2022

k8s-ci-robot commented Oct 14, 2022

k8s-triage-robot commented Nov 13, 2022

k8s-ci-robot commented Nov 13, 2022

varkey commented Nov 14, 2022

k8s-ci-robot commented Nov 14, 2022

k8s-triage-robot commented Feb 12, 2023

k8s-triage-robot commented Mar 14, 2023

vadasambar commented Mar 21, 2023

chrisyao2022 commented Nov 7, 2023

chrisyao2022 commented Nov 7, 2023

k8s-ci-robot commented Nov 7, 2023

naveenb29 commented Dec 27, 2021 •

edited

Loading