Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

sameshai · 2024-08-12T13:55:57Z

What happened:
As per PR below PR If the feature-gate HonorPVReclaimPolicy is enabled, create a pvc with a delete relaim policy, then delete the pvc, the pv stuck in deleting status, the error message is: #1155

csi-mockplugin-0/[email protected]: I0206 11:08:41.536819       1 controller.go:1523] delete "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": failed to remove finalizer for persistentvolume: persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
csi-mockplugin-0/[email protected]: W0206 11:08:41.536850       1 controller.go:989] Retrying syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38", failure 0
csi-mockplugin-0/[email protected]: E0206 11:08:41.536880       1 controller.go:1007] error syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
I0206 19:08:

I did try this with IBM VPC Block CSI driver and 5.0.2 provisioner but seems without added the patch permission I am still able to delete PVC/PV and no error.

sameershaikh@Sameers-MBP Downloads % k describe pvc
Name:     pvc-block
Namespace:   default
StorageClass: ibmc-vpc-block-5iops-tier
Status:    Bound
Volume:    pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:    <none>
Annotations:  pv.kubernetes.io/bind-completed: yes
        pv.kubernetes.io/bound-by-controller: yes
        volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
        volume.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
Finalizers:  [kubernetes.io/pvc-protection]
Capacity:   10Gi
Access Modes: RWO
VolumeMode:  Filesystem
Used By:    <none>
Events:    <none>
sameershaikh@Sameers-MBP Downloads % k describe pv 
Name:       pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:      <none>
Annotations:    pv.kubernetes.io/provisioned-by: vpc.block.csi.ibm.io
          volume.kubernetes.io/provisioner-deletion-secret-name: 
          volume.kubernetes.io/provisioner-deletion-secret-namespace: 
Finalizers:    [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer]
StorageClass:   ibmc-vpc-block-5iops-tier
Status:      Bound
Claim:       default/pvc-block
Reclaim Policy:  Delete
Access Modes:   RWO
VolumeMode:    Filesystem
Capacity:     10Gi
Node Affinity:   
 Required Terms:  
  Term 0:    failure-domain.beta.kubernetes.io/region in [us-south]
          failure-domain.beta.kubernetes.io/zone in [us-south-3]
Message:      
Source:
  Type:       CSI (a Container Storage Interface (CSI) volume source)
  Driver:      vpc.block.csi.ibm.io
  FSType:      ext4
  VolumeHandle:   r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
  ReadOnly:     false
  VolumeAttributes:   clusterID=cqsr5km10bghfqv414rg
              failure-domain.beta.kubernetes.io/region=us-south
              failure-domain.beta.kubernetes.io/zone=us-south-3
              iops=3000
              storage.kubernetes.io/csiProvisionerIdentity=1723447099267-9715-vpc.block.csi.ibm.io
              tags=
              volumeCRN=crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
              volumeId=r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
Events:
 Type  Reason        Age  From                      Message
 ----  ------        ----  ----                      -------
 Normal VolumeMetaDataSaved 2m20s ibm-vpc-block-csi-controller-588c75b988-xpkbt Success

sameershaikh@Sameers-MBP file-csi-driver % k describe clusterRole vpc-block-provisioner-role
Name:     vpc-block-provisioner-role
Labels:    addonmanager.kubernetes.io/mode=Reconcile
       app=ibm-vpc-block-csi-driver
Annotations: razee.io/build-url: https://travis.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/builds/22121223
       razee.io/source-url: https://github.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/commit/d7c3406c7f992ef727f3e9e3ce60f4e33c4f20c2
       version: 5.2.20_579
PolicyRule:
 Resources                    Non-Resource URLs Resource Names Verbs
 ---------                    ----------------- -------------- -----
 persistentvolumes                []         []       [get list watch create delete]
 persistentvolumeclaims             []         []       [get list watch update]
 nodes                      []         []       [get list watch]
 secrets                     []         []       [get list watch]
 csinodes.storage.k8s.io             []         []       [get list watch]
 storageclasses.storage.k8s.io          []         []       [get list watch]
 volumeattachments.storage.k8s.io        []         []       [get list watch]
 configmaps                   []         []       [get list]
 volumesnapshotcontents.snapshot.storage.k8s.io []         []       [get list]
 volumesnapshots.snapshot.storage.k8s.io     []         []       [get list]
 events                     []         []       [list watch create update patch]
sameershaikh@Sameers-MBP file-csi-driver %



},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-attacher Update v1 2024-08-12 13:05:53 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:finalizers\":{\"v:\\\"external-attacher/vpc-block-csi-ibm-io\\\"\":{}}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:13.994031    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994053    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994064    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994078    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994098    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:13.994456    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:19.569875    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Watch close - *v1.PersistentVolume total 13 items received
I0812 13:09:24.904270    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:24.904325    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
W0812 13:09:24.966351    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].key: failure-domain.beta.kubernetes.io/region is deprecated since v1.17; use "topology.kubernetes.io/region" instead
W0812 13:09:24.966364    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[1].key: failure-domain.beta.kubernetes.io/zone is deprecated since v1.17; use "topology.kubernetes.io/zone" instead
I0812 13:09:24.966561    1 controller.go:1606] "PersistentVolume deleted succeeded" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966783    1 controller.go:1171] "handleProtectionFinalizer" PV="&PersistentVolume{ObjectMeta:{pvc-351e6980-ac05-4441-a482-6ea544b3aad5  3f0a9a59-3442-4171-b67a-efdd232c3664 19236 0 2024-08-12 07:31:18 +0000 UTC 2024-08-12 13:09:24 +0000 UTC 0xc0005b1210 map[] map[pv.kubernetes.io/provisioned-by:vpc.block.csi.ibm.io volume.kubernetes.io/provisioner-deletion-secret-name: volume.kubernetes.io/provisioner-deletion-secret-namespace:] [] [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer] [{csi-provisioner Update v1 2024-08-12 13:01:59 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pv.kubernetes.io/provisioned-by\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-name\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-namespace\":{}},\"f:finalizers\":{\"v:\\\"external-provisioner.volume.kubernetes.io/finalizer\\\"\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:capacity\":{},\"f:claimRef\":{\".\":{},\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{},\"f:resourceVersion\":{},\"f:uid\":{}},\"f:csi\":{\".\":{},\"f:driver\":{},\"f:fsType\":{},\"f:volumeAttributes\":{\".\":{},\"f:clusterID\":{},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:24.966803    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966813    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966820    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966830    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966844    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:24.966899    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:25.194175    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:25.194222    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:25.207594    1 controller.go:1569] "Failed to delete persistentvolume" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" err="persistentvolumes \"pvc-351e6980-ac05-4441-a482-6ea544b3aad5\" not found"
I0812 13:09:25.207641    1 controller.go:1007] "Retrying syncing volume" key="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" failures=0
E0812 13:09:25.207795    1 controller.go:1025] error syncing volume "pvc-351e6980-ac05-4441-a482-6ea544b3aad5": persistentvolumes "pvc-351e6980-ac05-4441-a482-6ea544b3aad5" not found
I0812 13:09:48.412100    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.CSINode total 9 items received
I0812 13:10:11.430774    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.PersistentVolumeClaim total 15 items received
I0812 13:11:05.631537    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Watch close - *v1.StorageClass total 11 items received
I0812 13:11:58.431219    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.Node total 21 items received

What you expected to happen:
I was expecting RBAC error

How to reproduce it:

Create PVC with external provisioner
Delete PVC which will trigger the PV deletion
There is no error even if we dont have patch permission.

Anything else we need to know?:

Environment:

Driver version:
Kubernetes version (use kubectl version):
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

The text was updated successfully, but these errors were encountered:

Madhu-1 · 2024-09-18T09:37:54Z

@xing-yang @carlory In the above code we are still doing the update operation and in the RBAC we don't have update access rather as part of #1155 to address the same issue, can you please check this one and let me know if we need to add RBAC for update or change the code in provisioner to use patch as well

https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L984

https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L1676

we still need an update for adding the annotation and for updating finalizers we can use the patch

Madhu-1 · 2024-09-18T09:38:19Z

/assign

carlory · 2024-09-18T10:26:00Z

I did try this with IBM VPC Block CSI driver and 5.0.2 provisioner but seems without added the patch permission I am still able to delete PVC/PV and no error.

I will try to reproduce it with hostpath driver.

we still need an update for adding the annotation and for updating finalizers we can use the patch

Agreed.

According to xing-yang's comment, to add a new RBAC rule, we need to bump the major version.

carlory · 2024-09-18T10:35:54Z

we need to add some e2e tests to detect a similar issue. should the new tests be added to the external-provisioner repo? cc @xing-yang @jsafrane

Madhu-1 · 2024-09-18T10:52:32Z

/unassign

sorry i was wrong i like to the PVC update code not the PV update, i don't see code in main that require PV update access.

carlory · 2024-09-18T11:07:01Z

@sameshai Can you execute the following command with your service account name? A service account may be bound with more than one cluster role.

kubectl auth can-i patch pv --as=system:serviceaccount:default:csi-hostpathplugin-sa

carlory · 2024-09-18T11:09:26Z

If the answer is "yes", it means that the external provisioner can patch the persistent volume resource even if the external-provisioner cluster role does not have permission. It is allowed via other cluster roles.

k8s-triage-robot · 2024-12-17T11:12:51Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-ci-robot assigned Madhu-1 Sep 18, 2024

k8s-ci-robot unassigned Madhu-1 Sep 18, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

sameshai commented Aug 12, 2024

Madhu-1 commented Sep 18, 2024

Madhu-1 commented Sep 18, 2024

carlory commented Sep 18, 2024 •

edited

Loading

carlory commented Sep 18, 2024

Madhu-1 commented Sep 18, 2024

carlory commented Sep 18, 2024

carlory commented Sep 18, 2024

k8s-triage-robot commented Dec 17, 2024

Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

Comments

sameshai commented Aug 12, 2024

Madhu-1 commented Sep 18, 2024

Madhu-1 commented Sep 18, 2024

carlory commented Sep 18, 2024 • edited Loading

carlory commented Sep 18, 2024

Madhu-1 commented Sep 18, 2024

carlory commented Sep 18, 2024

carlory commented Sep 18, 2024

k8s-triage-robot commented Dec 17, 2024

carlory commented Sep 18, 2024 •

edited

Loading