no connections for driver: rook-ceph.rbd.csi.ceph.com

[subhamkrai]

I tested my pr on rook side rook/rook#12286 which created the network fence cr, cr was created but when checked the ceph osd blocklist ls the IP was not present in the list and the logs from there csi addons controller says

2023-06-23T07:37:48.023Z	ERROR	Failed to get NetworkFenceClient	{"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"ip-10-0-160-193.ec2.internal"}, "namespace": "", "name": "ip-10-0-160-193.ec2.internal", "reconcileID": "d8da1419-7035-4bc9-87eb-ab4273e47fcb", "DriverName": "rook-ceph.rbd.csi.ceph.com", "CIDRs": ["100.64.0.7:0"], "error": "no connections for driver: rook-ceph.rbd.csi.ceph.com"}

kc get networkfences.csiaddons.openshift.io ip-10-0-160-193.ec2.internal 
NAME                           DRIVER                       CIDRS              FENCESTATE   AGE     RESULT
ip-10-0-160-193.ec2.internal   rook-ceph.rbd.csi.ceph.com   ["100.64.0.7:0"]   Fenced       3m12s   
~/go/src/github.com/rook/deploy/examples
srai@192 ~ (fix-node-loss-rbd) $ kc get pv | grep rbd-pvc
pvc-0c65191a-ceb8-4769-bf31-5c0c113c5e1d   1Gi        RWO            Delete           Bound    rook-ceph/rbd-pvc                                       rook-ceph-block            39m
~/go/src/github.com/rook/deploy/examples
srai@192 ~ (fix-node-loss-rbd) $ kc get pv pvc-0c65191a-ceb8-4769-bf31-5c0c113c5e1d -oyaml | grep imageName:
      imageName: csi-vol-f9ad9b3d-4b7f-40d1-9f47-0d172d5153ba
~/go/src/github.com/rook/deploy/examples
srai@192 ~ (fix-node-loss-rbd) $ kubectl rook
[csi-addons-logs.txt](https://github.com/csi-addons/kubernetes-csi-addons/files/11847034/csi-addons-logs.txt)
-ceph rbd status replicapool/csi-vol-f9ad9b3d-4b7f-40d1-9f47-0d172d5153ba
Warning: rook version 'rook: v1.11.0-alpha.0.449.ge5bd73104-dirty' is running a pre-release version of Rook.

Watchers:
	watcher=100.64.0.7:0/4143960263 client.16345 cookie=18446462598732840961

I'm uploading the complete logs of csi-addons-controller

[subhamkrai]

csi-addons-logs.txt

[Madhu-1]

Have you enabled csi addons sidecar in Rook?

[subhamkrai]

no, I used the default config, which is false. Okay, I'll test with true and update. Thanks

[subhamkrai]

any idea about this error

ERROR	Reconciler error	{"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"ip-10-0-182-92.ec2.internal"}, "namespace": "", "name": "ip-10-0-182-92.ec2.internal", "reconcileID": "e4d01d80-1669-408c-b943-8a77ddb1e214", "error": "rpc error: code = Internal desc = failed to fence CIDR block [\"100.64.0.7:0\"]: failed to add blocklist range \"100.64.0.7:0\": failed to blocklist IP \"100.64.0.7:0\": an error (exit status 13) occurred while running ceph args: [osd blocklist range add 100.64.0.7:0 157784760 --id csi-rbd-provisioner --keyfile=***stripped*** -m 172.30.202.117:6789,172.30.195.232:6789,172.30.7.57:6789] stderr: \"Error EACCES: access denied\\n\""}```

[subhamkrai]

rbd provisioner logs

E0626 11:53:24.113922       1 networkfence.go:75] Failed to fence cluster network: rpc error: code = Internal desc = failed to fence CIDR block ["100.64.0.7:0"]: failed to add blocklist range "100.64.0.7:0": failed to blocklist IP "100.64.0.7:0": an error (exit status 13) occurred while running ceph args: [osd blocklist range add 100.64.0.7:0 157784760 --id csi-rbd-provisioner --keyfile=***stripped*** -m 172.30.195.186:6789,172.30.188.51:6789,172.30.55.42:6789] stderr: "Error EACCES: access denied\n"

networkFence cr status

 message: 'rpc error: code = Internal desc = failed to fence CIDR block ["100.64.0.7:0"]:
    failed to add blocklist range "100.64.0.7:0": failed to blocklist IP "100.64.0.7:0":
    an error (exit status 13) occurred while running ceph args: [osd blocklist range
    add 100.64.0.7:0 157784760 --id csi-rbd-provisioner --keyfile=***stripped*** -m
    172.30.195.186:6789,172.30.188.51:6789,172.30.55.42:6789] stderr: "Error EACCES:
    access denied\n"'
  result: Failed

[Madhu-1]

@subhamkrai you might be using wrong user who doesn't have access to execute blocklist command, check running CLI command with same user from toolbox pod.

[nixpanic]

Please note that this likely is a Rook/Ceph-CSI problem, not so much csi-addons-kubernetes. Please move this issue to the appropriate GitHub project and close this one.