Issue with applying linstor-csi

[ashkraba]

What happens:
Have a separate linstor cluster and try to use it in Kubernetes for PVC.
When try to install the manifest all pods start successfully accept linstor-csi-node get the next errors in the container csi-node-driver-registrar:

I0304 12:48:28.162239       1 main.go:112] Version: v2.0.1-0-g1aaa13ae
I0304 12:48:28.162268       1 main.go:122] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0304 12:48:28.162279       1 connection.go:151] Connecting to unix:///csi/csi.sock
I0304 12:48:28.163104       1 main.go:129] Calling CSI driver to discover driver name
I0304 12:48:28.163127       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0304 12:48:28.163133       1 connection.go:181] GRPC request: {}
I0304 12:48:28.165155       1 connection.go:183] GRPC response: {"name":"linstor.csi.linbit.com","vendor_version":"v1.0.0-3fbda5a7e31517786796ae4834c159caa9ad8ac7"}
I0304 12:48:28.165220       1 connection.go:184] GRPC error: <nil>
I0304 12:48:28.165225       1 main.go:139] CSI driver name: "linstor.csi.linbit.com"
I0304 12:48:28.165278       1 node_register.go:55] Starting Registration Server at: /registration/linstor.csi.linbit.com-reg.sock
I0304 12:48:28.165422       1 node_register.go:64] Registration Server started at: /registration/linstor.csi.linbit.com-reg.sock
I0304 12:48:28.165467       1 node_register.go:86] Skipping healthz server because port set to: 0
I0304 12:48:29.172598       1 main.go:79] Received GetInfo call: &InfoRequest{}
I0304 12:48:29.182034       1 main.go:89] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unknown desc = failed to retrieve node topology: failed to get storage pools for node: 404 Not Found,}
E0304 12:48:29.182067       1 main.go:91] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unknown desc = failed to retrieve node topology: failed to get storage pools for node: 404 Not Found, restarting registration container.

Linstor nodes are pingable from kubernetes cluster nodes.
Versions
Kubernetes 1.26.2
linstor 1.16.0
Used manifest - https://github.com/piraeusdatastore/linstor-csi/blob/master/examples/k8s/deploy/linstor-csi-1.19.yaml
only the next is updated in ENV variables

- name: LINSTOR_IP
   value: "http://10.0.51.10:3370"

Linstor controller properties:

root@node1:~# linstor controller list-properties
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Key                                       ┊ Value                                                                                       ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ Cluster/LocalID                           ┊ 0deed51f-5e52-4dc8-8238-195e46db25ce                                                        ┊
┊ DrbdOptions/AutoEvictAllowEviction        ┊ True                                                                                        ┊
┊ DrbdOptions/auto-add-quorum-tiebreaker    ┊ True                                                                                        ┊
┊ DrbdOptions/auto-diskful-allow-cleanup    ┊ True                                                                                        ┊
┊ DrbdOptions/auto-quorum                   ┊ io-error                                                                                    ┊
┊ DrbdOptions/auto-verify-algo-allowed-list ┊ crct10dif-pclmul;crct10dif-generic;sha384-generic;sha512-generic;sha256-generic;md5-generic ┊
┊ NetCom/DebugSslConnector/BindAddress      ┊ ::0                                                                                         ┊
┊ NetCom/DebugSslConnector/Enabled          ┊ true                                                                                        ┊
┊ NetCom/DebugSslConnector/KeyPasswd        ┊ linstor                                                                                     ┊
┊ NetCom/DebugSslConnector/KeyStore         ┊ ssl/keystore.jks                                                                            ┊
┊ NetCom/DebugSslConnector/KeyStorePasswd   ┊ linstor                                                                                     ┊
┊ NetCom/DebugSslConnector/Port             ┊ 3373                                                                                        ┊
┊ NetCom/DebugSslConnector/SslProtocol      ┊ TLSv1.2                                                                                     ┊
┊ NetCom/DebugSslConnector/TrustStore       ┊ ssl/certificates.jks                                                                        ┊
┊ NetCom/DebugSslConnector/TrustStorePasswd ┊ linstor                                                                                     ┊
┊ NetCom/DebugSslConnector/Type             ┊ SSL                                                                                         ┊
┊ NetCom/PlainConnector/BindAddress         ┊                                                                                             ┊
┊ NetCom/PlainConnector/Enabled             ┊ true                                                                                        ┊
┊ NetCom/PlainConnector/Port                ┊ 3376                                                                                        ┊
┊ NetCom/PlainConnector/Type                ┊ Plain                                                                                       ┊
┊ NetCom/SslConnector/BindAddress           ┊                                                                                             ┊
┊ NetCom/SslConnector/Enabled               ┊ true                                                                                        ┊
┊ NetCom/SslConnector/KeyPasswd             ┊ linstor                                                                                     ┊
┊ NetCom/SslConnector/KeyStore              ┊ ssl/keystore.jks                                                                            ┊
┊ NetCom/SslConnector/KeyStorePasswd        ┊ linstor                                                                                     ┊
┊ NetCom/SslConnector/Port                  ┊ 3377                                                                                        ┊
┊ NetCom/SslConnector/SslProtocol           ┊ TLSv1.2                                                                                     ┊
┊ NetCom/SslConnector/TrustStore            ┊ ssl/certificates.jks                                                                        ┊
┊ NetCom/SslConnector/TrustStorePasswd      ┊ linstor                                                                                     ┊
┊ NetCom/SslConnector/Type                  ┊ SSL                                                                                         ┊
┊ defaultDebugSslConnector                  ┊ DebugSslConnector                                                                           ┊
┊ defaultPlainConSvc                        ┊ PlainConnector                                                                              ┊
┊ defaultSslConSvc                          ┊                                                                                             ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[WanzenBug]

It looks like there is no LINSTOR node on the kubernetes node. The CSI plugin expects a LINSTOR Satellite with the same name as the name of the kubernetes node it is running on.

[ashkraba]

Hello @WanzenBug
Thanks a lot for your response it help, but now I get the next error when try to create a test pod:

Warning  FailedMount         21s                    kubelet                  Unable to attach or mount volumes: unmounted volumes=[demo-vol], unattached volumes=[kube-api-access-q2lxn demo-vol]: timed out waiting for the condition
  Warning  FailedAttachVolume  12s                    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-a4eaba2c-5c67-4f60-b914-a431d5097d18" : rpc error: code = Internal desc = ControllerPublishVolume failed for pvc-a4eaba2c-5c67-4f60-b914-a431d5097d18: Message: 'Satellite 'tech-worker2' does not support the following layers: [DRBD]'; Details: 'Node(s): 'tech-worker2', Resource: 'pvc-a4eaba2c-5c67-4f60-b914-a431d5097d18''; Reports: '[64071B4A-00000-000023]'

Linstor config:

root@node1:~# linstor sp list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node         ┊ Driver   ┊ PoolName               ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ node1        ┊ DISKLESS ┊                        ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ node2        ┊ DISKLESS ┊                        ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ node3        ┊ DISKLESS ┊                        ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ tech-worker1 ┊ DISKLESS ┊                        ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ tech-worker2 ┊ DISKLESS ┊                        ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ tech-worker3 ┊ DISKLESS ┊                        ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ pool1                ┊ node1        ┊ LVM      ┊ linstor-vg/linstor-lv1 ┊     1.55 TiB ┊      1.75 TiB ┊ False        ┊ Ok    ┊            ┊
┊ pool1                ┊ node2        ┊ LVM      ┊ linstor-vg/linstor-lv1 ┊     1.55 TiB ┊      1.75 TiB ┊ False        ┊ Ok    ┊            ┊
┊ pool1                ┊ node3        ┊ LVM      ┊ linstor-vg/linstor-lv1 ┊     1.55 TiB ┊      1.75 TiB ┊ False        ┊ Ok    ┊            ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@node1:~# linstor rg list
╭────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceGroup         ┊ SelectFilter                    ┊ VlmNrs ┊ Description ┊
╞════════════════════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp            ┊ PlaceCount: 2                   ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ linstor-basic-storage ┊ PlaceCount: 3                   ┊ 0      ┊             ┊
┊                       ┊ StoragePool(s): pool1           ┊        ┊             ┊
┊                       ┊ LayerStack: ['DRBD', 'STORAGE'] ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ rg1                   ┊ PlaceCount: 3                   ┊        ┊             ┊
┊                       ┊ StoragePool(s): pool1           ┊        ┊             ┊
╰────────────────────────────────────────────────────────────────────────────────╯

Storageclass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-basic-storage
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
parameters:
  linstor.csi.linbit.com/placementCount: "3"
  linstor.csi.linbit.com/storagePool: pool1
  linstor.csi.linbit.com/resourceGroup: linstor-basic-storage

PVC:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: linstor-basic-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10M

POD:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: default
spec:
  containers:
  - name: demo-pod-0
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 1000s; done"]
    volumeMounts:
      - mountPath: "/data"
        name: demo-vol
  volumes:
    - name: demo-vol
      persistentVolumeClaim:
        claimName: test-claim

Should I install DRBD on Kubernetes nodes too?
Thanks a lot in advance.

[WanzenBug]

Should I install DRBD on Kubernetes nodes too?

Yes, you need LINSTOR Satellite and also DRBD

[Zeratyl06]

Do I understand correctly that for the CSI driver to work, it is necessary to include the k8s worker node in the Linstor cluster, like satellites? Is there an option to configure the CSI driver without including the k8s worker node in the Linstor cluster? We have an external Linstor cluster.

[WanzenBug]

Yes, you need both CSI and a running satellite. That satellite can be part of your existing cluster without any storage of it's own.

See https://github.com/piraeusdatastore/piraeus-operator/blob/v2/docs/how-to/external-controller.md for an example how to achieve that with the operator.

[Zeratyl06]

If I don't use an operator, is it enough to install the necessary deb packages (Ubuntu) and manually add the k8s worker node host to the Linstor cluster? Need some additional k8s settings?

[WanzenBug]

Yes, manually installing also works. One thing to keep in mind: the CSI Driver expects the node names in kubernetes (kubectl get nodes) to match the node names in LINSTOR (linstor node list). Keep that in mind.

[Zeratyl06]

Thank you!