Run `livenessprobe` as non-root + Run all possible sidecars as nonRootGroup #65

prankulmahajan · 2024-01-08T18:24:17Z

Detailed analysis and test results in GHE https://github.ibm.com/alchemy-containers/armada-storage/issues/5301

This PR aims to do the following,

In deployment run all the possible sidecars as NonRootGroup. Currently, though the users are non-root, the group is still being used as root.
In deployment run livenessprobe sidecar as non-root user/group.

To achieve the 2nd point, had to change file permissions of CSI socket created by node server.

# Default permissions of csi socket inside node server -- (User 0, Group 0, Permissions 755) -- Only the root user has the permissions to access socket
[root@ibm-vpc-file-csi-node-hxm8w /]# id
uid=0(root) gid=0(root) groups=0(root)
[root@ibm-vpc-file-csi-node-hxm8w /]# 
[root@ibm-vpc-file-csi-node-hxm8w /]# ls -la /csi/
total 8
drwxr-xr-x 2 root root 4096 Jan 23 07:29 .
drwxr-xr-x 1 root root 4096 Jan 23 07:29 ..
srwxr-xr-x 1 root root    0 Jan 23 07:29 csi-vpc-file.sock

The socket file should have read and write permissions for access. i.e min access required is 600/660 depending on who is accessing the socket file

This PR changes the group of the csi socket from root to non-root(set 2121 in deployment configMap) and update permissions to 660. Setting "660" allows the root user and the non-root group to access the socket as expected.

# Permissions post changes - node server
root@ibm-vpc-file-csi-node-l74rw:/# id
uid=0(root) gid=0(root) groups=0(root)
root@ibm-vpc-file-csi-node-l74rw:/# ls -la /csi/
total 8
drwxr-xr-x 2 root root 4096 Jan 23 07:38 .
drwxr-xr-x 1 root root 4096 Jan 23 07:38 ..
srw-rw---- 1 root 2121    0 Jan 23 07:38 csi-vpc-file.sock

# Controller server
❯ kubectl exec ibm-vpc-file-csi-controller-7b4bcf6cb9-tqxpn -n kube-system -c iks-vpc-file-driver -it -- bash
groups: cannot find name for group ID 2121
I have no name!@ibm-vpc-file-csi-controller-7b4bcf6cb9-tqxpn:/$ id
uid=2121 gid=2121 groups=2121
I have no name!@ibm-vpc-file-csi-controller-7b4bcf6cb9-tqxpn:/$ cd /csi/
I have no name!@ibm-vpc-file-csi-controller-7b4bcf6cb9-tqxpn:/csi$ ls -la
total 8
drwxrwxrwx 2 root root 4096 Jan 23 07:38 .
drwxr-xr-x 1 root root 4096 Jan 23 07:38 ..
srwxr-xr-x 1 2121 2121    0 Jan 23 07:38 csi-vpc-file.sock
srwxr-xr-x 1 2121 2121    0 Jan 23 07:38 providerfile.sock

Test functionality and logs

❯ kubectl get pods -n kube-system -o wide| grep file
ibm-vpc-file-csi-controller-7b4bcf6cb9-tqxpn          5/5     Running   0          2m36s   172.17.35.19    10.240.0.8   <none>           <none>
ibm-vpc-file-csi-controller-7b4bcf6cb9-xt2vh          5/5     Running   0          2m36s   172.17.20.221   10.240.0.7   <none>           <none>
ibm-vpc-file-csi-node-l74rw                           4/4     Running   0          2m37s   172.17.35.18    10.240.0.8   <none>           <none>
ibm-vpc-file-csi-node-wnprn                           4/4     Running   0          2m37s   172.17.20.220   10.240.0.7   <none>           <none>

# Create PVC and attach to pod
❯ kubectl get pvc
NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
my-pvc   Bound    pvc-7e09f908-b11c-4542-8356-5b1743bdf16f   20Gi       RWX            ibmc-vpc-file-dp2   2m7s
❯ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-dep-5c76969568-fqdkw   1/1     Running   0          12s
❯ 
❯ kubectl exec my-dep-5c76969568-fqdkw -it -- bash
root@my-dep-5c76969568-fqdkw:/# df -h
Filesystem                                         Size  Used Avail Use% Mounted on
overlay                                             98G  9.2G   84G  10% /
tmpfs                                               64M     0   64M   0% /dev
tmpfs                                              7.8G     0  7.8G   0% /sys/fs/cgroup
10.240.0.16:/4e2e6e1d_ca0a_431c_94a0_228f62447686   20G  320K   20G   1% /mount-path
/dev/vda2                                           98G  9.2G   84G  10% /etc/hosts
shm                                                 64M     0   64M   0% /dev/shm
tmpfs                                               13G   16K   13G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                              7.8G     0  7.8G   0% /proc/acpi
tmpfs                                              7.8G     0  7.8G   0% /proc/scsi
tmpfs                                              7.8G     0  7.8G   0% /sys/firmware
root@my-dep-5c76969568-fqdkw:/# cd mount-path/
root@my-dep-5c76969568-fqdkw:/mount-path# echo "HI" > test.txt
root@my-dep-5c76969568-fqdkw:/mount-path# ls -la
total 8
drwxrwxr-x 2 nobody 4294967294 4096 Jan 23 07:46 .
drwxr-xr-x 1 root   root       4096 Jan 23 07:45 ..
-rw-r--r-- 1 nobody 4294967294    3 Jan 23 07:46 test.txt

contsto · 2024-01-17T11:22:18Z

👍 Coverage increased from [%] to [68.175%]

contsto · 2024-01-17T11:25:36Z

👍 Coverage increased from [%] to [68.725%]

contsto · 2024-01-19T05:05:52Z

👍 Coverage increased from [%] to [68.3375%]

contsto · 2024-01-19T10:01:06Z

👍 Coverage increased from [%] to [68.0125%]

contsto · 2024-01-22T12:47:37Z

👍 Coverage increased from [%] to [68.9875%]

deploy/kubernetes/driver/kubernetes/manifests/config-map.yaml

pkg/ibmcsidriver/server.go

ambiknai

Minor comments

contsto · 2024-01-23T04:42:57Z

👍 Coverage increased from [%] to [69.35%]

contsto · 2024-01-23T12:38:18Z

👍 Coverage increased from [%] to [69.55%]

pkg/ibmcsidriver/server.go

sameshai

@prankulmahajan in case of any error on this setup the driver wont be running ?

Signed-off-by: Prankul <[email protected]>

contsto · 2024-08-22T10:51:14Z

👍 Coverage increased from [%] to [82.0222%]

contsto · 2024-08-22T10:51:16Z