can't see any content in mnt #818

pbs-jyu · 2024-08-09T22:36:37Z

What happened: installed smb csi in two cluster, used the same secret, same code created sc, pvc, and a test pod. in one cluster, I can see all the content under /mnt/share in the pod, but did not see anything in the other cluster. I can reach the SMB server, ping works from the problematic cluster nodes.

What you expected to happen: Contents are definitely there, existing. should see my content after exec -it the pod; when run ls.

How to reproduce it: run the same code over and over again. even tried different share from the same SMB server, as long as tested in problematic cluster A, I just can't see it.

Anything else we need to know?:

Environment:

image: registry.k8s.io/sig-storage/smbplugin:v1.15.0

CSI Driver version: 1.15.0
Kubernetes version (use kubectl version): v1.29.7
OS (e.g. from /etc/os-release):
- PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
Kernel (e.g. uname -a): 127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Install tools: [Microk8s](my codes.txt)
Others:

The text was updated successfully, but these errors were encountered:

andyzhangx · 2024-08-10T09:07:17Z

what's the df -h or mount | grep cifs return in the pod in problem? is there any cifs mount on the node in problem, you could ssh to the agent node, and run df -h or mount | grep cifs to verify
@pbs-jyu

pbs-jyu · 2024-08-10T19:07:49Z

hi, Andy

Thanks for your response.
k exec -it test-pfs-lab-jyu -n devops -- sh

cd /mnt/pfs-lab

ls

df -h

Filesystem Size Used Avail Use% Mounted on
overlay 145G 33G 106G 24% /
tmpfs 64M 0 64M 0% /dev
/dev/mapper/vgubuntu-root 145G 33G 106G 24% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 7.7G 12K 7.7G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /proc/scsi
tmpfs 3.9G 0 3.9G 0% /sys/firmware

mount | grep cifs

exit

from the node where the pod test-pfs-lab-jyu is running on:

$ df -h
Filesystem tmpfs /dev/mapper/vgubuntu-root 145G tmpfs tmpfs efivarfs /dev/sda1 tmpfs shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm shm tmpfs $ mount | grep cifs
$ Size Used Avail Use% Mounted on
790M 4.0M 786M 1% /run
33G 106G 24% /
3.9G 0 3.9G 0% /dev/shm
5.0M 0 5.0M 0% /run/lock
256K 47K 205K 19% /sys/firmware/efi/efivars
511M 6.1M 505M 2% /boot/efi
790M 76K 790M 1% /run/user/128
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/8410b0440b905b5f28b44505e681d2646941fce669eaa34d843f1d7433d497b4/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/344d447de1f08df3e0726ecc9ef6c4c79f30b1045a0ff6da447eadf4acaa932e/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/70e16d6404c9ab7ee9a17e8d5376dcf2b1a111c682f77b352675f65e6ff9d5dc/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/c0d793813a5df5696476053840da4808a0289a0bac6c3ab85af66f375058e582/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/6b8356edaddd39b06bef1b6aee965cc6c97c032225f7ff4b45cdd843022593c9/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/65e4a4d8157a96d03a827ef834cebfef3c7bedd84552d53c67751ac8bdb75c85/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/ce72d88887d78212cd81333cb07f692fdfc18c44aec81f45302948a0abe94fd0/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/eaf4267ba03153bf6635c98d7c60d46c693ca2c160488e1286d4f48cb4720f3b/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/91f3ec56bbaca13253dccc79545e9daef9ec02c585e481329a5b5f3be14362a7/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/17fb06c9446e217e4dd919702a65b13dcff96d74989d076a58f8e70f063d9db2/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/51eeef23fc8858cc1a2993fa4f9ee52e99770d7ccdbc2cf9764ef56d9f70a399/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/3ed893fc62dde1c3ebe212f8b7437e149a2394d9ecba77b39b4f0fcf2960913e/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/3e5ac4cf25cfef87755be882bf0a2a83d7ac6090d86a95b77ea23dc803dfb758/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/f81b1865a8b2ec9a26a6b9620f7d410b37438d934d5c32caefb2d3cabfc2b481/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/841fd5456bf7b8f9e1ba6a9ce1b61e3220fd9108adfab24472cf73fe24328893/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/a0697e88baf6eb7e814f99cf324cf0161042667115a176b5905da51e952422d1/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/b30ae1f40240c524c8a219c9c29bfcb12f7ea2738e24ebf4e0352e29d8036562/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/79a31defa3897374e1e9a3c7c0138c0adb276a420e31088e3ad53c6c43714249/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/bb6e7e948f87064eb26a81d610bea2f1175bcd715f5ea3983fdfff22e34f6b4d/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/4b8ab11e2c6c008d5ef044c9382beb93341fa8375feedb834edf23bf51e86d6f/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/fa8a371ec096fc594a6bab0cca4aebb9021e915105e543160aeaf155c2b03a14/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/c50c4cc13059b16d095135c4446b45d5b1096eaf4d9dce8d3fe174428e9e1e58/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/3a9dec8f33093a0ac69330ae4cfb02af09769ddb15df8742abfd28e702511b0e/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/fe9ca79622d5223534146610d727eb482a2cba6c23ff7e1ed51cc2db0f844ed4/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/28e436367af32707448d6284ffdd77d15b75703093f5b64fe91547c62786d1f5/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5ed156136b8efbcdaa251f7039d657448c6c082e9c72facb521b62ec2774467c/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/fae4f0756fbeb20a476de482ef6080fa92e676a65dc7c8b93799d488f8eba779/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/79d05019b337228b7373e819af849a8fdde4f0146a1ea0313cda8e0b4c14fb51/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/2c29389c0db5ea1e889b8987da331d7e405d093efcf992ca2973d1b7a8c93dab/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/c988358222df842da9153fbb0d72018f882e3dc861e5aca8c1768361ddeb1389/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/078bd9098866745cef7ca09f1f24d2227385971e8748d97267d54ea673d93b71/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/395df87bc79ca7efc66d506da366d4d68614bf91bbc32fb64be5c09d56bb7b69/shm
64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/86924c8bf6aa3a5a0bc4951d84bc3517152772dc24d334817d30b2ad9338a2ef/shm
790M 64K 790M 1% /run/user/1000

andyzhangx · 2024-08-11T06:55:58Z

there is no cifs mount on the node, does manual mount work on the node?

mkdir /tmp/test
sudo mount -v -t cifs //smb-server/fileshare /tmp/test -o vers=3.0,username=accountname,password=accountkey,dir_mode=0777,file_mode=0777,cache=strict,actimeo=30

pbs-jyu · 2024-08-11T15:25:46Z

$ dig smbserver-fqdn # return the IP properly
$ nslookup smbserver-fqdn # return the IP properly

running the following manually on the cluster node:

This works: sudo mount -t cifs -o username=myaccount,password='mypassword',domain=mydomain,vers=2.0 //smbserver-ip/share /mnt/share
This does not work: ssudo mount -t cifs -o username=myaccount,password='mypassword',domain=mydomain,vers=2.0 //smbserver-fqdn/share /mnt/share

pbs-jyu · 2024-08-11T22:16:53Z

I have installed cift-utils on each of the cluster nodes, then sudo mount -t cifs -o username=myaccount,password='mypassword',domain=mydomain,vers=2.0 //smbserver-fqdn/share /mnt/share, then I can see /mnt/share contents on the host nodes directly. But I still can't see the contents within the pod after created the secret, sc, pvc/pv, pod.

pbs-jyu · 2024-08-12T03:55:55Z

in the end, I got this working. But still something does not make sense to me.

Initally, when I can't see the any contents within the pod under /mnt/share, I did the following to find out where the problem lands.

I tried to manually mount the share on the k8s cluster node:
first, it only works with: sudo mount -t cifs -o username=myaccount,password='mypassword',domain=mydomain,vers=2.0 //smbserver-ip/share /mnt/share ; then "ls /mnt/share" will return contents.
I must install cifs-utils on the nodes, before I got "sudo mount -t cifs -o username=myaccount,password='mypassword',domain=mydomain,vers=2.0 //smbserver-fqdn/share /mnt/share" work.

At least at this stage, I can tell the problem is not related to networking/firewall, nor DNS.

then I tried:
created secret; storage classs, pvc which will automatically created a pv, created a pod with the following. I still can't see anything "ls /mnt/share"

apiVersion: v1
kind: Pod
metadata:
name: test-share-jyu-w-utils
namespace: dev
spec:
containers:

name: nginx
image: ubuntu:latest
command: ['sh', '-c', 'apt-get update && apt-get install -y cifs-utils nginx && nginx -g "daemon off;"'] (# add this later, it still does not work)
volumeMounts:
- name: smb-share
  mountPath: /mnt/share
  securityContext:
  privileged: true
  capabilities:
  add: ["SYS_ADMIN"]

volumes:

name: smb-share
persistentVolumeClaim:
claimName: pvc-sc-share

in the end, I must manually run "umount /mnt/share" then "sudo mount -t cifs -o username=myaccount,password='mypassword',domain=mydomain,vers=2.0 //smbserver-fqdn/share /mnt/share", Finally I can see the content when run "ls /mnt/share"

Even it is working now. But I do not understand why I must run "umount /mnt/share" and "sudo mount -t cifs -o username=myaccount,password='mypassword',domain=mydomain,vers=2.0 //smbserver-fqdn/share /mnt/share"" within the pod? Be aware that I have already created the storagclass used proper credetial, with the proper mountOptions, I have created the pvc, and the pod mounted with the proper pvc. After the pod was created, I should be able to see the content under /mnt/share without manually mount it.

pbs-jyu · 2024-08-13T19:58:41Z

Anyone can help with this bug? The manual umount and manually mount just help us to lock where this problem fall into. But this won't be a fix.

pbs-jyu · 2024-08-13T20:13:57Z

@andyzhangx I saw this old issue: #705 But that was very old one. I have install SMB v0.15.0.

pbs-jyu · 2024-08-13T20:48:19Z

fyi. I have used:
curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/v1.15.0/deploy/install-driver.sh | bash -s v1.15.0 -- --set controller.runOnControlPlane=true --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" note: we use microk8s cluster.

pbs-jyu · 2024-08-13T23:20:46Z

I FINALLY got this working. For Microk8s cluster, we need to add the last one.

Could anyone who has write access to the /csi-driver-smb/charts/README.md/Tips section?
--set controller.runOnControlPlane=true
--set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet"
--set "controller.nodeSelector.node.kubernetes.io/microk8s-controlplane"=microk8s-controlplane

pbs-jyu · 2024-08-14T01:56:14Z

Even it looks much better now. But still something not right.

when I used the following:
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.15.0 --set controller.runOnControlPlane=true --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" --set "controller.nodeSelector.node.kubernetes.io/microk8s-controlplane"=microk8s-controlplane
Error: INSTALLATION FAILED: 1 error occurred:
* Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal object into Go struct field PodSpec.spec.template.spec.nodeSelector of type string

Even with this error message, the smb mount inside of the pod behavior properly. I can see the content of the mount as expected. No error messages for each csi-smab-node : "kubectl logs csi-smb-node-xxx -n kube-system"

one thing important here is that I did not see pod csi-smb-controller-xxx running !!

	kubectl get all -n kube-system | grep csi
	pod/csi-smb-node-6kz58                                3/3     Running   0              43s
	pod/csi-smb-node-fdqzv                                3/3     Running   0              43s
	pod/csi-smb-node-xpd49                                3/3     Running   0              43s
	daemonset.apps/csi-smb-node                       3         3         3       3            3           kubernetes.io/os=linux   43s

when I used the following:
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.15.0 --set controller.runOnControlPlane=true --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" --set "controller.nodeSelector.node.kubernetes.io/microk8s-controlplane"=microk8s-controlplane # this installation did not through error, but the controller is pending status.

	kubectl get all -n kube-system | grep csi
	csi-smb-controller-667ff888fb-bwsg6               0/3     Pending   0               6m23s
	csi-smb-node-c8xfq                                3/3     Running   0               6m23s
	csi-smb-node-pd7kh                                3/3     Running   0               6m23s
	csi-smb-node-qbcbq                                3/3     Running   0               6m23s

" Warning FailedScheduling 92s default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling."

kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
microk8s-node1 Ready 51d v1.29.7 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/fluentd-ds-ready=true,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=s-mick8s1-dt1-x,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane
microk8s-node2 Ready 51d v1.29.7 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/fluentd-ds-ready=true,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=s-mick8s2-dt1-x,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane
microk8s-node3 Ready 51d v1.29.7 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/fluentd-ds-ready=true,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=s-mick8s3-dt1-x,kubernetes.io/os=linux,microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane

andyzhangx · 2024-08-14T15:04:14Z

@pbs-jyu can you remove --set controller.runOnControlPlane=true in helm install and try again?

pbs-jyu · 2024-08-20T15:27:23Z

hi, Andy

Thanks for your response. I have done the following tests and the results are as follows:

#: This does not work
kubectl label node your-cluster-nodes node-role.kubernetes.io/control-plane-
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.15.0

This works:

kubectl label node your-cluster-nodes node-role.kubernetes.io/control-plane-
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.15.0 --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet"

This works:

kubectl label node your-cluster-nodes node-role.kubernetes.io/control-plane=""
helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.15.0 --set controller.runOnControlPlane=true --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet"

pbs-jyu · 2024-08-22T15:54:31Z

Thank you for your help, Andy. I will close the case.

pbs-jyu closed this as completed Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can't see any content in mnt #818

can't see any content in mnt #818

pbs-jyu commented Aug 9, 2024

andyzhangx commented Aug 10, 2024

pbs-jyu commented Aug 10, 2024

andyzhangx commented Aug 11, 2024

pbs-jyu commented Aug 11, 2024

pbs-jyu commented Aug 11, 2024

pbs-jyu commented Aug 12, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 14, 2024 •

edited

Loading

andyzhangx commented Aug 14, 2024

pbs-jyu commented Aug 20, 2024

pbs-jyu commented Aug 22, 2024

can't see any content in mnt #818

can't see any content in mnt #818

Comments

pbs-jyu commented Aug 9, 2024

andyzhangx commented Aug 10, 2024

pbs-jyu commented Aug 10, 2024

cd /mnt/pfs-lab

ls

df -h

mount | grep cifs

exit

from the node where the pod test-pfs-lab-jyu is running on:

andyzhangx commented Aug 11, 2024

pbs-jyu commented Aug 11, 2024

running the following manually on the cluster node:

pbs-jyu commented Aug 11, 2024

pbs-jyu commented Aug 12, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 13, 2024

pbs-jyu commented Aug 14, 2024 • edited Loading

Even with this error message, the smb mount inside of the pod behavior properly. I can see the content of the mount as expected. No error messages for each csi-smab-node : "kubectl logs csi-smb-node-xxx -n kube-system"

one thing important here is that I did not see pod csi-smb-controller-xxx running !!

andyzhangx commented Aug 14, 2024

pbs-jyu commented Aug 20, 2024

This works:

This works:

pbs-jyu commented Aug 22, 2024

pbs-jyu commented Aug 14, 2024 •

edited

Loading