MountVolume.SetUp failed for volume "registration-dir"

[jp83]

I don't have this /var/lib/kubelet/plugins_registry directory on my k8s nodes. I mentioned before I'm running RancherOS and currently running v1.15.10. Do I just need to create this directory, or am I missing some assumption in my setup?

Warning | FailedMount | Unable to mount volumes for pod "democratic-csi-1584812152-node-svm87_default(7934f18a-df44-4fbc-a335-460cf95a5dc0)": timeout expired waiting for volumes to attach or mount for pod "default"/"democratic-csi-1584812152-node-svm87". list of unmounted volumes=[registration-dir]. list of unattached volumes=[socket-dir plugins-dir registration-dir mountpoint-dir iscsi-dir dev-dir modules-dir localtime udev-data iscsi-info sys-dir host-dir config democratic-csi-1584812152-node-sa-token-k95m4] | a few seconds ago
-- | -- | -- | --
Warning | FailedMount | MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /var/lib/kubelet/plugins_registry is not a directory | a minute ago

This is all I have:

[rancher@k8s-node-2 ~]$ ls /var/lib/kubelet/
volumeplugins

Looking a templates/node.yaml I missing several other paths including:

/var/lib/kubelet/plugins
/var/lib/kubelet/plugins_registry
/etc/iscsi
/var/lib/iscsi

[jp83]

I guess there's a similar issue with rook csi, this issue/comment seems relevant

rook/rook#4133 (comment)

I have:

[rancher@k8s-node-2 ~]$ ls -1 /opt/rke/var/lib/kubelet
cpu_manager_state
plugin-containers
plugins
plugins_registry
pod-resources
pods

But this doesn't resolve the iscsi related directories, maybe this is the other issue you were referring to previously.

[travisghansen]

OK, interesting stuff. Are you running with rke or using rancher to deploy k8s?

Does rancherOS not come with iscsi tools installed by default?

[travisghansen]

Also if you could send over the data from docker inspect kubelet on one of your nodes. That will give me the full picture of how the thing is launch in RancherOS.

[jp83]

Um, I used rancher to deploy, but I think it uses a lot of RKE, but I haven't converted/exported my cluster it to a yaml file or anything.

I had to install iscsi to use OpenEBS, basically followed these instructions
openebs/openebs#1146 (comment)

[jp83]

One caveat is that I was on older rancher version (still 2+) and k8s 1.13, and updated a few weeks ago and had a few problems, mostly dns. I intend to start fresh again once I'm comfortable with this new underlying storage.

[rancher@k8s-node-2 ~]$ docker inspect kubelet
[
    {
        "Id": "da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e",
        "Created": "2020-02-25T14:14:05.472255563Z",
        "Path": "/opt/rke-tools/entrypoint.sh",
        "Args": [
            "kubelet",
            "--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem",
            "--cloud-provider=",
            "--fail-swap-on=false",
            "--volume-plugin-dir=/var/lib/kubelet/volumeplugins",
            "--authentication-token-webhook=true",
            "--cluster-dns=10.43.0.10",
            "--kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml",
            "--network-plugin=cni",
            "--cni-conf-dir=/etc/cni/net.d",
            "--read-only-port=0",
            "--authorization-mode=Webhook",
            "--resolv-conf=/etc/resolv.conf",
            "--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
            "--hostname-override=k8s-node-2",
            "--pod-infra-container-image=rancher/pause:3.1",
            "--anonymous-auth=false",
            "--cgroups-per-qos=True",
            "--address=0.0.0.0",
            "--event-qps=0",
            "--cni-bin-dir=/opt/cni/bin",
            "--cluster-domain=cluster.local",
            "--root-dir=/opt/rke/var/lib/kubelet",
            "--make-iptables-util-chains=true",
            "--streaming-connection-idle-timeout=30m",
            "--v=2"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2252,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-02-29T19:58:08.689659683Z",
            "FinishedAt": "2020-02-29T18:58:36.107019579Z"
        },
        "Image": "sha256:cdf19c45b54666841b869dd4211237a2b25758f28f2f363f5a8d19b4db275dcb",
        "ResolvConfPath": "/etc/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e/hostname",
        "HostsPath": "/var/lib/docker/containers/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e/hosts",
        "LogPath": "/var/lib/docker/containers/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e/da4ad41662aa8981f1ea5a138e6e26371ac240c72edb6c6249c0f3f6321bfd6e-json.log",
        "Name": "/kubelet",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/kubernetes:/etc/kubernetes:z",
                "/etc/cni:/etc/cni:rw,z",
                "/opt/cni:/opt/cni:rw,z",
                "/opt/rke/var/lib/cni:/var/lib/cni:z",
                "/var/lib/calico:/var/lib/calico:z",
                "/etc/resolv.conf:/etc/resolv.conf",
                "/sys:/sys:rprivate",
                "/var/lib/docker:/var/lib/docker:rw,rslave,z",
                "/opt/rke/var/lib/kubelet:/opt/rke/var/lib/kubelet:shared,z",
                "/var/lib/rancher:/var/lib/rancher:shared,z",
                "/var/run:/var/run:rw,rprivate",
                "/run:/run:rprivate",
                "/opt/rke/etc/ceph:/etc/ceph",
                "/dev:/host/dev:rprivate",
                "/var/log/containers:/var/log/containers:z",
                "/var/log/pods:/var/log/pods:z",
                "/usr:/host/usr:ro",
                "/etc:/host/etc:ro",
                "/var/lib/kubelet/volumeplugins:/var/lib/kubelet/volumeplugins:shared,z"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "2",
                    "max-size": "25m"
                }
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [
                "service-sidekick"
            ],
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "host",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417-init/diff:/var/lib/docker/overlay2/db0aefad5d29f05608bf2b52b1586929923bf8983ef03e719a41658c7ba8d8e3/diff:/var/lib/docker/overlay2/aa3fac404d7843a2c259bbfb5adda5bc91f4b48ec7104d6fe6d719b7e3911b3a/diff:/var/lib/docker/overlay2/bfddd63b2beeb2ac41c85ec4084cc56fe348e7d25b3eff3ecc60d6f008ca1401/diff:/var/lib/docker/overlay2/3ab650be37172d94da0d958ee2916d498376f5b654a5b8f25a9a9228a2f4909a/diff:/var/lib/docker/overlay2/b17b21e31698d1f2b7a440fde6f6cdd8327b4199cf32c43fdb993609d1dcfe3c/diff:/var/lib/docker/overlay2/1dbae6017447c251c3a151ba9e1d44519abdcfd39396ffe77e82911063426b4d/diff:/var/lib/docker/overlay2/e22b392641b881b2e54c7326165b9b131f94fd1b52041fe91c46819fc541e608/diff:/var/lib/docker/overlay2/4f953174368b6dc884e3f3e35f3f74cb7bd5ac1cb6dff5a1e3b29022ffff4d9e/diff:/var/lib/docker/overlay2/b58aef9bd78434f08ffe3c854b0dd2a107ae95ad34027f3d715db919205e8dfd/diff:/var/lib/docker/overlay2/a4b573f773d173ca061ad84127905d849bff9d51651ff59509685ee995bf7276/diff",
                "MergedDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417/merged",
                "UpperDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417/diff",
                "WorkDir": "/var/lib/docker/overlay2/84e96aa2138937c8b41d1fbab798ba42e61872f55edf7e4fc403e4cd97ffe417/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/calico",
                "Destination": "/var/lib/calico",
                "Mode": "z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/rancher",
                "Destination": "/var/lib/rancher",
                "Mode": "shared,z",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/var/run",
                "Destination": "/var/run",
                "Mode": "rw,rprivate",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/kubernetes",
                "Destination": "/etc/kubernetes",
                "Mode": "z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/resolv.conf",
                "Destination": "/etc/resolv.conf",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/dev",
                "Destination": "/host/dev",
                "Mode": "rprivate",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/opt/cni",
                "Destination": "/opt/cni",
                "Mode": "rw,z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "ac6941434ae742add3d7fc5bf135020671baa78dbe6c701b3d5f87f6fe6ec806",
                "Source": "/var/lib/docker/volumes/ac6941434ae742add3d7fc5bf135020671baa78dbe6c701b3d5f87f6fe6ec806/_data",
                "Destination": "/opt/rke-tools",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/usr",
                "Destination": "/host/usr",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/opt/rke/var/lib/kubelet",
                "Destination": "/opt/rke/var/lib/kubelet",
                "Mode": "shared,z",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/sys",
                "Destination": "/sys",
                "Mode": "rprivate",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/log/containers",
                "Destination": "/var/log/containers",
                "Mode": "z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/cni",
                "Destination": "/etc/cni",
                "Mode": "rw,z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc",
                "Destination": "/host/etc",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/opt/rke/var/lib/cni",
                "Destination": "/var/lib/cni",
                "Mode": "z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/volumeplugins",
                "Destination": "/var/lib/kubelet/volumeplugins",
                "Mode": "shared,z",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/opt/rke/etc/ceph",
                "Destination": "/etc/ceph",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/run",
                "Destination": "/run",
                "Mode": "rprivate",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/docker",
                "Destination": "/var/lib/docker",
                "Mode": "rw,rslave,z",
                "RW": true,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/var/log/pods",
                "Destination": "/var/log/pods",
                "Mode": "z",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "k8s-node-2.int.jonathanphelps.com",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "rancher/hyperkube:v1.15.10-rancher1",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/opt/rke-tools/entrypoint.sh",
                "kubelet",
                "--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem",
                "--cloud-provider=",
                "--fail-swap-on=false",
                "--volume-plugin-dir=/var/lib/kubelet/volumeplugins",
                "--authentication-token-webhook=true",
                "--cluster-dns=10.43.0.10",
                "--kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml",
                "--network-plugin=cni",
                "--cni-conf-dir=/etc/cni/net.d",
                "--read-only-port=0",
                "--authorization-mode=Webhook",
                "--resolv-conf=/etc/resolv.conf",
                "--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
                "--hostname-override=k8s-node-2",
                "--pod-infra-container-image=rancher/pause:3.1",
                "--anonymous-auth=false",
                "--cgroups-per-qos=True",
                "--address=0.0.0.0",
                "--event-qps=0",
                "--cni-bin-dir=/opt/cni/bin",
                "--cluster-domain=cluster.local",
                "--root-dir=/opt/rke/var/lib/kubelet",
                "--make-iptables-util-chains=true",
                "--streaming-connection-idle-timeout=30m",
                "--v=2"
            ],
            "OnBuild": null,
            "Labels": {
                "io.rancher.rke.container.name": "kubelet",
                "org.label-schema.build-date": "2020-02-13T19:18:10Z",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.vcs-ref": "902bbaa24c1d91a5ee867670fb5977d925132362",
                "org.label-schema.vcs-url": "https://github.com/rancher/hyperkube.git"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "57782bcc8e0cbb4610ebdb644a1324f90fa45d3971b01022c4e9ef336db86cfa",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "9a06d398b53b02d07584ef533bc260e7d45e6ec0c6791ff4b46e377d57e3e7d2",
                    "EndpointID": "f11ba92f7d7c198781abfd9c0fef514248044c81c5a560b1005829d280dada11",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

[travisghansen]

Yeah, I see they aren't doing a blanket /opt/rke/var/lib/kubelet mount. I'm running pure rke against CentOS 7 nodes ATM and this is what the mounts look like for me.

        "HostConfig": {
            "Binds": [
                "/etc/kubernetes:/etc/kubernetes:z",
                "/etc/cni:/etc/cni:rw,z",
                "/opt/cni:/opt/cni:rw,z",
                "/var/lib/cni:/var/lib/cni:z",
                "/var/lib/calico:/var/lib/calico:z",
                "/etc/resolv.conf:/etc/resolv.conf",
                "/sys:/sys:rprivate",
                "/var/lib/docker:/var/lib/docker:rw,rslave,z",
                "/var/lib/kubelet:/var/lib/kubelet:shared,z",
                "/var/lib/rancher:/var/lib/rancher:shared,z",
                "/var/run:/var/run:rw,rprivate",
                "/run:/run:rprivate",
                "/etc/ceph:/etc/ceph",
                "/dev:/host/dev:rprivate",
                "/var/log/containers:/var/log/containers:z",
                "/var/log/pods:/var/log/pods:z",
                "/usr:/host/usr:ro",
                "/etc:/host/etc:ro",
                "/vault:/vault",
                "/usr/local/sbin/iscsiadm:/usr/local/sbin/iscsiadm",
                "/:/host-iscsi"
            ],

I suspect if you're on a newer version those issues may go away, otherwise you'll have to call out each dir individually.

You'll also likely need to work around this here: rancher/rke#1846 in whatever form that takes with RancherOS + Rancher (not sure if you can customize stuff with rancher directly like that).

I realize it's a lot of moving parts here so appreciate the patience getting it going. I'd love to have steps detailed out using rancher + rancherOS though from a documentation standpoint.

The general issues are the same as what you mentioned in the openebs link (they aren't unique to this project). It's just the nature of how csi + containerized kubelet + iscsi all have to interact. I know the openebs folks and we've actually discussed collaborating on a few things (they like some of the work involved with democratic-csi that can be re-used by other projects). In any case, I'm confident you can get it going, but will probably take a few tweaks to get all the stars aligned properly.