Hostpath across multiple nodes

[htbase]

Hi folks

I am trying to use the hostpath csi across multiple nodes but am running into an issue

This is the error I see when I create my app:

`
Events:
Type Reason Age From Message

---

Normal Scheduled 14m default-scheduler Successfully assigned default/my-csi-app to node4
Normal SuccessfulAttachVolume 14m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-1227aa63-6dee-11e9-af3e-02e1d85c598e"
Warning FailedMount 73s (x6 over 12m) kubelet, node4 Unable to mount volumes for pod "my-csi-app_default(3a1766c5-6dee-11e9-af3e-02e1d85c598e)": timeout expired waiting for volumes to attach or mount for pod "default"/"my-csi-app". list of unmounted volumes=[my-csi-volume]. list of unattached volumes=[my-csi-volume default-token-pglw4]
Warning FailedMount 8s (x15 over 14m) kubelet, node4 MountVolume.SetUp failed for volume "pvc-1227aa63-6dee-11e9-af3e-02e1d85c598e" : rpc error: code = NotFound desc = volume id 12311aad-6dee-11e9-aece-42010a960fc8 does not exit in the volumes list
`

I've changed the plugin deployment yaml to use DaemonSet instead, here is what it looks like:

kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-hostpathplugin
spec:
selector:
matchLabels:
app: csi-hostpathplugin
template:
metadata:
labels:
app: csi-hostpathplugin
spec:
hostNetwork: true
containers:
- name: node-driver-registrar
image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/csi-hostpath /registration/csi-hostpath-reg.sock"]
args:
- --v=5
- --csi-address=/csi/csi.sock
- --kubelet-registration-path=/var/lib/kubelet/plugins/csi-hostpath/csi.sock
securityContext:
privileged: true
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
volumeMounts:
- mountPath: /csi
name: socket-dir
- mountPath: /registration
name: registration-dir
- mountPath: /csi-data-dir
name: csi-data-dir
- name: hostpath
image: quay.io/k8scsi/hostpathplugin:v1.1.0
args:
- "--v=5"
- "--endpoint=$(CSI_ENDPOINT)"
- "--nodeid=$(KUBE_NODE_NAME)"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
securityContext:
privileged: true
ports:
- containerPort: 9898
name: healthz
protocol: TCP
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 2
volumeMounts:
- mountPath: /csi
name: socket-dir
- mountPath: /var/lib/kubelet/pods
mountPropagation: Bidirectional
name: mountpoint-dir
- mountPath: /var/lib/kubelet/plugins
mountPropagation: Bidirectional
name: plugins-dir
- mountPath: /csi-data-dir
name: csi-data-dir
- name: liveness-probe
volumeMounts:
- mountPath: /csi
name: socket-dir
image: quay.io/k8scsi/livenessprobe:v1.1.0
args:
- --csi-address=/csi/csi.sock
- --connection-timeout=3s
- --health-port=9898
volumes:
- hostPath:
path: /var/lib/kubelet/plugins/csi-hostpath
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /var/lib/kubelet/pods
type: DirectoryOrCreate
name: mountpoint-dir
- hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
name: registration-dir
- hostPath:
path: /var/lib/kubelet/plugins
type: Directory
name: plugins-dir
- hostPath:
path: /storage/
type: DirectoryOrCreate
name: csi-data-dir

I have 2 master and 2 worker nodes. When creating on one of the nodes, it works fine, but on the second one, it gives me the error above

NAME READY STATUS RESTARTS AGE
csi-hostpath-attacher-0 1/1 Running 0 19m
csi-hostpath-provisioner-0 1/1 Running 0 19m
csi-hostpath-snapshotter-0 1/1 Running 0 19m
csi-hostpath-socat-0 1/1 Running 0 19m
csi-hostpathplugin-75pwx 3/3 Running 0 19m
csi-hostpathplugin-xx2hj 3/3 Running 0 19m
my-csi-app 0/1 ContainerCreating 0 17m

Also, all nodes have the same file system and directory structure

Any help would be much appreciated

Thanks!

[htbase]

Btw, just to add info, the /storage/ folder directory is a shared directory that all nodes, both master and workers, can see this folder, so volumes created on top of that directory, is visible and accessible to all nodes

[pohly]

I am trying to use the hostpath csi across multiple nodes but am running into an issue

Why do you want to do that?

It is just an example driver that is used for testing. It is not designed to support running on multiple nodes. To make it reliable, topology support would have to be added.

[htbase]

We want to test it across nodes to test the storage layer underneath it, which is the presented globally across all nodes

I am not sure if we need a topology, because the same filesystem is presented across all nodes in the K8s cluster

Is there a way we can use it that way @pohly ?

[pohly]

Can't you use the builtin hostpath driver?

There are persistent local volumes (https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/) which you could configure to access your storage. Alternatively, an ephemeral hostpath volume (https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) could also expose that path inside a pod for testing.

[htbase]

We could, but dont want to have the users having to include both the path and node names, but rather, have the CSI do it @pohly

[pohly]

I guess I still don't understand the goal. If it's just for testing (#49 (comment)), why are users involved (#49 (comment))?

[htbase]

Because we want to test all nodes creating and attaching the volumes on top of that global filesystem before we dedicate the time to create a full csi

Is it possible with this hostpath driver @pohly ? Again to test and enable users to see how it would work with that filesystem

[pohly]

No, I don't think this can be done with the hostpath driver. The driver assumes that the volume gets provisioned on the node where the volume is then also going to be used. It sounds like you want to provision once and then use on different nodes.

[htbase]

I was thinking of DaemonSets, can we use it in this case?

The volume and container can be provisioned in the same nodes still, what I want is for the driver to use all available nodes to provision the volume and container instead of only one node

I created a daemonset and was able to replicate the hostpath driver on all nodes, but for some reason, it fails when creating the volume and container on different nodes (again, they can be on the same node, I just want it to use all nodes to do that instead of only 1 node) @pohly

[pohly]

Bruno Andrade <[email protected]> writes:

I was thinking of DaemonSets, can we use it in this case? The volume and container can be provisioned in the same nodes still, what I want is for the driver to use all available nodes to provision the volume and container instead of only one

There can be only one provisioner. It will then talk to one hostpath driver and provision there, and only there.