Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] RWX PVC mount failure in guest RKE2 cluster mount.nfs: Network is unreachable #7158

Open
hoo29 opened this issue Dec 13, 2024 · 7 comments
Open

[BUG] RWX PVC mount failure in guest RKE2 cluster mount.nfs: Network is unreachable #7158

hoo29 opened this issue Dec 13, 2024 · 7 comments
Assignees
@Vicente-Cheng
Labels
area/guest-cluster-rwx RWX for guest cluster kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/needed Reminder to add a reproduce label and to remove this one require/investigate Identified the issue but require further investigation for resolution (won't be stale) severity/needed Reminder to add a severity label and to remove this one

Comments

@hoo29
Copy link

hoo29 commented Dec 13, 2024

Describe the bug
Pods using RWX PVCs in a guest RKE2 cluster cannot start and have FailedMount error stderr mount.nfs: Network is unreachable

To Reproduce
Steps to reproduce the behavior:

  1. Follow https://docs.harvesterhci.io/v1.4/rancher/csi-driver#rwx-volumes-support
  2. Create pod in guest RKE2 cluster which uses a RWX PVC.

Expected behavior
The pods starts.

Support bundle
Sent via email

Environment

  • Harvester ISO version: 1.4.0
  • Underlying Infrastructure: Baremetal

Guest RKE2 version 1.31.2, CSI driver 0.1.20.

Additional context
RWO volumes in the guest cluster work without issue.

The harvester VMs with the guest RKE2 cluster use a VLAN tagged compute network and storage network.

I get the same result with Storage Network for RWX Volume Enabled: true and false in longhorn (albeit with different IPs)

The logs for the harvester cluster should be in the support bundle but here are some snippets from the guest cluster.

From the pod in the guest rke2 cluster

MountVolume.MountDevice failed for volume "pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93" : rpc error: code = Internal desc = Could not mount 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 for global path: failed to execute: nsenter [--mount=/proc/4533/ns/mnt --net=/proc/4533/ns/net --ipc=/proc/4533/ns/ipc mount -t nfs -o vers=4 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount], output , stderr mount.nfs: Network is unreachable

From the harvester csi driver pod on the node the pod is scheduled on in the guest cluster

time="2024-12-13T10:21:54Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:21:54Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:21:54Z" level=error msg="GRPC error: rpc error: code = Internal desc = NetworkFS pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 is not ready"
time="2024-12-13T10:21:54Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:21:54Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:21:54Z" level=error msg="GRPC error: rpc error: code = Internal desc = NetworkFS pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 is not ready"
time="2024-12-13T10:21:56Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:21:56Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:21:56Z" level=error msg="GRPC error: rpc error: code = Internal desc = NetworkFS pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 is not ready"
time="2024-12-13T10:21:58Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:21:58Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:21:58Z" level=error msg="GRPC error: rpc error: code = Internal desc = NetworkFS pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 is not ready"
time="2024-12-13T10:22:02Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:22:02Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:22:02Z" level=info msg="Mounting volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 to /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount"
time="2024-12-13T10:22:02Z" level=error msg="GRPC error: rpc error: code = Internal desc = Could not mount 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 for global path: failed to execute: nsenter [--mount=/proc/4533/ns/mnt --net=/proc/4533/ns/net --ipc=/proc/4533/ns/ipc mount -t nfs -o vers=4 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount], output , stderr mount.nfs: Network is unreachable\n: exit status 32"
time="2024-12-13T10:22:10Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:22:10Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:22:10Z" level=info msg="Mounting volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 to /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount"
time="2024-12-13T10:22:10Z" level=error msg="GRPC error: rpc error: code = Internal desc = Could not mount 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 for global path: failed to execute: nsenter [--mount=/proc/4533/ns/mnt --net=/proc/4533/ns/net --ipc=/proc/4533/ns/ipc mount -t nfs -o vers=4 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount], output , stderr mount.nfs: Network is unreachable\n: exit status 32"
time="2024-12-13T10:22:26Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:22:26Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:22:26Z" level=info msg="Mounting volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 to /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount"
time="2024-12-13T10:22:26Z" level=error msg="GRPC error: rpc error: code = Internal desc = Could not mount 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 for global path: failed to execute: nsenter [--mount=/proc/4533/ns/mnt --net=/proc/4533/ns/net --ipc=/proc/4533/ns/ipc mount -t nfs -o vers=4.2,noresvport,softerr,timeo=600,retrans=5 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount], output , stderr mount.nfs: Network is unreachable\n: exit status 32"
time="2024-12-13T10:22:58Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:22:58Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:22:58Z" level=info msg="Mounting volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 to /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount"
time="2024-12-13T10:22:58Z" level=error msg="GRPC error: rpc error: code = Internal desc = Could not mount 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 for global path: failed to execute: nsenter [--mount=/proc/4533/ns/mnt --net=/proc/4533/ns/net --ipc=/proc/4533/ns/ipc mount -t nfs -o vers=4.2,noresvport,softerr,timeo=600,retrans=5 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount], output , stderr mount.nfs: Network is unreachable\n: exit status 32"
time="2024-12-13T10:24:02Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:24:02Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:24:02Z" level=info msg="Mounting volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 to /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount"
time="2024-12-13T10:24:02Z" level=error msg="GRPC error: rpc error: code = Internal desc = Could not mount 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 for global path: failed to execute: nsenter [--mount=/proc/4533/ns/mnt --net=/proc/4533/ns/net --ipc=/proc/4533/ns/ipc mount -t nfs -o vers=4.2,noresvport,softerr,timeo=600,retrans=5 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount], output , stderr mount.nfs: Network is unreachable\n: exit status 32"
time="2024-12-13T10:26:04Z" level=info msg="GRPC call: /csi.v1.Node/NodeStageVolume request: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"hostStorageClass\":\"longhorn-rwx\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1734041585520-8081-driver.harvesterhci.io\"},\"volume_id\":\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\"}"
time="2024-12-13T10:26:04Z" level=info msg="NodeStageVolume is called with req volume_id:\"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93\" staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"hostStorageClass\" value:\"longhorn-rwx\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1734041585520-8081-driver.harvesterhci.io\" > "
time="2024-12-13T10:26:04Z" level=info msg="Mounting volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 to /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount"
time="2024-12-13T10:26:04Z" level=error msg="GRPC error: rpc error: code = Internal desc = Could not mount 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 for global path: failed to execute: nsenter [--mount=/proc/4533/ns/mnt --net=/proc/4533/ns/net --ipc=/proc/4533/ns/ipc mount -t nfs -o vers=4.2,noresvport,softerr,timeo=600,retrans=5 10.150.8.44:/pvc-89de0195-07ec-4ec7-8da1-62c2b6463da3 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.harvesterhci.io/26d62cd5077c7d294e48cd335475912b0a8e90b60e002cbedacdbb3b298845a8/globalmount], output , stderr mount.nfs: Network is unreachable\n: exit status 32"

From the csi driver controller pod in the guest cluster

I1213 10:21:51.196338 1 controller.go:837] successfully created PV pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 for PVC pvc-200mb-rwx and csi volume name pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93
I1213 10:21:51.196351 1 controller.go:853] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:driver.harvesterhci.io,VolumeHandle:pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{hostStorageClass: longhorn-rwx,storage.kubernetes.io/csiProvisionerIdentity: 1734041585520-8081-driver.harvesterhci.io,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,}}
I1213 10:21:51.196401 1 controller.go:1439] provision "NAMESPACE/pvc-200mb-rwx" class "harvester-longhorn-rwx": volume "pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93" provisioned
I1213 10:21:51.196440 1 controller.go:1456] provision "NAMESPACE/pvc-200mb-rwx" class "harvester-longhorn-rwx": succeeded
I1213 10:21:51.196448 1 volume_store.go:154] Saving volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93
I1213 10:21:51.205393 1 volume_store.go:157] Volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93 saved
I1213 10:21:51.205466 1 controller.go:1093] Claim processing succeeded, removing PVC 3d0004c4-4885-4b19-bd6d-ff261f4a2f93 from claims in progress
I1213 10:21:51.205497 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"NAMESPACE", Name:"pvc-200mb-rwx", UID:"3d0004c4-4885-4b19-bd6d-ff261f4a2f93", APIVersion:"v1", ResourceVersion:"22551201", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93
I1213 10:21:51.230394 1 controller.go:291] Started PVC processing "NAMESPACE/pvc-200mb-rwx"
I1213 10:21:51.230429 1 controller.go:334] No need to resize PVC "NAMESPACE/pvc-200mb-rwx"
I1213 10:21:51.940561 1 leaderelection.go:273] successfully renewed lease kube-system/external-resizer-driver-harvesterhci-io
I1213 10:21:53.581227 1 controller.go:208] Started VA processing "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.581267 1 csi_handler.go:218] CSIHandler: processing VA "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.581280 1 csi_handler.go:245] Attaching "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.581294 1 csi_handler.go:424] Starting attach operation for "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.581394 1 csi_handler.go:344] Adding finalizer to PV "pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93"
I1213 10:21:53.590574 1 csi_handler.go:353] PV finalizer added to "pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93"
I1213 10:21:53.590622 1 csi_handler.go:742] Found NodeID rke2-agent-3 in CSINode rke2-agent-3.DOMAIN
I1213 10:21:53.590662 1 csi_handler.go:306] VA finalizer added to "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.590696 1 csi_handler.go:320] NodeID annotation added to "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.597892 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1213 10:21:53.597914 1 connection.go:183] GRPC request: {"node_id":"rke2-agent-3","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":5}},"volume_context":{"hostStorageClass":"longhorn-rwx","storage.kubernetes.io/csiProvisionerIdentity":"1734041585520-8081-driver.harvesterhci.io"},"volume_id":"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93"}
I1213 10:21:53.610596 1 connection.go:185] GRPC response: {}
I1213 10:21:53.610626 1 connection.go:186] GRPC error: <nil>
I1213 10:21:53.610634 1 csi_handler.go:258] Attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.610642 1 util.go:37] Marking as attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616434 1 util.go:51] Marked as attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616453 1 csi_handler.go:264] Fully attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616460 1 csi_handler.go:234] CSIHandler: finished processing "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616492 1 controller.go:208] Started VA processing "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616500 1 csi_handler.go:218] CSIHandler: processing VA "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616504 1 csi_handler.go:245] Attaching "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616509 1 csi_handler.go:424] Starting attach operation for "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616525 1 csi_handler.go:338] PV finalizer is already set on "pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93"
I1213 10:21:53.616538 1 csi_handler.go:742] Found NodeID rke2-agent-3 in CSINode rke2-agent-3.DOMAIN
I1213 10:21:53.616546 1 csi_handler.go:298] VA finalizer is already set on "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616551 1 csi_handler.go:312] NodeID annotation is already set on "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.616558 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1213 10:21:53.616562 1 connection.go:183] GRPC request: {"node_id":"rke2-agent-3","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":5}},"volume_context":{"hostStorageClass":"longhorn-rwx","storage.kubernetes.io/csiProvisionerIdentity":"1734041585520-8081-driver.harvesterhci.io"},"volume_id":"pvc-3d0004c4-4885-4b19-bd6d-ff261f4a2f93"}
I1213 10:21:53.622340 1 connection.go:185] GRPC response: {}
I1213 10:21:53.622374 1 connection.go:186] GRPC error: <nil>
I1213 10:21:53.622384 1 csi_handler.go:258] Attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.622394 1 util.go:37] Marking as attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.627552 1 util.go:51] Marked as attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.627576 1 csi_handler.go:264] Fully attached "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.627588 1 csi_handler.go:234] CSIHandler: finished processing "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.627629 1 controller.go:208] Started VA processing "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.627647 1 csi_handler.go:218] CSIHandler: processing VA "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
I1213 10:21:53.627656 1 csi_handler.go:240] "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4" is already attached
I1213 10:21:53.627665 1 csi_handler.go:234] CSIHandler: finished processing "csi-0d23f8c892e5c9bf310daa817863d4dddd087f4068b8e45cce3bbdabc17588e4"
@hoo29 hoo29 added kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/needed Reminder to add a reproduce label and to remove this one severity/needed Reminder to add a severity label and to remove this one labels Dec 13, 2024
@bpedersen2
Copy link

Could this be related to #7041 and #3960 ?

@hoo29
Copy link
Author

hoo29 commented Dec 13, 2024

Forgot some additional context - we upgraded from 1.3.2 to 1.4.0.

All other networking appears to be working fine, all VMs are externally reachable, can communicate internally, storage network is verified accessible, and VMs can reach the harvester VIP fine.

@Vicente-Cheng
Copy link
Contributor

Hi @hoo29,

Did your guest VM have the same vlan network?
Also, the Storage Network for RWX Volume Enabled needs to be configured to True.

@hoo29
Copy link
Author

hoo29 commented Dec 14, 2024

Hi @Vicente-Cheng

The guest VMs are all on the same compute network. I believe there is only storage network which can be configured so everything is using it?

Confirmed Storage Network for RWX Volume Enabled is enabled in lonhorn and has been applied.

@innobead innobead added the area/guest-cluster-rwx RWX for guest cluster label Dec 14, 2024
@WebberHuang1118
Copy link
Member

Hi @hoo29
Can you provide the SB and the Harvester CSI logs?

@Vicente-Cheng
Copy link
Contributor

Hi @WebberHuang1118,

There are already some updates on the Slack channel. Sorry for missing an update here.
You can check here.

We are now checking the VM vlan and StorageNetwork.

@dfaltum
Copy link

dfaltum commented Dec 16, 2024

Also related : #7138

@bk201 bk201 added the require/investigate Identified the issue but require further investigation for resolution (won't be stale) label Dec 17, 2024
@Vicente-Cheng Vicente-Cheng mentioned this issue Jan 2, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Vicente-Cheng Vicente-Cheng

Labels
area/guest-cluster-rwx RWX for guest cluster kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/needed Reminder to add a reproduce label and to remove this one require/investigate Identified the issue but require further investigation for resolution (won't be stale) severity/needed Reminder to add a severity label and to remove this one
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@innobead @Vicente-Cheng @bk201 @bpedersen2 @hoo29 @WebberHuang1118 @dfaltum