Broken zvol links

[piexil]

I'm having the same issue experienced by this person on the proxmox forums
https://forum.proxmox.com/threads/no-zvol-device-link-found-after-300-sec-found-issue.94514/

tl;dr VMs fail to start with error 'no zvols found after 300 secs'

[fabianishere]

Which kernel version are you using?

[piexil]

happens on both pve-kernel-5.13-edge and pve-kernel-5.14-edge. pve-kernel-5.12-edge works fine

[fabianishere]

I am now updating the ZFS version to 2.1.1. Hopefully that resolves the issue.

[piexil]

TASK ERROR: timeout: no zvol device link for 'vm-103-disk-0' found after 300 sec found.
Still happens on v5.14.7-2

[piexil]

ok it seemed to work after a second boot of the VM this time...weird.

[piexil]

fyi 5.14.7-2 still reports as -1
root@epyc:~# uname -a Linux epyc 5.14.7-1-edge #1 SMP 5.14.7-1-edge generic (Thu, 24 Sep 2021 12:30:00 +0000) x86_64 GNU/Linux root@epyc:~#
root@epyc:~# pveversion -v proxmox-ve: 7.0-2 (running kernel: 5.14.7-1-edge) pve-manager: 7.0-9 (running version: 7.0-9/228c9caa) pve-kernel-helper: 7.0-4 pve-kernel-5.11: 7.0-3 pve-kernel-5.14.7-1-edge: 5.14.7-2

[fabianishere]

Glad to hear that you got it working. Let me know if the issue reappears.

fyi 5.14.7-2 still reports as -1
root@epyc:~# uname -a Linux epyc 5.14.7-1-edge #1 SMP 5.14.7-1-edge generic (Thu, 24 Sep 2021 12:30:00 +0000) x86_64 GNU/Linux root@epyc:~#
root@epyc:~# pveversion -v proxmox-ve: 7.0-2 (running kernel: 5.14.7-1-edge) pve-manager: 7.0-9 (running version: 7.0-9/228c9caa) pve-kernel-helper: 7.0-4 pve-kernel-5.11: 7.0-3 pve-kernel-5.14.7-1-edge: 5.14.7-2

That number represents the kernel ABI revision and not the Debian release. I currently do not track ABI changes, so I will remove this number from v5.15.x onwards.

[piexil]

Happens again on 5.15.0, reverting back to 5.14.16 works.

[piexil]

Actually seems reverting to 5.14.16 did not work, happening there after attempting 5.15.0 again

[fabianishere]

Does it work again if you try to reboot the VM?

[piexil]

nope. However sometimes rebooting the whole machine makes it work, it's working this second but I'm scared to reboot it., it took quite a few to get it working.

[piexil]

ok i don't know what causes this but the problem is not every link that's supposed to get created does. After booting for VM-103, I have

root@epyc:~# ls -la /dev/zvol/rpool/data/ | grep -i vm-103
lrwxrwxrwx 1 root root  15 Nov  6 00:39 vm-103-disk-0-part1 -> ../../../zd48p1
lrwxrwxrwx 1 root root  15 Nov  6 00:39 vm-103-disk-0-part2 -> ../../../zd48p2
lrwxrwxrwx 1 root root  15 Nov  6 00:39 vm-103-disk-0-part3 -> ../../../zd48p3

when it should be

root@epyc:~# ls -la /dev/zvol/rpool/data/ | grep -i vm-103
lrwxrwxrwx 1 root root  13 Nov  6 00:51 vm-103-disk-0 -> ../../../zd48
lrwxrwxrwx 1 root root  15 Nov  6 00:39 vm-103-disk-0-part1 -> ../../../zd48p1
lrwxrwxrwx 1 root root  15 Nov  6 00:39 vm-103-disk-0-part2 -> ../../../zd48p2
lrwxrwxrwx 1 root root  15 Nov  6 00:39 vm-103-disk-0-part3 -> ../../../zd48p3

If I manually create the link
ln -s ../../../zd48 /dev/zvol/rpool/data/vm-103-disk-0
the VM boots

[fabianishere]

Your issue looks similar to the one reported upstream: openzfs/zfs#12507.
There it is also mentioned that this issue possibly exists since Linux 5.13: openzfs/zfs#12301

This probably means that you'll have to wait until openzfs/zfs#12301 is fixed.

[amoiseiev]

The problem seems severe enough, wondering if it makes sense to add a temporary patch reverting:

torvalds/linux@a8ed1a0

and then dropping it when it's fixed in ZFS upstream. Obviously, not ideal but likely better than having random VM failures

[fabianishere]

@amoiseiev I agree, I'll create a patch reverting this change for the v5.15.x and v5.14.x branches.

[fabianishere]

@piexil Could you check whether the issue still appears in v5.14.17-1 or v5.15.1-1?

[piexil]

sorry for the late reply, have been away.
5.15.2 seems to be okay so far

[fabianishere]

This issue should be resolved in the latest builds.

[dac2020]

What's going on that this still happened to me using pve 7.1.7 and 7.2?
no zvol device link for...

I lost a bunch of guests out of the blue on a node in a cluster. Does it mean the others will vanish too?
Never had this happen on vmware esx, what does it mean?

[fabianishere]

@dac2020 What kernel version are you using?

[dac2020]

Hi,

I started this thread; https://forum.proxmox.com/threads/cannot-migrate-guests.114340/

It's now this;
proxmox-ve: 7.2-1 (running kernel: 5.15.39-4-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-9
pve-kernel-helper: 7.2-9
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4

But the problem started while the system was still 7.1.7 which I've since upgraded to 7.2 thinking it might help but didn't.
If I've done something wrong, so be it, hope to learn from it but as far as I recall, these guests were simply shut off and there were no problems what so ever until I wanted to migrate them before upgrading this node. The thread explains it all.

I can simply rebuild it but wanted to share in case it's something I didn't do and important to the proxmox devs.

[fabianishere]

Since you are using the stock kernel (and not this project), I am unable to help with this issue.

[dac2020]

No problem, wanted to share because it seems important before I rebuild.