Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ZVOL creation not showing up in /dev/<...> (RCA: active zfs version older than user expected) #7889

Closed
fermulator opened this issue Sep 12, 2018 · 8 comments
Closed

ZVOL creation not showing up in /dev/<...> (RCA: active zfs version older than user expected) #7889

fermulator opened this issue Sep 12, 2018 · 8 comments

Comments

@fermulator
Copy link

System information

Type Version/Name
Distribution Name Fedora
Distribution Version 27
Linux Kernel 4.3.4-200.fc22.x86_64
Architecture x86_64
ZFS Version 0.7.9-1
SPL Version 0.7.9-1

Expectations

Normally, (and I've done this many times in the past), I create a ZVOL, and it shows up in /dev/

Example (from a long time ago):

zfs create zmain/STORAGE/vms/house
zfs create -V 60G zmain/STORAGE/vms/house/disk0

$ ll /dev/zvol/zmain/STORAGE/vms/house
total 0
lrwxrwxrwx. 1 root root 18 Jun 29 15:45 disk0 -> ../../../../../zd0
lrwxrwxrwx. 1 root root 20 Jun 29 15:45 disk0-part1 -> ../../../../../zd0p1

Describe the problem you're observing

Today creation of a ZVOL is failing to show up in /dev/zvol/....

zfs create zmain/STORAGE/vms/nextcloud
zfs create -V 60G zmain/STORAGE/vms/nextcloud/disk0

$ ll /dev/zvol/zmain/STORAGE/vms/ | grep nextcloud
<EMPTY>

ZFS is fully functional otherwise (many other datasets and ZVOLs active...)

$ sudo systemctl status zfs.target
● zfs.target - ZFS startup target
   Loaded: loaded (/usr/lib/systemd/system/zfs.target; enabled; vendor preset: enabled)
   Active: active since Fri 2018-06-29 15:45:38 EDT; 2 months 13 days ago

$ sudo zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zmain   556G   110G   446G         -      -    19%  1.62x  ONLINE  -

$ sudo zfs list 
NAME                                                     USED  AVAIL  REFER  MOUNTPOINT
zmain                                                    291G   304G   156K  /zmain
zmain/ROOT                                              21.9G   304G   136K  none
zmain/ROOT/fedora                                       21.9G   304G  5.23G  /
zmain/ROOT/fedora@AutoW-2018-36                         81.5M      -  5.23G  -
zmain/ROOT/fedora@AutoD-2018-09-10                      27.6M      -  5.23G  -
zmain/ROOT/fedora@AutoD-2018-09-11                      29.0M      -  5.23G  -
zmain/ROOT/fedora/swap0                                 16.5G   315G  4.97G  -
(snip)
zmain/STORAGE                                            264G   304G   164K  /zmain/STORAGE
(snip)
zmain/STORAGE/vms                                        249G   304G  21.9G  /zmain/STORAGE/vms
(snip)
zmain/STORAGE/vms/house                                 19.3G   304G   136K  /zmain/STORAGE/vms/house
zmain/STORAGE/vms/house/disk0                           19.3G   304G  17.2G  -
zmain/STORAGE/vms/house/disk0@AutoW-2018-36              804M      -  17.2G  -
zmain/STORAGE/vms/house/disk0@AutoD-2018-09-10           633M      -  17.2G  -
zmain/STORAGE/vms/house/disk0@AutoD-2018-09-11           542M      -  17.2G  -
(snip)
zmain/STORAGE/vms/nextcloud                              208K   304G   136K  /zmain/STORAGE/vms/nextcloud
zmain/STORAGE/vms/nextcloud/disk0                         72K   304G    72K  -

Here are the block devices and outputs

$ ll /dev/zvol/zmain/STORAGE/vms/* | grep zd | grep -v part
lrwxrwxrwx. 1 root root 19 Jun 29 15:45 disk0 -> ../../../../../zd64
lrwxrwxrwx. 1 root root 19 Jun 29 15:45 disk1 -> ../../../../../zd80
lrwxrwxrwx. 1 root root 19 Jun 29 15:45 disk0 -> ../../../../../zd48
lrwxrwxrwx. 1 root root 18 Jun 29 15:45 disk0 -> ../../../../../zd0
lrwxrwxrwx. 1 root root 19 Jun 29 15:45 disk0 -> ../../../../../zd32
lrwxrwxrwx. 1 root root 19 Jun 29 15:45 disk1 -> ../../../../../zd16
lrwxrwxrwx. 1 root root 19 Jun 29 15:45 disk0 -> ../../../../../zd96

$ ls -al /sys/block/ | grep zd | grep -v part
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd0 -> ../devices/virtual/block/zd0 <root>
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd112 -> ../devices/virtual/block/zd112  <NFS device (not shown)>
lrwxrwxrwx.  1 root root 0 Jun 29 15:47 zd128 -> ../devices/virtual/block/zd128  <swap ZVOL (not shown)>
lrwxrwxrwx.  1 root root 0 Sep 11 21:00 zd144 -> ../devices/virtual/block/zd144  <**_NOMATCH_**>
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd16 -> ../devices/virtual/block/zd16 <ZVOL>
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd32 -> ../devices/virtual/block/zd32 <ZVOL>
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd48 -> ../devices/virtual/block/zd48 <ZVOL>
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd64 -> ../devices/virtual/block/zd64 <ZVOL>
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd80 -> ../devices/virtual/block/zd80 <ZVOL>
lrwxrwxrwx.  1 root root 0 Jun 29 15:45 zd96 -> ../devices/virtual/block/zd96 <ZVOL>

zd144 does not seem to match to any known ZVOL (it COULD be the one missing, which means the block device is there but udev isn't matching it?)

I tried the workaround in #599 ...

$ sudo udevadm trigger -v --subsystem-match=block --sysname-match=zd* --action=change
/sys/devices/virtual/block/zd0
/sys/devices/virtual/block/zd0/zd0p1
/sys/devices/virtual/block/zd112
/sys/devices/virtual/block/zd112/zd112p1
/sys/devices/virtual/block/zd128
/sys/devices/virtual/block/zd144
/sys/devices/virtual/block/zd16
/sys/devices/virtual/block/zd16/zd16p1
/sys/devices/virtual/block/zd32
/sys/devices/virtual/block/zd32/zd32p1
/sys/devices/virtual/block/zd32/zd32p2
/sys/devices/virtual/block/zd48
/sys/devices/virtual/block/zd48/zd48p1
/sys/devices/virtual/block/zd48/zd48p2
/sys/devices/virtual/block/zd64
/sys/devices/virtual/block/zd64/zd64p1
/sys/devices/virtual/block/zd64/zd64p2
/sys/devices/virtual/block/zd80
/sys/devices/virtual/block/zd80/zd80p1
/sys/devices/virtual/block/zd96
/sys/devices/virtual/block/zd96/zd96p1

No affect.

I've also tried the "rename" trick and both sparse and non-sparse... to no avail

$ sudo zpool history zmain | grep nextcloud
2018-09-11.16:12:53 zfs create zmain/STORAGE/vms/nextcloud
2018-09-11.16:14:57 zfs create -V 60G zmain/STORAGE/vms/nextcloud/disk0
2018-09-11.16:20:13 zfs destroy zmain/STORAGE/vms/nextcloud/disk0
2018-09-11.16:20:18 zfs create -s -V 60G zmain/STORAGE/vms/nextcloud/disk0
2018-09-11.20:53:00 zfs destroy zmain/STORAGE/vms/nextcloud/disk0
2018-09-11.20:53:28 zfs create -V 60G zmain/STORAGE/vms/nextcloud/disk0
2018-09-11.20:53:41 zfs set refreservation=none zmain/STORAGE/vms/nextcloud/disk0
2018-09-11.21:08:40 zfs rename zmain/STORAGE/vms/nextcloud zmain/STORAGE/vms/nextcloud-TEMP
2018-09-11.21:08:48 zfs rename zmain/STORAGE/vms/nextcloud-TEMP zmain/STORAGE/vms/nextcloud
2018-09-11.21:09:21 zfs rename zmain/STORAGE/vms/nextcloud/disk0 zmain/STORAGE/vms/nextcloud/disk0-TEMP
2018-09-11.21:09:27 zfs rename zmain/STORAGE/vms/nextcloud/disk0-TEMP zmain/STORAGE/vms/nextcloud/disk0

Hints/Notes

Recently I started taking daily snapshots ... could this have hit a bug?

Describe how to reproduce the problem

Simply creation of the ZVOL, and its not showing up in /dev. (sorry I do not have steps to repro from a greenfield system - hoping for help debug)

Include any warning/errors/backtraces from the system logs

There are no (relevant) logs in dmesg.
@fermulator
Copy link
Author

fermulator commented Sep 12, 2018 •
edited
Loading

One crazy thing I'm just noticing ...

$ sudo modprobe zfs
modprobe: FATAL: Module zfs not found in directory /lib/modules/4.3.4-200.fc22.x86_64

$ find /lib/modules | grep zfs
/lib/modules/4.17.2-100.fc27.x86_64/extra/zfs.ko.xz
/lib/modules/4.16.12-200.fc27.x86_64/extra/zfs.ko.xz
/lib/modules/4.16.13-200.fc27.x86_64/extra/zfs.ko.xz

what

$ uname -a
Linux (HOSTNAME) 4.3.4-200.fc22.x86_64 #1 SMP Mon Jan 25 13:37:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/fedora-release 
Fedora release 27 (Twenty Seven)

$ rpm -qa | grep zfs
zfs-0.7.9-1.fc27.x86_64
zfs-dracut-0.7.9-1.fc27.x86_64
zfs-release-1-5.fc22.noarch
zfs-dkms-0.7.9-1.fc27.noarch
libvirt-daemon-driver-storage-zfs-3.7.0-4.fc27.x86_64
libzfs2-0.7.9-1.fc27.x86_64

I am a bit confused to that (huh, I'm running Fedora27 but w/ a Fedora22 kernel?) ... could be related (though all the zfs filesystems are fully functional at this time ... worried to reboot the system now LoL)

@bunder2015
Copy link
Contributor

yes, something does seem to be wrong here, did you forget to update your grub after upgrading kernels, or forget to install the new ones into /boot? i would double check and reboot to make sure you are using the (whichever is) correct version you intend to be running.

@fermulator
Copy link
Author

fermulator commented Sep 12, 2018 •
edited
Loading

Well, after the reboot, the /dev/zvol got updated properly now

$ ll /dev/zvol/zmain/STORAGE/vms/nextcloud/
total 0
lrwxrwxrwx. 1 root root 20 Sep 11 22:23 disk0 -> ../../../../../zd112

(confusingly this blockID changed from previous boot)

--

Still though my kernel boot is screwed up, maybe it is to blame since it looks like I'm booting older kernal and older ZFS/SPL modules

$ dmesg | egrep "SPL|ZFS"
[    0.000000] Command line: BOOT_IMAGE=/ROOT/fedora@/boot/vmlinuz-4.3.4-200.fc22.x86_64 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain
[    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/fedora@/boot/vmlinuz-4.3.4-200.fc22.x86_64 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain
[    2.489666] SPL: Loaded module v0.6.5.4-1
[    2.575121] ZFS: Loaded module v0.6.5.4-1, ZFS pool version 5000, ZFS filesystem version 5
[   20.242733] SPL: using hostid 0x007f0100

@fermulator
Copy link
Author

No idea why grub-config doesn't rebuild /boot/grub2/grub.cfg, but definitely it was still tied to the WAY OLD F22!

BEFORE

# grep ROOT ~/grub.cfg_F22_backup
	linuxefi /ROOT/fedora@/boot/vmlinuz-4.3.4-200.fc22.x86_64 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain 
	initrdefi /ROOT/fedora@/boot/initramfs-4.3.4-200.fc22.x86_64.img
	linuxefi /ROOT/fedora@/boot/vmlinuz-4.0.4-301.fc22.x86_64 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain 
	initrdefi /ROOT/fedora@/boot/initramfs-4.0.4-301.fc22.x86_64.img
	linuxefi /ROOT/fedora@/boot/vmlinuz-0-rescue-72b97c7344964f3d9c98cacf0caca2f8 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain 
	initrdefi /ROOT/fedora@/boot/initramfs-0-rescue-72b97c7344964f3d9c98cacf0caca2f8.img

Fixed that

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.17.19-100.fc27.x86_64
Found initrd image: /boot/initramfs-4.17.19-100.fc27.x86_64.img
Found linux image: /boot/vmlinuz-4.17.2-100.fc27.x86_64
Found initrd image: /boot/initramfs-4.17.2-100.fc27.x86_64.img
Found linux image: /boot/vmlinuz-4.3.4-200.fc22.x86_64
Found initrd image: /boot/initramfs-4.3.4-200.fc22.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-72b97c7344964f3d9c98cacf0caca2f8
Found initrd image: /boot/initramfs-0-rescue-72b97c7344964f3d9c98cacf0caca2f8.img
done

AFTER

# grep ROOT /boot/grub2/grub.cfg
	linuxefi /ROOT/fedora@/boot/vmlinuz-4.17.19-100.fc27.x86_64 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain 
	initrdefi /ROOT/fedora@/boot/initramfs-4.17.19-100.fc27.x86_64.img
	linuxefi /ROOT/fedora@/boot/vmlinuz-4.17.2-100.fc27.x86_64 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain 
	initrdefi /ROOT/fedora@/boot/initramfs-4.17.2-100.fc27.x86_64.img
	linuxefi /ROOT/fedora@/boot/vmlinuz-4.3.4-200.fc22.x86_64 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain 
	initrdefi /ROOT/fedora@/boot/initramfs-4.3.4-200.fc22.x86_64.img
	linuxefi /ROOT/fedora@/boot/vmlinuz-0-rescue-72b97c7344964f3d9c98cacf0caca2f8 root=ZFS=zmain/ROOT/fedora ro rhgb quiet efi_no_storage_paranoia boot=zfs rpool=zmain 
	initrdefi /ROOT/fedora@/boot/initramfs-0-rescue-72b97c7344964f3d9c98cacf0caca2f8.img

@fermulator
Copy link
Author

fermulator commented Sep 12, 2018 •
edited
Loading

(of note, attempts to boot the newer kernel fail because Fedora seems to have not auto-build via dkms zfs/spl modules there ;/) - working on fixing that - but that's a different story/issue

roughly: https://icesquare.com/wordpress/centosrhel-7-no-zfs-after-updating-the-kernel/ is to blame (I have the same issues on other RHEL-based systems)

@fermulator
Copy link
Author

PS: I think its a reasonable conclusion to have closed this issue and mark it as "user error" (or whatever resolution status fits for this project)

The disaster of Fedora27 still booting an older Fedora22 kernel which had super old zfs/spl versions likely meant that I was running a version older than when the fix for the bug I ran into was delivered (though I didn't verify this explicitly).

Further, fwiw, I'm attempting to root cause and fully understand what's going on with Fedora's dumb kernel upgrade problem (which might be some dumb user error from long ago trickling forward), and how to manage dkms zfs/spl auto-build failures with Fedora. See https://ask.fedoraproject.org/en/question/126473/how-to-manage-kernel-versions-and-dkms-zfsspl-modules/. (pending moderation)

@fermulator fermulator changed the title ZVOL creation not showing up in /dev/<...> ZVOL creation not showing up in /dev/<...> (RCA: active zfs version older than user expected) Sep 12, 2018
@mskarbek
Copy link
Contributor

@fermulator sadly the best answer to problems with DKMS is to use kmod packages, I'm maintaining my own repo with those just to avoid issues with kernel upgrades on Fedora. This is especially helpful when you are using ZFS as root storage.

@fermulator
Copy link
Author

@mskarbek - any docs/refs for zfs+kmod solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mskarbek @fermulator @bunder2015 and others