No kABI-tracking kmod for RHEL-based Linux 9.5

[namazso]

System information

Type	Version/Name
Distribution Name	Oracle Linux
Distribution Version	9.5
Kernel Version	Linux 5.14.0-503.14.1.el9_5.x86_64
Architecture	amd64
OpenZFS Version	kmod-zfs-2.1.15-3.el9.x86_64

Describe the problem you're observing

Following the docs doesn't produce a usable ZFS install

Describe how to reproduce the problem

1. Install Oracle Linux 9.5
2. Switch to RHCK
3. Follow the instructions from the earlier linked docs

Include any warning/errors/backtraces from the system logs

# /sbin/modprobe zfs
modprobe: FATAL: Module zfs not found in directory /lib/modules/5.14.0-503.14.1.el9_5.x86_64

[namazso]

Possibly related to #16746 and #16196, however those are about DKMS and not kABI-tracking.

[clhedrick]

I'm running on RHEL 9.5 using rpm's generated according to those instructions. The only problem I had was getting it to install. If you are upgrading from 9.4 and you have the same version of ZFS already installed, installing the rpms built under 9.5 won't do anything, because they have the same name as the 9.4 ones and dnf thinks they're already installed. I had to do a reinstall, which unfortunately removed the kernel modules from any old 9.4 kernels but did give me a working 9.5 installation.

I should note that I'm using ZFS 2.2.6, while you're using 2.1.15. Version 2.1.16 is currently being prepared. The notes say that the 9.5 kernel broke the build. Take a look at #16781. You may need to apply the patches shown there or move to 2.2.6. I think 2.2 is stable now. If you're concerned, don't do a feature upgrade. But we've been running with 2.2.6 with zpool upgrade done in production on a variety of systems with no trouble. BRT is still not on by default.

[namazso]

Yes I’m running stable, and a package reinstall didn’t solve the issue. Depending on how the PR progresses I might switch to testing instead.

[paulvonbehren]

I'm seeing the same problems with ZFS ABI on AlmaLinux 9.5. All updates are done with dnf. On separate Alma 9.5 system, zfs is okay. On the 2nd system I'm holding off dnf updates until this is resolved.

[namazso]

@clhedrick I tried testing, with # dnf update --enablerepo=zfs-testing-kmod zfs but it did not improve the situation

[paulvonbehren]

As far as I can tell, there is no way to revert to AlmaLinux 9.4 from 9.5. Any attempt to undo using dnf history rollback or undo results in a list of missing older packages. I will reinstall 9.4 next...

[clhedrick]

You can't revert the whole of 9.5, but you should be able to boot a 9.4 kernel. That's probably ok. I'd expect that the update would leave old kernels. However you may have to build zfs for that kernel and install it in the old /lib/modules directory, as updating to 9.5 may have removed the 9.4 zfs. modules.`Or pull it from backups.

[albertdipp]

Thank goodness I came across this, thought I lost my marbles for a moment as to why the zfs module just isn't loading after upgrading to 9.5... Bad timing to build a new storage box it seems :D

[mouchyntamu]

Is there an update on this? We have exposed systems with critical kernel vulnerabilities because we have to keep the system booted to the RHEL 9.4 kernel...

[paulvonbehren]

@clhedrick is correct, simply reverting the kernel to a 9.4 version makes zfs work after upgrading to 9.5. You can switch the kernel two ways:

• use the boot-time grub menu to select the 9.4 kernel, or
• use use grubby CLI to select the 9.4 kernel, then reboot

I had used grub before and it was easy to use; I don't remember the details. But grubby was new to me so I took notes for other noobs.

Some background, the system where I needed grubby has no other purpose than as the target for rsync backups and has no monitor or keyboard. I connect using ssh. I wrote a script for routine procedures, including zfs scrub and dnf update - which is what caused this zfs problem (oops!). Here's how to change the default boot image on headless systems:

Make sure you know what kernel you are running

    # grubby --default-kernel
    /boot/vmlinuz-5.14.0-503.14.1.el9_5.x86_64

The files /boot/vmlinuz-* are the kernels installed by dnf. The dnf version number is "el9_5" before the trailing HW architecture (x86_64); separated by periods. Find the latest (or only) 9.4 image; for me it was

  /boot/vmlinuz-5.14.0-427.42.1.el9_4.x86_64

Use this to set the new default:

# grubby --set-default=/boot/vmlinuz-5.14.0-427.42.1.el9_4.x86_64
The default is /boot/loader/entries/4ab40aae34b34654a0bcbf8e8e2f63b9-5.14.0-427.42.1.el9_4.x86_64.conf with index 2 and kernel /boot/vmlinuz-5.14.0-427.42.1.el9_4.x86_64

Then reboot. Verify the running kernel is 9.4 with uname. zfs/zpool commands should work; if you uninstalled zfs (perhaps out of panic), you need to re-install.

[tonyhutter]

@mouchyntamu sorry 2.1.16 is taking so long. We had an issue where we couldn't run the zfs-2.1.16 test suite under the github runners, but I may have come up with a workaround: c2e5712. If that looks good then we can go forward with the release.

[clhedrick]

I had yet another issue of upgrading problems. There are probably things I don't know about complicating the situation, but to upgrade to a new ZFS build I ended up uninstalling all ZFS related packages (which weren't that easy to find. "dnf list --installed | grep @@commandline" actually did the best, though it also included a couple of things I didn't want to deinstall), and then installing the new ones.

Of course zfs had to be built on the kernel I wanted to use. That wasn't true in my case. I built on 9.5, but there was a slightly newer kernel in /boot. I ended up copying zfs.ko and spl.ko into /lib/modules/5.14.0-503.15.1.el9_5.x86_64/extra/zfs/, then running depmod 5.14.0-503.15.1.el9_5.x86_64.
I'm finding a bit too much magic needed here.

[tonyhutter]

On 12/6/24 we released zfs-2.1.16 with the EL 9.5 patches: https://github.com/openzfs/zfs/releases/tag/zfs-2.1.16. Please update to get the new kABI packages.