ZFS kmod for CentOS Stream

[shodanshok]

Describe the feature would like to see added to OpenZFS

CentOS is diverging from RHEL, quickly becoming a CentOS-Stream (ie: rolling release) only distro. Please read the following links for more details:
https://blog.centos.org/2020/12/future-is-centos-stream/
https://www.redhat.com/en/blog/centos-stream-building-innovative-future-enterprise-linux
https://centos.org/distro-faq/

At the moment it is unclear if the rolling-release distribution will be blindly applied to kernel updates. If so, a concrete risk of ABI break between kernel updated exists. This can cause big issues on the the ZFS kmod packages. This GitHub ticket was opened with the idea of identify what, if any, we can do to avoid kernel/ZFS update issues for us using CentOS. The only things I can think of are:

• introduce CentOS stream in the CI suite
• only use whitelisted kernel symbols

Note: kmod package issues are nothing new, see here . However, the move to CentOS Stream can bring them to a completely new (bad) levels.

[sempervictus]

Big miscalculation on IBMs part IMO, CentOS was the stepping stone to RHEL... Lets see if Suse gains traction :).

Rolling releases are easy enough though, we maintain Arch packaging in-tree and as DKMS and its actually much less hassle this way because changes are gradual instead of "version rollover" madness.

[zorinlynx]

Looks like I'll be versionlocking kernel updates if I stick with CentOS stream, and doing them manually when there's security issues.

IBM/RedHat has pissed off a lot of people with this decision. There's a lot of pushback and we may see a compromise, even if it's just supporting CentOS 8 through its original timeline.

[behlendorf]

Adding CentOS Stream to the CI makes sense and should be straight forward to add. That will at least ensure ZFS builds correctly and is compatible with the shipping CentOS kernel. What's not clear to me is if CentOS Stream will make any guarantees about kABI stability between kernel releases (i.e. there is no whitelist). Hopefully they do, but if not then DKMS packages may be the most realistic option.

[shodanshok]

@behlendorf How OpenZFS 2.0 fares with regard to whitelisted symbols? From what I remember, you had a concerted attempt to only use stable, whitelisted symbols. Any news on the matter?

[behlendorf]

Regarding the whiltelist a lot of progress has been made. We've been pursuing it on two fronts. On one side RedHat has added many of the symbols we need to their whitelist for RHEL/CentOS 8.0, and on the other side we've tried to limit the number of symbols we depend on. This reduced the number of non-whitelisted symbols dependencies considerably but not to zero. As I recall there were a couple more troublesome ones as well which were unlikely to be added to the whitelist and it would be difficult for us to do without. So definitely some big steps in the right direction, but we're not there yet.

[behlendorf]

I've opened PR #11374 which resolves the CentOS Stream build issue. Surprisingly enough the fix involved the blk_queue_alloc symbol which is on their whitelist. It may be that even strictly adhering to the whitelist isn't going to be sufficient for CentOS Stream.

[carlwgeorge]

Adding CentOS Stream to the CI makes sense and should be straight forward to add. That will at least ensure ZFS builds correctly and is compatible with the shipping CentOS kernel.

That's absolutely the correct answer here. This will also have the benefit of ensuring ZFS is ready for the next RHEL minor release kABI, which has been a pain point in the past (see #9856, #10466, #11195).

What's not clear to me is if CentOS Stream will make any guarantees about kABI stability between kernel releases (i.e. there is no whitelist). Hopefully they do, but if not then DKMS packages may be the most realistic option.

CentOS doesn't make any guarantees per say, but the kernel that gets shipped in CS8 has the kABI that is planned for the next RHEL minor release (currently 8.4). The development mentality is still the same, the world just gets to see it early now. You can expect the same amount of changes that have happened in previous minor releases.

Surprisingly enough the fix involved the blk_queue_alloc symbol which is on their whitelist. It may be that even strictly adhering to the whitelist isn't going to be sufficient for CentOS Stream.

Would you mind filing a bug for this? It may just be a difference between the 8.3 and 8.4 kABIs, but if it is indeed a mistake then the kernel maintainers want to hear about it.

[behlendorf]

@carlwgeorge thanks for pointing me at the appropriate issue tracker. I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1911343 to bring this kABI change to the kernel maintainers attention. I'll also see about adding CentOS stream to our CI.

[carlwgeorge]

kernel-4.18.0-294.el8, which adds back blk_alloc_queue and blk_queue_make_request, is now available in CentOS Stream 8, which unblocks this issue as far as I can tell.

[behlendorf]

Yes, I believe we can close this now. The latest CentOS Stream kernel provided the removed interfaces again, the ZFS code was updated to use either the old or new interfaces depending on what's available, a CentOS 8 Stream builder was added to the CI.

[gdevenyi]

#10333