Latest Fedora 38 openzfs root instructions result in Failed to mount API filesystems

[chrisjsimpson]

The latest Fedora 38 openzfs root instructions appear to result in "Failed to mount API filesystems"

Looks to be related to a SELinux / relabelfrom issue (from my limited understanding)

Strings from the boot process:

• Unable to fix SELinux security context of /run: Permission denied (clearly a permissions issue: This is despite using fixfiles -F onboot)
• Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission Denied
• (see image)

Steps to reproduce:

• Follow latest Fedora 38 openzfs root instructions verbatim, non-encrypted

Have ran install from scratch twice.

I will be testing with Fedora 37 to see if this can be more isolated.

Apologies for the image over text, this is via a virtual canvas terminal.

Related 6be2e8c

[ghost]

Interesting. It seems that, the error is caused by SELinux denying systemd the permission to set SELinux context on /run. There is no need to perform a reinstallation. Set SELinux to permissive at boot time by setting the following kernel parameter at boot time, inside GRUB menu enforcing=0 Then let systemd do its job. After that, reboot. On this reboot SELinux will be enforced. See if the issue persists.

[chrisjsimpson]

I can confirm setting enforcing=0 during first time boot one-time does indeed allow the bootstrap process to complete.

Ideally we'd want to identify the cause so that the guide is successful (or add the above to the guide). I'm happy to investigate more, however I'm not sure how/what to instrument here- happy for suggestions.

[ghost]

I can confirm setting `enforcing=0` during first time boot one-time does indeed allow the bootstrap process to complete.

`enforcing=0` option only sets SELinux to permissive mode on that boot. For all subsequent boots, SELinux will be enforced. So, my question is, does the system still boot after completing the bootstrap process, without `enforcing=0`? If it boots, then we can say we have found a solution for the problem, which is to set SELinux to permissive for first time boot, and set to enforcing after the initial bootstrap completed.

[chrisjsimpson]

For all subsequent boots, SELinux will be enforced. So, my question is,
does the system still boot after completing the bootstrap process,
without enforcing=0?

Yes. I can confirm setting enforcing=0 during first time boot one-time does indeed allow the bootstrap process to complete, and boots to completion.

If it boots, then we can say we have found a solution for the problem,
which is to set SELinux to permissive for first time boot, and set to
enforcing after the initial bootstrap completed.

Whilst the one-time grub change does work, it would be useful to identify why fixfiles -F onboot appears to not have the expected effect since only fixfiles -F onboot was needed on the Fedora 37 Openzfs root docs. With that, no one-time grub change would be required. Happy to have found a way forward nonetheless.

[ghost]

chrisjsimpson ***@***.***> writes:

Yes. I can confirm setting enforcing=0 during first time boot one-time does indeed allow the bootstrap process to complete, and boots to completion.

Good to know.

Whilst the one-time grub change does work, it would be useful to identify why `fixfiles -F onboot` appears to not have the expected effect since only `fixfiles -F onboot` was needed on the Fedora 37 Openzfs root docs. With that, no one-time grub change would be required. Happy to have found a way forward nonetheless.

This problem is due to changes in Fedora 38. I do not have time to investigate further but if I were to harzard a guess, it would be due to a incomplete SELinux profile, which didn't cover paths like /run. In the meantime, I would use `enforcing=0` as a temporary solution. I need a confirmation for the steps. Does the text below describe the steps completely? - Initial installation in chroot, via Alpine Linux - Reboot. SELinux profile incompletely applied, system automatically reboots. - Reboots automatically, append `enforcing=0` to GRUB menu `linux` line. - Rest of SELinux profile applied. - Manually reboot. Now boot Fedora with unmodified kernel command line. - SELinux enforcing.

…

-- Reply to this email directly or view it on GitHub: #439 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[chrisjsimpson]

I need a confirmation for the steps. Does the text below describe the
steps completely?

• Initial installation in chroot, via Alpine Linux
• Reboot. SELinux profile incompletely applied, system automatically reboots.
• Reboots automatically, append enforcing=0 to GRUB menu linux line.
• Rest of SELinux profile applied.
• Manually reboot. Now boot Fedora with unmodified kernel command line.
• SELinux enforcing.

Yes I have followed those steps again on a new system and can confirm those steps describe the steps completely.