Kernel too old for mounting /boot produced by Qubes 4.3 / Fedora 41

[marmarek]

Please identify some basic details to help process the report

A. Provide Hardware Details

1. What board are you using? (Choose from the list of boards here)

2. Does your computer have a dGPU or is it iGPU-only?

   • dGPU (Distinct GPU other then internal GPU)
   • iGPU-only (Internal GPU, normally Intel GPU)

3. Who installed Heads on this computer?

   • Insurgo (Issues to be reported at https://github.com/linuxboot/heads/issues)
   • Nitrokey (Issues to be reported at https://github.com/Nitrokey/heads/issues)
   • Purism (Issues to be reported at https://source.puri.sm/firmware/pureboot/-/issues)
   • Novacustom (Issues to be reported at https://github.com/Dasharo/dasharo-issues)
   • HardnenedVault (Issues to be reported at https://github.com/hardenedvault/vaultboot/issues)
   • Other provider
   • Self-installed

4. What PGP key is being used?

   • Librem Key (Nitrokey Pro 2 rebranded)
   • Nitrokey Pro
   • Nitrokey Pro 2
   • Nitrokey 3 NFC
   • Nitrokey 3 NFC Mini
   • Nitrokey Storage
   • Nitrokey Storage 2
   • Yubikey
   • Other

5. Are you using the PGP key to provide HOTP verification?

   • Yes
   • No
   • I don't know

B. Identify how the board was flashed

1. Is this problem related to updating heads or flashing it for the first time?

   • First-time flash
   • Updating heads

2. If the problem is related to an update, how did you attempt to apply the update?

   • Using the Heads menus
   • Flashrom via the Recovery Shell
   • External flashing

3. How was Heads initially flashed?

   • External flashing
   • Internal-only / 1vyprep+1vyrain / skulls
   • Don't know

4. Was the board flashed with a maximized or non-maximized/legacy rom?

   • Maximized
   • Non-maximized / legacy
   • I don't know

5. If Heads was externally flashed, was IFD unlocked?

   • Yes
   • No
   • Don't know

C. Identify the rom related to this bug report

1. Did you download or build the rom at issue in this bug report?

   • I downloaded it
   • I built it

2. If you downloaded your rom, where did you get it from?

   • Heads CircleCi
   • Purism
   • Nitrokey
   • Dasharo DTS (Novacustom)
   • Somewhere else (please identify)

   Please provide the release number or otherwise identify the rom downloaded

3. If you built your rom, which repository:branch did you use?

   • Heads:Master (3fef9e0)
   • Other (please identify)

4. What version of coreboot did you use in building?
   { You can find this information from github commit ID or once flashed, by giving the complete version from Sytem Information under Options --> menu}

5. In building the rom, where did you get the blobs?

   • No blobs required
   • Provided by the company that installed Heads on the device
   • Extracted from a backup rom taken from this device
   • Extracted from another backup rom taken from another device (please identify the board model)
   • Extracted from the online bios using the automated tools provided in Heads
   • I don't know

Please describe the problem

Describe the bug

Mounting /boot read-write fails with couldn't mount RDWR because of unsupported optiional features (10000)

To Reproduce
Steps to reproduce the behavior:

1. Install Qubes 4.3 (for example https://qubes.notset.fr/iso-testing/Qubes-4.3.202409102200-x86_64.iso, or one of openQA iso)
2. Try to reset TPM
3. Observe it failing

Expected behavior

Should mount /boot normally

Screenshots

Additional context

Looks like 5.10 kernel is still too old for this.

[marmarek]

This looks to be EXT4_FEATURE_RO_COMPAT_ORPHAN_PRESENT, introduced in Linux 5.15.

[tlaurion]

Hmm that's problematic if the ecosystem is moving things like that... Talos is stuck with 5.10 most probably forever for everything linuxboot based in initrd of coreboot first linux based payload. Not sure how to handle that.

[marmarek]

Theoretically I could force mkfs in R4.3 to disable this feature on /boot. But that's very short term workaround and wont help with other distributions...

[tlaurion]

Will dig at my return. @JonathonHall-Purism https://news.ycombinator.com/item?id=39566205....
White rabbit: https://www.kernel.org/doc/Documentation/filesystems/ext4/orphan.rst

[marmarek]

Some info also here: https://git.kernel.org/torvalds/c/02f310fcf47fa9311d6ba2946a8d19e7d7d11f37

[marmarek]

Note that the change is backwards compatible when the filesystem is
clean - the existence of the orphan file is a compat feature, we set
another ro-compat feature indicating orphan file needs scanning for
orphaned inodes when mounting filesystem read-write. This ro-compat
feature gets cleared on unmount / remount read-only.

So, maybe the issue applies only to unclean shutdown? I'll check few more cases.

[natterangell]

Merely chiming in to confirm this issue affects me too. While testing the flashprog rom in #1769 I noticed the autoboot feature working, and couldn't understand why it didn't work after flashing the master zip back. I've had a couple of unclean shutdowns on the T420 I have due to poor battery.

I get the "couldn't mount RDWR because of unsupported optional features (10000)" when trying to reseal disk key to TPM.

I'm on Fedora Silverblue, installed a week ago with EXT4, 6.10 kernel.

[natterangell]

Well for what it's worth, my error went away after manually unmounting and mounting boot partition from the running os.

[marmarek]

At this point I can also confirm the issue applies to unclean shutdown only. But it's still a problem...

[tlaurion]

Hmm that's problematic if the ecosystem is moving things like that... Talos is stuck with 5.10 most probably forever for everything linuxboot based in initrd of coreboot first linux based payload. Not sure how to handle that.

Talos II would need kernel bump to 6.6.16+new patches or go unmaintained.

https://gitlab.raptorengineering.com/openpower-firmware/machine-talos-ii/op-build/-/commit/f89ea5cc89392de0d1bd6a554009809b5f68692c

[tlaurion]

@marmarek following discussions at qubesos summit https://youtube.com/watch?v=mAb_kHrF6SQkernel, kernel version bump for all boards would need to happen prior of feature freeze for next firmware release prior of qubesos 4.3 being released, downstream planning to do 1-2 releases a year.

When is QubesOS 4.3 planned so that downstream forks boards roms include needed kernel version bump?

[marmarek]

When is QubesOS 4.3 planned

There is no specific schedule set yet, but the optimistic scenario is feature freeze at the end of the year, and then 3-4 months of stabilization (release candidates etc).

[SergiiDmytruk]

Don't see it being mentioned, but this ext4 feature can be disabled on an existing filesystem:

tune2fs -O ^orphan_file /dev/your-device

Not that it's a solution, but a workaround for anyone annoyed by this.

[tlaurion]

@macpijan #1802 work still needed even though all other boards kernel version bumped to 6.1.8