Tailscale Fails Health Check on F41

[AlexNPavel]

Describe the bug

When running tailscale status on system running aurora-dx:latest (41.20241030.0), tailscale prints this failed health check:

# Health check:
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/usr/sbin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.

Tailscale pings seem to work correctly otherwise, but DNS does not get configured. I mainly use tailscale for a remote pihole, so I noticed that wasn't working anymore.

What did you expect to happen?

Tailscale runs without any errors.

Output of rpm-ostree status

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: inactive
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx:latest
                   Digest: sha256:200c66f230f15fb77450dea9d0910f6992d123a1bbef8d3ce084955214533e43
                  Version: 41.20241030.0 (2024-10-30T04:44:13Z)
          LayeredPackages: bees btop byobu

  ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx:latest
                   Digest: sha256:b169896d19967cd89066cf339712f0b19526a4f41a6b5508d421c4cbd7c7e693
                  Version: 41.20241029.0 (2024-10-29T18:27:51Z)
          LayeredPackages: bees btop byobu

Output of groups

apavel wheel docker incus-admin lxd libvirt

Extra information or context

No response

[AlexNPavel]

Seems to be a known issue with the latest kernels, so we may just need to wait: tailscale/tailscale#13863

[castrojo]

:stable and :gts are on 6.11.3, here's how to rebase: https://docs.projectbluefin.io/administration#upgrades-and-throttle-settings

[bb010g]

The recently released 6.11.6 includes a fix for this.

[castrojo]

@bsherman @m2Giles Let's pin to 6.11.3 on gts/stable before the F41 promotion next week, confirmed that :latest is affected.

[m2Giles]

Will add in the pr

[bsherman]

@bsherman @m2Giles Let's pin to 6.11.3 on gts/stable before the F41 promotion next week, confirmed that :latest is affected.

I'm going to be shocked if Fedora CoreOS stable rolls forward with a kernel having this bug.

But it won't hurt to test the pinning.

[inffy]

This should be fixed in 6.11.6 which has been sent to Fedora stable just now (not on stable repos yet)

[tulilirockz]

Latest is currently on 6.11.5 😭 so close

[inffy]

Latest is currently on 6.11.5 😭 so close

Latest is on .6 since like a day ago 😁

Can confirm that atleast here there are no errors in Health check

[tulilirockz]

It works completely fine now!

[castrojo]

The stable stream moved to 41 but unfortunately it's on 6.11.5: https://fedoraproject.org/coreos/release-notes?arch=x86_64&stream=stable

I'm on the beta stream I can confirm that it is still in an issue. Now the question is, do we pin to .3 or .6 in stable before we promote?

[m2Giles]

#1921 pins the kernel on GTS and Stable.

For Stable this means staying on F40 since there is not a 6.11.3 kernel for F41 coreos-stable right now.

[befanyt]

I see Fedora CoreOS stable stream has moved to 6.11.6, so that would solve this issue.

[castrojo]

Feels like we almost have enough people to be able to realistically pin major regressions relatively quickly. Every time we do one of these it means the model is protecting users, that's pretty awesome!

Cheating with the weekly builds-as-a-default continues to pay dividends. 😄