Actually move iptables to the nft backend

[dustymabe]

In Fedora 32, Fedora moved the backend of iptables to use the nft implementation by default (see the change proposal. In #26 we decided to ship all the tools and accept the Fedora defaults.

Unfortunately, applying the default of iptables-nft in Fedora 32 didn't work on FCOS because it was implemented via alternatives and alternatives don't currently work with the way rpm-ostree composes the OS (see #677).

Now that we know we're misconfigured, let's figure out how to move to get back in line with Fedora's defaults while we wait on alternatives to get fixed (probably won't happen til Fedora 34).

There were some previous musings about this (before the f32 change landed) in #342.

Rolling out:

• wait until scheduled migration date for next-devel (when rebasing to f36)
• merge overlay.d: add 35coreos-iptables fedora-coreos-config#1324
• wait until the new overlay dir is propagated to next-devel
• rebase and merge [next-devel] manifest: default to iptables-nft fedora-coreos-config#1460
• send email (draft)
• cut next release, possibly making it a barrier
• revert 35coreos-iptables inclusion in next-devel's manifest.yaml (second commit in [next-devel] manifest: default to iptables-nft fedora-coreos-config#1460)
• wait until scheduled migration date for testing-devel (when rebasing to f36)
• lower iptables-nft postprocess into a shared manifest into testing-devel and out of next-devel's manifest.yaml (first commit in [next-devel] manifest: default to iptables-nft fedora-coreos-config#1460)
• add 35coreos-iptables inclusion to testing-devel's manifest.yaml
• cut testing release, cherry-picking 35coreos-iptables inclusion to its manifest.yaml, and possibly making it a barrier
• revert 35coreos-iptables inclusion to testing-devel and testing
• cut stable release, cherry-picking 35coreos-iptables inclusion to its manifest.yaml, and possibly making it a barrier
• revert 35coreos-iptables inclusion to stable

[dustymabe]

One proposal:

1. leave existing systems on the old implementation (similar to the way the Fedora Change intended)
2. group the switch to iptables-nft into the f33 rebase on the stable stream (would need to go through testing first)

Either way let's discuss it in next week's meeting to see what the most appropriate path forward is.

[dustymabe]

cc @nullr0ute for awareness - since this likely affects IoT

[jlebon]

I think conceptually we want the default in the OSTree commit to be nft, and have upgraded systems therefore carry a delta in their /etc for staying on legacy. This is tricky to do though because the symlinks are managed by OSTree itself, so as soon as we change the commit, upgrading nodes will inherit the change.

Hmm, I think to implement this we'd need two barrier releases:

• do a first barrier release with a systemd service which marks a stamp file at e.g. /var/lib/iptables-nft.stamp if the machine is already using the nft backend
• in a second barrier release
  • we add a postprocess script that does ln -sf as needed to link /etc/alternatives/iptables to /usr/sbin/iptables-nft (and similarly for the related symlinks) -- once alternatives becomes OSTree-compatible, that postprocess script should no longer be necessary
  • we add a systemd service in the initramfs which determines whether to keep with the new default (nft), or revert back to legacy
    • if it's the first boot, then do nothing (we want freshly booted machines to use nft)
    • if /sysroot/var/lib/iptables-nft.stamp is present, then delete the stamp file and do nothing (we want upgrading machines which previously used nft to stay on nft)
    • if the symlinks are already different from nft, then do nothing (we don't want to touch upgrading machines which previously somehow used neither nft nor legacy, just to be safe)
    • otherwise, revert the /etc symlinks back to legacy; this will mark the symlinks as modified as per ostree admin config-diff

[jlebon]

Another approach which requires user action is a single barrier where we add a service which optionally sets the backend to legacy if some stamp file in /var is present. Then we inform users that if no action is taken, their nodes will be moved to iptables-nft. Otherwise, they can touch that stamp file before $release_date. This is more aggressive, though is much simpler in implementation. (I personally lean towards this).

[dustymabe]

• do a first barrier release with a systemd service which marks a stamp file at e.g. /var/lib/iptables-nft.stamp if the machine is already using the nft backend

Could we get away with doing just one barrier release if we just take a peak at what's in the rollback deployment's /etc/ to determine if it was using nft already or not?

[jlebon]

Could we get away with doing just one barrier release if we just take a peak at what's in the rollback deployment's /etc/ to determine if it was using nft already or not?

Hmm, I think that would work in 99% of cases, but it's not entirely foolproof, because there are no guarantees about correspondence between "previous deployment" and "rollback deployment" (what's considered the rollback is more a convention based on deployment ordering) and whether it still exists.

[jlebon]

This was discussed in the community meeting today:

AGREED: we will stay with the legacy backend for now while we pursue a fixed alternatives for OSTree systems. meanwhile, we will document how users can move to the nft backend

[travier]

Upstream issue for easier reference: fedora-sysv/chkconfig#9

[cgwalters]

If we switch the default to nft, do we then document+support a way to stay on legacy for new systems too? It'd probably make sense because we need to carry most of that logic for previous systems too, but it is a different case.

[ibotty]

Without having anything to say: why support legacy iptables at all? AFAICT there are more bugs in legacy iptables and many features get easier with nft. Most people would not notice anyway.

Isn't current RHEL and RHCOS also on nft? I don't see a reason for legacy iptables.

[nullr0ute]

cc @nullr0ute for awareness - since this likely affects IoT

We actually use firewalld, as opposed to an iptables based static firewall, and it defaults to the nft backend so I'm not sure how much this applies by default to IoT. Thanks for the heads up, let me know if I've missed something.

[dustymabe]

If people who know much more about this than me tell me that switching the nft on an upgrade is a relatively safe thing to do then we could consider that option. Keep in mind that FCOS auto-updates, so the goal is to not break people's systems that are updating themselves. Do we have a high degree of confidence that iptables-nft is a safe upgrade?

/me wonders why the rpm doesn't just default to it now instead of relying on updating priorities via alternatives in %post scriptlets.

[dghubble]

Host iptables legacy vs nft has been a pain point in Kubernetes-land for years (example). kube-proxy should detect the mode in theory. Other network-related apps, components, or tools need to be prepared as well (e.g. CNIs). I'd have a healthy fear on this topic.

[champtar]

Not all CNI autodetect, and they might not autodetect by default.
Also k8s-dns-node-cache support autodetection only since 1.16.0 / nov 16 (kubernetes/dns#367)
You can today run with iptables-nft, but switching automatically will break some prod :)

[jlebon]

So if we want to err on the side of caution and not auto-migrate systems on legacy but only affect new nodes, that takes us back to something like #676 (comment).

[jdbarnes-isi]

Looking forward to the release that has this default. Will it be in whatever is after Stable v 35.20220131.3.0 ? Is it already in Testing v 35.20220213.2.0?

[dustymabe]

It will be when the rebase to Fedora Linux 36 happens. See the email Jonathan sent out.

[travier]

@jdbarnes-isi Note that you can switch to the new backend today with https://docs.fedoraproject.org/en-US/fedora-coreos/alternatives/

[dustymabe]

The fix for this went into next stream release 36.20220325.1.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 36.20220505.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 36.20220505.3.2.