Please document need for add. RPMs with CNI plugins when using podman in Fedora Silverblue 36

[Unveraenderbar]

Under Fedora 36 Silverblue, additional RPMs containing CNI plugins have to be installed in order to get a kind cluster up:
For release 36, the Fedora Silverblue team decided to remove the RPMs "containernetworking-plugins" and "podman-plugins" from the default OS image base layer (they were included in earlier Silverblue releases).
As a consequence, before adding these packages e.g. by

rpm-ostree install containernetworking-plugins podman-plugins

and rebooting into the extended image, kind install will fail as the internal network is not functional.
I'd like to suggest to add a note on this to the documentation on using podman as runtime for kind.

[aojea]

But ... That means that podman network will not work... That's independent of kind

[Unveraenderbar]

Yes, I think that's true... I also thought already about opening an issue against the Sivlerblue 36 OS base layer set.
But as the Fedora toolbox works w/o problems, stripping the CNI plugins from the base layer might have been done deliberately so I thought adding a note to the kind documentation could be useful.

[Unveraenderbar]

Wait -- it turns out to be a consequence of upgrading podman to version 4.x in Silverblue 36, which in turn means that the default network backend is netavark and no longer CNI. So podman does not need the CNI plugins except when it finds existing containers that are configured for CNI-based networking (quite possible when upgrading a Silverblue based system to 36), in which case it falls back to CNI, and consequently fails to setup container networks due to the missing plugins.

Details are given here, and I checked that after

• deleting all containers,
• pinning the networking backend to netavark,
• removing the CNI plugin RPMs from the OS image and rebooting

kind install works fine.

I'm no longer sure what the best way to document this in kind would be... maybe the procedure given above is cleaner than installing additional RPMs. OTOH, if one wants to operate an existing kind cluster over a Silverblue upgrade, there's probably no other choice than to extend the OS base layer RPM set.

[BenTheElder]

This doesn't really sound in-scope, this sounds like the Linux distro botched upgrades and should document it or fix it ... We're not even telling podman we want any particular networking implementation just that we want a network.

[BenTheElder]

I think "existing containers are not broken by OS / package upgrade" is very much something the OS / package should be handling ...

[aojea]

it seems we have to add a new additional check on the podman provider based on the linked issue #2821 (comment)

Please run and check podman info (shortcut: podman info | grep -i -A3 net) to see what networking mode you use. If it still says CNI, you should likely switch.

[BenTheElder]

We could add another "known issues" entry, as we already have a section dedicated to Fedora breakages, but I think this is something that should be fixed in the upstream packaging. If podman is still defaulting to CNI as https://discussion.fedoraproject.org/t/how-to-get-podman-dns-plugin-container-name-resolution-to-work-in-fedora-coreos-36-podman-plugins-podman-dnsname/39493/6#why-does-not-it-switch-by-default-4 says, then it really ought to keep shipping the required binaries??

[BenTheElder]

I think we're leaving this to podman x Fedora. This isn't a kind issue.