Automatic Network Configuration in Cloud with NetworkManager (nm-cloud-setup)

[dustymabe]

The networkmanager community reached out to the Fedora Cloud community to ask about a proposed nm-cloud-setup utility/service: https://lists.fedoraproject.org/archives/list/[email protected]/message/FSQR6KL4KA37WHTUAXL774SXSWIBSYGI/

• How does this fit in with what we are doing already with afterburn?
• Does this have any implications on us for ignition?

[dustymabe]

cc @lucab @ajeddeloh @bgilbert @jlebon

[lucab]

The tool they started writing may have indeed quite some overlap with Afterburn, but at the moment they seem to target a different niche/usecase.

From a quick skim, relevant differences/points from my side are:

• Afterburn network configuration happens in initramfs.
• Afterburn is a non-recurring oneshot service.
• Afterburn currently only integrates with systemd-networkd (but NM is in progress due to Network Management #24).
• Afterburn currently only supports DigitalOcean, Packet and IBM Cloud.
• Afterburn is written in a memory-safe and strongly-typed language.
• Afterburn does not have a dedicated development team, nor deep network expertise among its developers.

As it stands, nm-cloud-setup seems to be designed as a recurrently-running network-configuration reconciliation tool, while Afterburn network logic tries to take care of one-time bootstrapping the network in early-boot.

/cc @thom311 @bengal

[thom311]

As it stands, nm-cloud-setup seems to be designed as a recurrently-running network-configuration reconciliation tool, while Afterburn network logic tries to take care of one-time bootstrapping the network in early-boot.

nm-cloud-setup can be called once/manually, but I think a main purpose to pick up changes dynamically. For that it can be triggered via a systemd.timer or a NetworkManager dispatcher script. If you can afford to pre-generate static configuration early at boot, then you don't really need this tool. You could instead just pre-generate the static connection profiles.

Another difference is that the configuration created by nm-cloud-setup is not persisted to file system (not even to /run/NetworkManager). It uses the Reapply() API of NetworkManager to perform runtime only changes. That was done intentionally, it seems that the information that we fetch from the meta data should not be persisted to disk. What is does is similar to calling nmcli device "$IFNAME" modify ipv4.routes ... ipv4.addresses ... ipv4.rules ....

I think another purpose of the tool is to provide a solution for users who use NetworkManager but don't use something like cloud-init/Afterburn.

[lucab]

Some time ago we did some work to add support in Ignition for platform-specific fragments base.platform.d/ in (coreos/ignition#1108) and to verify whether it was powerful enough to enable+configure nm-cloud-setup.

The result of that exploration (coreos/fedora-coreos-config#760) was positive, so there should be no further blockers if we want to proceed. In the meanwhile, upstream added a docpage with all the details about that component: https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-cloud-setup.html.

However we didn't reach consensus on whether we wanted to 1) ship the binary, plus 2) ship the fragment for auto-configuration, plus 3) enable the timer unit by default on FCOS. I'm putting this up for discussion for the next IRC meeting.

[lucab]

In the last IRC meeting we reached consensus that the next steps for nm-cloud-setup integration are:

• refresh fedora-coreos-base: add nm-cloud-setup, forward platform variables fedora-coreos-config#760, keeping only the new binary and automatic platform env flags forwarding
• once landed in the OS, add kola tests for the platforms where we have CI (AWS and GCP) so that we are confident the cloud-specific logic works
• re-evaluate whether we can automatically enable the timer unit, or if OKD/OCP needs to fix their GCP logic first

[cgwalters]

xref https://bugzilla.redhat.com/show_bug.cgi?id=1977984#c56 - may have some negative interactions with OpenShift SDN and I suspect more generally anyone doing nontrivial networking setup outside of the OS.

If we did add this it may warrant a "provisioning discontinuity" at least where only new nodes get it? (like cgroups v2)

Of note though if we did add it, disabling it should just be masking the systemd unit.