Make BootFromDisk method overridable using ServerClass/Environment configs

[keithamus]

When a machine has been allocate and provisioned, Sidero calls into the bootFromDiskHandler to tell the machine to boot from it's provisioned disk. There are 3 methods which are somewhat hardware dependant:

https://github.com/talos-systems/sidero/blob/d0df929eb1d1067636caaa2a95c7126be6c02713/app/sidero-controller-manager/internal/ipxe/ipxe_server.go#L89-L100

BootFromDisk here is configured by command line flag in the sidero-controller-manager:

https://github.com/talos-systems/sidero/blob/d0df929eb1d1067636caaa2a95c7126be6c02713/app/sidero-controller-manager/internal/ipxe/ipxe_server.go#L130

(defaultBootFromDiskMethod is set in RegisterIPXE):
https://github.com/talos-systems/sidero/blob/d0df929eb1d1067636caaa2a95c7126be6c02713/app/sidero-controller-manager/internal/ipxe/ipxe_server.go#L192

(RegisterIPXE is called from main()):
https://github.com/talos-systems/sidero/blob/ab12b81ef00ad1762aaa251fbaa6b022c298ae62/app/sidero-controller-manager/main.go#L224

(bootFromDiskMethod is a StringVar and so set via CLI flags):
https://github.com/talos-systems/sidero/blob/ab12b81ef00ad1762aaa251fbaa6b022c298ae62/app/sidero-controller-manager/main.go#L93

The problem here is in JBOC ("just a bunch of computer") setups hardware may vary to the point of incompatible PXE implementations. For example in my testing RPis with usb mSata drives like the ipxe-sanboot boot from disk method, while Intel NUC or similar like the ipxe-exit method.

Right now there are only a few options which are all non-ideal:

• Provision machines and after provisioning manually change the bios/efi to point to disk as the first boot option for some/all of your machines, going against part of the reason for Sidero
• Somehow load balance between two copies of Sidero (e.g. by changing the dhcp-next-server) to differently configured pods based on the environment
• Throw out hardware and buy stuff that all sings from the same hymn sheet.

Ideally Sidero would allow this option to be overridden per-enviornment. This would mean extending the Environment type here:

https://github.com/talos-systems/sidero/blob/acb5f57f65a0a226d604ad124f189afe5752974a/app/sidero-controller-manager/api/v1alpha1/environment_types.go#L83-L96

to include additional boot options (perhaps using bootOptions as a key?) which could then provide a bootMethod key, which would override the defaultBootFromDiskMethod. I believe it'd look a little something like:

- bootFromDiskHandler(defaultBootFromDiskMethod, w, r) 
+ bootFromDiskHandler(env.Spec.BootOptions.BootMethod, w, r) 

[smira]

thank you, that makes perfect sense, I added it to our board to include in the next Sidero release