Possibly use static pods for temporary control-plane rather than compiled-in components

[aaronlevy]

We currently compile the control-plane components (api-server, scheduler, controller-manager) as part of the bootkube binary. This works well enough as a means of running the temporary control plane, but it also adds some complications.

The original reason for doing this is that we wanted bootkube to be a "network only" tool that we might be able to use remotely via ssh-tunneling / reverse proxies. A hypothetical situation would be running bootkube locally (or on a "seed" box) then via an ssh reverse proxy, have a remote kubelet bootstrap a self-hosted cluster by connecting to this external control-plane on local interfaces (requiring bootkube to have no direct access to remote volumes).

Over time, it seemed like the above wasn't strictly necessary - and it was reasonable to expect an operator to ssh into a box, run bootkube once, then exit. This also had the benefit of being a relatively simple workflow.

However, compiling the components into bootkube makes installing varying versions of kubernetes difficult (without re-vendoring / recompiling the tool). Then also, if we want to start self-hosting other tools (e.g. etcd) we may also need to compile these into bootkube itself.

Alternatively, we could have bootkube manage the temporary control-plane using static pods. For example:

1. Bootkube starts, and drops api-server, scheduler, controller-manager, and etcd pod manifests into: /etc/kubernetes/manifests.
2. This static (and temporary) control-plane starts, then bootkube injects daemonsets, deployments, secrets, & configMaps which make up the self-hosted control-plane.
3. Bootkube waits for the self-hosted control-plane to start, and when all required pods are running, bootkube deletes the static manifests in /etc/kubernetes/manifests and exists (leaving us with a self-hosted cluster)

---

I'm not sure this is the route we want to take -- but it's relatively simple, and would work today without any upstream changes (e.g. adding a kubelet-pod-api might be an alternative sometime in the future).

This also might be more palatable to a tool like kubeadm to launch a self-hosted cluster by shuffling around static assets, rather than running a compiled-in control-plane.

[ericchiang]

I'm for this. It helps users see and understand what bootkube is doing as well, which is an extremely important part of using the tool.

[dhawal55]

@aaronlevy I like this too. Are you going to pick this up?

[aaronlevy]

I was hoping to sometime in the next few weeks -- but haven't currently picked this up. Is this something you were interested in working on?

It might end up being a fairly large change -- so if it is something you wanted to take a look at, might help to start out with straw-man proposal / brief design doc before spending too much time coding.

[aaronlevy]

One blocker here that was discussed is that we can't easily do this in combination with kubelet TLS bootstrapping. The kubelet would need to start the temporary api-server (as a static pod) - but the kubelet never enters its pod sync loop because it's waiting on the TLS bootstrap.

There is likely a solution here - but just wanted to call out the potential issue

[jamiehannaford]

@aaronlevy I was just looking at this and it seems that the TLS bootstrapping logic is optional. So if the --experimental-bootstrap-kubeconfig flag is left blank for the initial system kubelet, it would just generate the self-signed keypair and that would be enough for it to enter the pod sync loop, since no pre-existing API server needs to approve anything. Unless I'm missing something

[jamiehannaford]

If this is still a viable and wanted feature, I'd like to take this on

[aaronlevy]

@jamiehannaford this is still wanted - but also something that is being worked on currently. I'll update the issue assignments (need to figure out best way of handling this given not everyone in kubernetes org and won't let me assign otherwise).