Add MutatingWebhookConfiguration objects to install YAML

[klingerf]

Now that #2163 has merged, the way in which the proxy-injector's MutatingWebhookConfig is provisioned is much simpler. It now works as follows:

• linkerd install --proxy-auto-inject emits the config for the proxy-injector pod
• Once the proxy-injector pod starts, it creates a new CA bundle in memory
• It then calls the Kubernetes API to create or update the MWC, which contains the CA bundle

This means that our install YAML does not include the MWC itself, since it's created programmatically after the proxy-injector pod starts. Ideally, however, the install YAML would include all objects that are actually installed, including the MWC. But this would require the CLI to create the bundle at install time and provide it as a secret to the linkerd-proxy-injector pod.

[klingerf]

Relates to #2164.

[adleong]

I think the way this should work is that at install time, Linkerd install would generate a root CA and key, create the MWC with the CA (trust root) and save the key into a secret. When a webhook instance starts up, it should load the CA key from the secret, generate a serving key and CSR, and use the CA key to generate a serving cert.

The same approach should be used for validating webhooks and for api extensions. This is a prerequisite to running any of these in HA mode.

Does this seem right?

[adleong]

How would this interact with linkerd upgrade? For upgrade to give consistent results, the root CA would have to be stored in the linkerd config configmap, or upgrade would need to read it from the existing MWC, right?

[alpeb]

@adleong Your reasoning sounds right to me. As for linkerd upgrade, it appears the approach is similar to what we already have for Identity (with the difference that we have the option to not include the CA in the configmap if it's already in the MWC, like you pointed out). There's an ongoing discussion in #2679 about upgrade and secrets that might affect this issue as well.

[ihcsim]

I agree that in HA mode the root CA needs to be persisted. I'm not sure about persisting the key though, because we have a tls package that can generate in-memory cert and private key for the CA. IMO, unless necessary, the less we write out to config maps/secrets etc., the better it is for things like upgrade.

or upgrade would need to read it from the existing MWC

If other components will be sharing this CA, it cannot be saved in the MWC, because the MWC gets re-created everytime the webhook is restarted. See #2425.

[adleong]

In HA mode, there will be one MWC and multiple webhook pods. This means that all pods must be serving using certificates signed by the same CA.

When a webhook pod starts up, we dont want to regenerate the MWC because changing the CA cert in the MWC will mean that other pods will no longer be serving certs signed by the new CA.

When a webhook pods starts up, it needs to get a certificate signed by the CA. This is only possible by using the CA's private key. This is why we have to persist that private key in a secret rather than generating a new one for each pod.

Does this make sense?

[ihcsim]

Per slack convo with @adleong, we can't assume that the MWC and VWC won't get regenerated, especially during upgrade. It will make sense to save the CA cert in the linkerd-config config map, as all the data in there are persisted across upgrades.

And yes, the CA private key needs to be saved because the pkg/tls package works for newly created in-memory TLS assets, with nothing on re-using existing CA to sign certs.

[grampelberg]

I've been hesitant about the key being in secrets, got talked into it yesterday. It does suggest that we want to strongly encourage folks to install the control plane into its own namespace, even if it isn't named linkerd explicitly.