Certificates rotation with Salt beacon & reactor #1887

alexandre-allard · 2019-10-10T08:47:02Z

Component: Salt

Why this is needed: To be able to rotate metalk8s certificates and generate short living certificates to improve the security of the metalk8s cluster.

What should be done: A system that handle the certificate rotation

Implementation proposal (strongly recommended): Based on Salt beacon & reactor

Test plan: TBD

This beacon will trigger an event with all master role related certificates already expired or that will expire in less than 15 days. This event will then be catched by a reactor which will handle the certificate renewals. Refs: #1887

This beacon will trigger an event with all etcd role related certificates already expired or that will expire in less than 15 days. This event will then be catched by a reactor which will handle the certificate renewals. Refs: #1887

This beacon will trigger an event with all ca role related certificates already expired or that will expire in less than 15 days. This event will then be catched by a reactor which will handle the certificate renewals. Refs: #1887

This beacon will trigger an event with all bootstrap role related certificates already expired or that will expire in less than 15 days. This event will then be catched by a reactor which will handle the certificate renewals. Refs: #1887

This pillar entry will be consumed by the reactor listening for certificate expiration events. If the path of an expired certificate matches one in this list, the related `sls` will be run. Refs: #1887

This orchestrate will be called by the reactor when it will receive an event for an expired certificates. It will run `sls` defined under `certs_renewal` pillar entry for each expired certificate. Refs: #1887

This reactor will be called when an expired certificate event will be received. It will then launch an orchestrate `orchestrate.certs.renew`, passing the list of expired certificates, to renew them. Refs: #1887

Set up the configuration in salt master cfg for the certificate expiration reactor. Refs: #1887

This beacon will trigger an event with all bootstrap role related certificates already expired or that will expire in less than 15 days. This event will then be catched by a reactor which will handle the certificate renewals. Refs: #1887

This pillar entry will be consumed by the reactor listening for certificate expiration events. If the path of an expired certificate matches one in this list, the related `sls` will be run. Refs: #1887

This orchestrate will be called by the reactor when it will receive an event for an expired certificates. It will run `sls` defined under `certs_renewal` pillar entry for each expired certificate. Refs: #1887

This reactor will be called when an expired certificate event will be received. It will then launch an orchestrate `orchestrate.certs.renew`, passing the list of expired certificates, to renew them. Refs: #1887

Set up the configuration in salt master cfg for the certificate expiration reactor. Refs: #1887

This beacon will trigger an event with all bootstrap role related certificates already expired or that will expire in less than 15 days. This event will then be catched by a reactor which will handle the certificate renewals. Refs: #1887

This pillar entry will be consumed by the reactor listening for certificate expiration events. If the path of an expired certificate matches one in this list, the related `sls` will be run. Refs: #1887

This pillar entry will be consumed by the Salt formulas configuring etcd, the beacon and the reactor listening for certificate expiration events. If the path of an expired certificate matches one in this list, the sls under `regen_sls` will be run. Refs: #1887

This pillar entry will be consumed by Salt formulas configuring master nodes, beacon and reactor listening for certificate expiration events. If the path of an expired certificate matches one in this list, the sls under `regen_sls` will be run. Refs: #1887

Replace hardcoded path for calico kubeconfig in the related formulas, using the new entries under certificates key in the `defaults.yaml` file. Refs: #1887

Replace hardcoded path for kubelet kubeconfig in the related formulas, using the new entries under certificates key in the `defaults.yaml` file. Refs: #1887

This orchestrate will be called by the reactor when it will receive an event for an expired certificates. It will run `sls` defined under `certs_renewal` pillar entry for each expired certificate. Refs: #1887

This reactor will be called when an expired certificate event will be received. It will then launch an orchestrate `orchestrate.certs.renew`, passing the list of expired certificates, to renew them. Refs: #1887

Set up the configuration in salt master cfg for the certificate expiration reactor. Refs: #1887

Set up the configuration in salt master cfg for the kubeconfig expiration reactor. Refs: #1887

This tests reconfigure the beacons and override the pillar configuration to force the renewal of all the certificates and kubeconfigs. The goal is to ensure that beacons work well and that nothing is broken in the cluster even when everything is triggered at the very same time. Refs: #1887

This way this method can be called by anything to check if the kubeconfig file needs to be regenerated. The purpose is to be able to call it from a custom beacon to check whether embedded certificates are expired or not. Refs: #1887

It allows to dynamically pass the number of days left to consider the certificates in kubeconfig as expired. This change is needed for the custom beacon used to watch kubeconfig expiry as this parameter will be configurable like it is done for other beacons (e.g. cert_info). Refs: #1887

We now only check the CN in the kubeconfig certificate if the `expected_cn` is not None. This is needed in order to avoid checking the CN in the kubeconfig beacon which will rely on this method. Refs: #1887

This will allow to change the time a kubeconfig is valid (by changing the embedded certificates validity period). Refs: #1887

This beacon takes a list of kubeconfig as input and checks whether they need to be renewed or not triggering an event on Salt bus when needed. To configure this beacon, a section must be added, either in minion configuration or through the pillar, as follows: beacons: metalk8s_kubeconfig_info: - files: - /etc/kubernetes/calico.conf - /etc/kubernetes/admin.conf: notify_days: 30 - interval: 86400 - notify_days: 15 Default notify_days, if not provided, is 45. It can be overridden for a specific kubeconfig as shown above. Refs: #1887

Refs: #1887

These defaults will be merged with the pillar and can be overriden, they'll be used by both the certificate & kubeconfig expiry beacons, the related reactor (certs renewal) and the `x509.certificate_managed` state. Refs: #1887

This sls setup the two beacons used to watch certificates and kubeconfig expiry. We also need to install pyOpenSSL package for cert_info beacon to work. Refs: #1887

Since this is needed on almost any node, let's deploy the beacons on all nodes, if there is no certificate to watch it will do nothing anyway. Refs: #1887

This pillar entry will be consumed by the Salt beacon configuration formula. This beacon watches certificate expirations. Refs: #1887

This pillar entry will be consumed by the Salt formulas configuring bootstrap role nodes, the beacon and the reactor listening for certificate expiration events. If the path of an expired certificate matches one in this list, the sls under `regen_sls` will be run. Refs: #1887