Config map errors in kubelet after 365 days of operation

[thirstler]

Kubelet, Metalk8s version 2.4.0:

Nodes went into "NotReady" state with kubelet logs indicating problems with configmaps:

Nov 04 06:39:44 HOSTNAME kubelet[6620]: W1104 06:39:44.503497    6620 reflector.go:302] object-"default"/"zenko-mongodb-replicaset-init": watch of *v1.ConfigMap ended with: too old resource version: 237820782 (237823171)
Nov 04 06:40:06 HOSTNAME  kubelet[6620]: W1104 06:40:06.729419    6620 reflector.go:302] object-"default"/"zenko-zenko-queue-metrics": watch of *v1.ConfigMap ended with: too old resource version: 237819870 (237823350)
Nov 04 06:41:09 HOSTNAME  kubelet[6620]: W1104 06:41:09.571876    6620 reflector.go:302] object-"kube-system"/"kube-proxy": watch of *v1.ConfigMap ended with: too old resource version: 237820584 (237823834)
...

The issue started exactly 365 days after the system was installed. Just the prior week we had run a 'salt-call state.highstate' on the servers to fix some certs (calico) that were due to expire at the 365 day mark.

A restart of the kubelet process fixed the issue. We were not able to capture how long the kubelet processes itself had been running before we restarted it.

[NicolasT]

Some comments for whoever will look into this further:

We're likely either not restarting the kubelet service after recreating its client cert (when applying state.highstate and the cert gets regenerated because it's about to expire), and/or we have kubelet cert-updating through K8s CSR API enabled, but requests are not being (automatically) accepted and certs minted by controller-manager (see https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/ and https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/).

We may want to disable the latter completely (assuming it's enabled today), and solely rely on Salt-provisioned certs.

Also note kubelet has some state (including X509 stuff) in /etc and some in /var/lib/kubelet.

[c2rohwer]

Just wanted to add some additional info here:

Nov 5th: The Certs were updated, state.highstate was applied, and we then restarted kubelet on those servers:

Nov 05 09:47:29 HOSTNAME systemd[1]: Started kubelet: The Kubernetes Node Agent.
Nov 05 09:47:29 HOSTNAME systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Nov 05 09:47:29 HOSTNAME systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Nov 05 09:47:29 HOSTNAME kubelet[23139]: I1105 09:47:29.538510  23139 server.go:425] Version: v1.15.3
Nov 05 09:47:49 HOSTNAME kubelet[23139]: I1105 09:47:49.904387  23139 server.go:1083] Started kubelet

Nov 11-12th: 3 servers ended up in a not ready state:
There were 3 hosts in the environment that ended up in a NotReady state. The first two got hung up some time November 11th so we got kubelet restarted on those two to clear the NotReady state on November 12th. Then the third node got hung up on Nov 12th and here are some the logs lines around the time of the hang. The 3rd server then had kubelet restarted.

The log info from November 12th on the 3rd server a short time before the restart of kubelet. These messages were also on the first two servers as well, but lots of things logged between these and the actual restart of kubelet due to hang happening the day before:

Nov 12 15:00:46 HOSTNAME kubelet[23139]: E1112 15:00:46.139105  23139 controller.go:125] failed to ensure node lease exists, will retry in 200ms, error: Unauthorized
Nov 12 15:00:46 HOSTNAME kubelet[23139]: E1112 15:00:46.340296  23139 controller.go:125] failed to ensure node lease exists, will retry in 400ms, error: Unauthorized
Nov 12 15:00:46 HOSTNAME kubelet[23139]: E1112 15:00:46.741467  23139 controller.go:125] failed to ensure node lease exists, will retry in 800ms, error: Unauthorized
Nov 12 15:00:47 HOSTNAME kubelet[23139]: E1112 15:00:47.542495  23139 controller.go:125] failed to ensure node lease exists, will retry in 1.6s, error: Unauthorized
Nov 12 15:00:49 HOSTNAME kubelet[23139]: E1112 15:00:49.143560  23139 controller.go:125] failed to ensure node lease exists, will retry in 3.2s, error: Unauthorized

6 minutes later was restarted by the customer - HOSTNAME - Restarted kubelet:

Nov 12 15:06:23 HOSTNAME systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Nov 12 15:06:23 HOSTNAME systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Nov 12 15:06:23 HOSTNAME systemd[1]: Started kubelet: The Kubernetes Node Agent.
Nov 12 15:06:23 HOSTNAME kubelet[9173]: I1112 15:06:23.469741    9173 server.go:425] Version: v1.15.3
Nov 12 15:06:43 HOSTNAME kubelet[9173]: I1112 15:06:43.862186    9173 server.go:1083] Started kubelet

If additional info from the data dump to a file from the 'journalctl -u kubelet' command from the three servers is needed please let me know.

[NicolasT]

That Unauthorized is interesting, it's as-if the kubelet is still using the old cert for some reason. I re-read https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/ (@scality/metalk8s team should as well ;-)), and checked stuff in /var/lib/kubelet, and there I find a client cert with a symlink (kubelet-client-current.pem) to it, whose contents comes from /etc/kubernetes/kubelet.conf (cat /etc/kubernetes/kubelet.conf | awk '/client-certificate-data/{print $2}' | base64 -d | openssl x509 -noout -text).

So, what could be the case, is that kubelet still uses that cert as long as it's valid, in the meantime tries to use the CSR API to renew it (kubectl get csr indeed lists a bunch of them!), and only falls back to the cert from /etc/kubernetes/kubelet.conf (at startup) once the on in /var/lib/kubelet/pki actually expired.

So, we either need to

• configure kubelet to not use the CSR API, and simply accept the authn information provided in /etc/kubernetes/kubelet.conf, which we manage through Salt like any other certificate (also rotated thanks to Automatically rotate certificates #2910)
• not use Salt to manage kubelet client certificates, and instead use (and properly configure) the built-in K8s CSR API and kubelet auto-rotation

Approach 1 above is more in line with other certificates we manage throughout the system, and likely 'easier' to implement. Approach 2 would be useful if at some point in time we move away from Salt managing all kinds of (client) certs, including e.g., those used in a Calico client config file, and instead use provisioning through Jobs (with a proper ServiceAccount and access to the CSR API, getting certs signed etc.).

[alexandre-allard]

I did some tests, as @NicolasT already said Kubelet is generating CSR but these requests are simply ignored by controller manager because Kubelet does not have sufficient right.
I successfully managed to make the rotation works by adding some cluster role binding for kubelet and options to controller manager.
I will do a PR today to fix this.
Not directly related to this issue, but we should also use --bootstrap-kubeconfig option instead of --kubeconfig and let kubelet handle its certificates once the bootstrap phase done.

[NicolasT]

@alexandre-allard-scality Well, I'm not entirely convinced what you suggest as a fix is definitely the way to go: indeed, this would make kubelet cert management be 'quite different' from how we manage other certs in the stack. Did you consider the alternative option, of disabling kubelet built-in cert management/rotation altogether, and solely rely on the Salt-based mechanisms we have in place (and also need for other types of certs) and, as such, need to properly industrialize? Using the built-in rotation means we need to maintain, monitor,... two distinct mechanisms.

[alexandre-allard]

@NicolasT AFAICT, there is nothing to maintain and for the monitoring part it's more or less the same as what we could do for other certificates (something checking the expiration date of the certificate, whether we are managing this cert or not).
But, I agree that we would have two distinct mechanisms and maybe that's not what we want.
The only drawback of handling Kubelet certificates with Salt is that we need to restart Kubelet when renewing its kubeconfig, but it should not be an issue.

[alexandre-allard]

Kubelet certificate rotation mechanism has been disabled in #2939, closing this issue as now a simple call to Salt highstate fixes that by renewing the Kubelet certificate and restarting it.
It will also be automatically done once #2914 will be merged.