certificate expired and rotate

[liyongxian]

Hi
By default,k3s components' certificates is 1 year,those components include:kube-apiserver,scheduler,cloud-controller,K3s should rotate those certificates.

K3s version: v1.17.4+k3s1

A clear and concise description of what you want to happen.

The components' certificates expire,k3s will not work.
Thanks.

[Bowser1704]

Hi
It bothers me, too.
Do you have a good way to update the certificate automatically?|
Thanks.

[brandond]

Certificates rotate automatically if they are <90 days from expiration when k3s is started. This has been the case since v0.10: #805

As long as you are patching and restarting your k3s infrastructure at least every couple months, you shouldn't run into expired certificates.

[Bowser1704]

@brandond
Does it mean we have to restart k3s if we want to update certificate?
Thanks.

[brandond]

Yes. Make sure you're running at least 0.10, and restart k3s. If the certs are less than 90 days from expiration they will be rotated when it starts. I don't believe there's a way to manually rotate them or make them valid for longer... other than poking at the files manually with a 3rd party tool like openssl of course.

You're restarting your k3s nodes at least every couple months to apply OS patches, right?

[Bowser1704]

@brandond
OK Thanks.

[ieugen]

Since the question has been answered, maybe the issue can be closed. cc @liyongxian .

[davidnuzik]

Closing the issue. If there are further questions let us know and we can reopen if necessary.

[ethinx]

Hi @brandond ,

I did a test on VM, modify system time to the day before expired time ( < 90 days) and restart k3s, then checking cert info by using openssl command as following, found the cert has no change.

openssl s_client -connect localhost:6443 -showcerts < /dev/null 2>&1 | openssl x509 -noout -enddate

I tried reset the time several times even clean up the certs in /var/lib/rancher/k3s/server/tls, but unlucky, k3s unable to start up. It seems that somewhere cached the certificates?

Jun  2 17:15:06 localhost k3s: E0602 17:15:06.593592    7212 reflector.go:153] k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.VolumeAttachment: Get "https://127.0.0.1:6444/apis/storage.k8s.io/v1/volumeattachments?limit=500&resourceVersion=0": x509: certificate has expired or is not yet valid: current time 2020-06-02T17:15:06Z is before 2022-04-26T23:02:59Z

Could you please advise if there is any misunderstanding?

Version: v1.17.5+k3s1

Thanks,

[brandond]

@ethinx did you set the time into the future, start k3s, and then set it back again? According to the error, your cert was generated almost two years in the future and is not yet valid:

x509: certificate has expired or is not yet valid: current time 2020-06-02T17:15:06Z is before 2022-04-26T23:02:59Z

[ethinx]

@brandond since I tried to clean up the certs under /var/lib/rancher/server/tls and screw things up.

Make it simpler, I did a fresh install and test again as following, do I miss something here?

# INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh

# openssl s_client -connect localhost:6443 -showcerts </dev/null 2>&1 | openssl x509 -noout -startdate -enddate
notBefore=Jun  3 13:27:43 2020 GMT
notAfter=Jun  3 13:27:43 2021 GMT

[root@localhost ~]# date -s 20210520
Thu May 20 00:00:00 UTC 2021
[root@localhost ~]# hwclock -w

[root@localhost ~]# date
Thu May 20 00:00:08 UTC 2021

# systemctl restart k3s

# openssl s_client -connect localhost:6443 -showcerts </dev/null 2>&1 | openssl x509 -noout -startdate -enddate
notBefore=Jun  3 13:27:43 2020 GMT
notAfter=Jun  3 13:27:43 2021 GMT

# k3s -v
k3s version v1.17.5+k3s1 (58ebdb2a)

[brandond]

@ethinx Don't set the time forward while it's running. If you're going to test it that way, stop k3s before changing the system time. Also, make sure you've disabled any time sync daemons - ntpd, chrony, systemd-timesyncd, etc.

[ethinx]

Hi @brandond,

It seems that not about the time sync daemons, even though I stop chronyd (I didn't notice it before), cert keep unchanged after I stop k3s -> set time to future -> start k3s.

I think I find where the cert be cached, it's the k3s-serving secret under namespace kube-system.

The cert get updated after I remove the secret and local cache file dynamic-cert.json. You could check the procedures in my gist: https://gist.github.com/ethinx/c0ede41a7a6a340681476ef564ea75f2

Just have a glance on the code, seems that the local file certificate generation is different to the secret.

Hope you could share more about it, as we don't have documents about this.

[brandond]

I think you might be correct. @galal-hussein @erikwilson it looks like the kube-apiserver, kubelet, and auth-proxy certs all call through certutil.LoadOrGenerateKeyFile which handles rotation, but the dynamiclistener/factory stuff that creates the k3s-serving key has its own CA loader and cert generation code that does not appear to check expiration or handle renewal.

[davidnuzik]

Reopening the issue based on comments it seems like we may want to investigate this further.

[ia-josephp]

Hi there,

Following the Rancher 2.X installation instructions for Rancher on K3s, in this document: https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/kubernetes-rke/, we get to copying the kubectl config to a local machine to try to connect and we receive the following error:

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2020-06-24T14:40:08-05:00 is after 2017-10-24T21:09:52Z

Any thoughts?

Thanks!

[liyongxian]

@ieugen please close it.I did it by changing k3s code and rebuild it. Thanks.

[ieugen]

@liyongxian I can't close it since I did not start the thread.
You should be able to close it or someone from the project.
It would be nice if you can share a link to the steps you used to solve it.

[ShylajaDevadiga]

Certificates are rotated if they are < 90 days from expiration.

1. Install k3s (v1.17.4+k3s1)
2. Check certificate expiration date
3. Stop time sync
4. Update date to <90 days from expiration
5. Restart k3s
6. Verify certificates are rotated.

Before expiration:

client-admin.crt
notAfter=Jul 13 19:22:39 2021 GMT
client-auth-proxy.crt
notAfter=Jul 13 19:22:39 2021 GMT
client-ca.crt
notAfter=Jul 11 19:22:39 2030 GMT
client-cloud-controller.crt
notAfter=Jul 13 19:22:39 2021 GMT
client-controller.crt
notAfter=Jul 13 19:22:39 2021 GMT
client-k3s-controller.crt
notAfter=Jul 13 19:22:39 2021 GMT
client-kube-apiserver.crt
notAfter=Jul 13 19:22:39 2021 GMT
client-kube-proxy.crt
notAfter=Jul 13 19:22:39 2021 GMT
client-scheduler.crt
notAfter=Jul 13 19:22:39 2021 GMT
request-header-ca.crt
notAfter=Jul 11 19:22:39 2030 GMT
server-ca.crt
notAfter=Jul 11 19:22:39 2030 GMT
serving-kube-apiserver.crt
notAfter=Jul 13 19:22:39 2021 GMT

Update date

date -s 20210514

Confirm certs are rotated

client-admin.crt
notAfter=May 14 00:00:14 2022 GMT
client-auth-proxy.crt
notAfter=May 14 00:00:14 2022 GMT
client-ca.crt
notAfter=Jul 11 19:22:39 2030 GMT
client-cloud-controller.crt
notAfter=May 14 00:00:14 2022 GMT
client-controller.crt
notAfter=May 14 00:00:14 2022 GMT
client-k3s-controller.crt
notAfter=May 14 00:00:14 2022 GMT
client-kube-apiserver.crt
notAfter=May 14 00:00:14 2022 GMT
client-kube-proxy.crt
notAfter=May 14 00:00:14 2022 GMT
client-scheduler.crt
notAfter=May 14 00:00:14 2022 GMT
request-header-ca.crt
notAfter=Jul 11 19:22:39 2030 GMT
server-ca.crt
notAfter=Jul 11 19:22:39 2030 GMT
serving-kube-apiserver.crt
notAfter=May 14 00:00:14 2022 GMT

[ShylajaDevadiga]

Further testing: If the system date is set to after exipiration date, rotation works but certificate are not updated. Since the certificates being rotated by default within <90 of expiration should solve the issue.

Suppose the expiration date is

notAfter=Jul 13 21:29:36 2021 GMT

and set the date to < 90 days to expire.

date -s 20210514
systemctl restart k3s

Certs are rotated and updated

-> notAfter=May 14 00:00:19 2022 GMT

kubectl get nodes
NAME              STATUS   ROLES    AGE    VERSION
ip-172-31-2-249   Ready    master   304d   v1.17.4+k3s1

Suppose date is set to after expire

date -s 20220516

Certificates are rotated but not updated

kubectl get nodes
Unable to connect to the server: x509: certificate has expired or is not yet valid

systemctl restart k3s

-> notAfter=May 16 00:00:38 2023 GMT

kubectl get nodes
Unable to connect to the server: x509: certificate has expired or is not yet valid

[ShylajaDevadiga]

Verified using K3s commit a2471a1. k3s-serving cert rotation succeeded.
Before rotation

root@ip-172-31-10-200:/var/lib/rancher/k3s/server/tls# for i in `ls *.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
client-admin.crt
notAfter=Aug 20 08:32:18 2021 GMT
client-auth-proxy.crt
notAfter=Aug 20 08:32:18 2021 GMT
client-ca.crt
notAfter=Aug 18 08:32:18 2030 GMT
client-cloud-controller.crt
notAfter=Aug 20 08:32:18 2021 GMT
client-controller.crt
notAfter=Aug 20 08:32:18 2021 GMT
client-k3s-controller.crt
notAfter=Aug 20 08:32:18 2021 GMT
client-kube-apiserver.crt
notAfter=Aug 20 08:32:18 2021 GMT
client-kube-proxy.crt
notAfter=Aug 20 08:32:18 2021 GMT
client-scheduler.crt
notAfter=Aug 20 08:32:18 2021 GMT
request-header-ca.crt
notAfter=Aug 18 08:32:18 2030 GMT
server-ca.crt
notAfter=Aug 18 08:32:18 2030 GMT
serving-kube-apiserver.crt
notAfter=Aug 20 08:32:18 2021 GMT
root@ip-172-31-10-200:/var/lib/rancher/k3s/server/tls# kubectl get secret -n kube-system k3s-serving -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | grep Not
            Not Before: Aug 20 08:32:18 2020 GMT
            Not After : Aug 20 08:32:18 2021 GMT

Certs are rotated when date is manually set to <90 to expiration

date -s 20210710
root@ip-172-31-10-200:/var/lib/rancher/k3s/server/tls# for i in `ls *.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
client-admin.crt
notAfter=Jul 10 00:00:10 2022 GMT
client-auth-proxy.crt
notAfter=Jul 10 00:00:10 2022 GMT
client-ca.crt
notAfter=Aug 18 08:32:18 2030 GMT
client-cloud-controller.crt
notAfter=Jul 10 00:00:10 2022 GMT
client-controller.crt
notAfter=Jul 10 00:00:10 2022 GMT
client-k3s-controller.crt
notAfter=Jul 10 00:00:10 2022 GMT
client-kube-apiserver.crt
notAfter=Jul 10 00:00:10 2022 GMT
client-kube-proxy.crt
notAfter=Jul 10 00:00:10 2022 GMT
client-scheduler.crt
notAfter=Jul 10 00:00:10 2022 GMT
request-header-ca.crt
notAfter=Aug 18 08:32:18 2030 GMT
server-ca.crt
notAfter=Aug 18 08:32:18 2030 GMT
serving-kube-apiserver.crt
notAfter=Jul 10 00:00:10 2022 GMT
root@ip-172-31-10-200:/var/lib/rancher/k3s/server/tls# kubectl get secret -n kube-system k3s-serving -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | grep Not
            Not Before: Aug 20 08:32:18 2020 GMT
            Not After : Jul 10 00:00:40 2022 GMT

[thetruechar]

same issue here.

systemctl restart k3s
root@chirpersite-k3s-ecs:~/.kube# kubectl get pods     
Unable to connect to the server: x509: certificate has expired or is not yet valid

[recipedude]

root@k3s1:~# systemctl restart k3s
root@k3s1:~# k3s kubectl get pods
Unable to connect to the server: x509: certificate signed by unknown authority

similar issue here

[brandond]

Anyone experiencing this issue should create a new issue and fill out the issue template with all the relevant information. Adding me-too comments on a closed issue makes it very difficult to track whether you're experiencing a regression, or are perhaps just not using a release where this is fixed.