unreachable pods by nodeport via worker nodes (timeout)

[reisman234]

Hello,

Today, I faced a strange error. After I have done some changes on a running deployment and its nodeport service.

1. removing the service
2. scaling the deployment down to 0
3. scaling up and creating service again

However, every request ended with a timeout despite the pod running. Also a port-forwart into the pod showed that the service was responding.
With a test deployment (nginx) and a nodeport service, I found out that every request against the worker-nodes:ip and nodeport ends with a timed out error, only the master node ip does respond. In the logs of each worker node, I found the following error message.

Mai 09 21:43:26 node-4 rke2[152187]: E0509 21:43:26.055113  152187 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Unauthorized

For now, a restart of the rke service solved the problem, but how to prevent such an error?

[brandond]

The error message suggests that you're on Rke2 1.22, which is end of life. Are you able to reproduce this error on more recent versions?

[reisman234]

Thanks for your response!

At this time, I didn't take care about updating the rke2 cluster, but I will catch up.

In the Upgrade Overview, it is suggested to use terraform as IaC to do the upgrade.

• Is there an example recipe how this could look like?

My current setup is a bare metal cluster with 4 nodes which of one is a GPU node, so it has an extra configuration

• Do I have to take care of this during the upgrade, or should this no problem?