-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DWD not working as expected on some infrastructures where traffic to LB stays in-cluster (gets rerouted internally) #85
Comments
To provide a little more context: Dependency watchdog tries to check the API server via its extern domain name. Depending on the infrastructure, the corresponding kubernetes service has either a domain name (AWS) or an IP address (other infrastructures). In case the kubernetes service includes an IP address On AWS it looks like this:
On other infrastructures:
It is alway different from the internal dependency watchdog probe:
Depending on what is desired, this might or might not be sufficient. Loadbalancer issues can only be found on infrastructures with domain names in the kubernetes service (AWS). Istio issues should be visible everywhere. Problems in the shoot network, e.g. firewall rules, or between shoot and seed are never detectable in this approach. |
MOMAttendees: @ashwani2k, @ScheererJ, @unmarshall and @rishabh-11 We discussed the issue of kube-proxy path optimization in context of DWD and @ScheererJ has already left a comment above describing the problem that we see today. It seems that this issue has been there for a while now since kube-proxy path optimization is not a new feature. Possible solutions that we discussed:
Option-1 There are essentially two ways a pod running in the shoot can reach its KAPI:
Pros:
Cons:
Option-2 @ScheererJ suggested as an alternative to do something similar to what a BTP availability service does. Essentially try and have a probe running in another seed instead of a shoot. The issue is that this would not check the network path from the shoot to its control plane KAPI. However, even today (prior to knowing that the external probe does not work) - DWD was only checking the route from DWD -> LB -> Istio Ingress Gateway -> KAPI which is also not an accurate representation of the network path from the shoot. Since I do not have much information on what BTP availability service does exactly i will not mention the pros and cons. More details are sought. Option-3 Have a brand new component that is setup in a HA manner and runs possible in the This option was only mentioned briefly but not completely thought through. Overall it is clear that we currently do not have a fool proof way to know if the connectivity to KAPI from the shoot cluster is broken. Therefore we need to discuss and evaluate the next best thing considering what we have had till now was also far from perfect. @ScheererJ, @ashwani2k, @rishabh-11 - feel free to add anything that i might have missed to capture or make corrections if something has been stated incorrectly or inaccurately. |
MOMAttendees: @ashwani2k, @elankath, @rishabh-11, @unmarshall All were briefed about the problem and pros and cons were discussed for all the options discussed and documented above. Conclusion:
We concluded that we will look further into checking if kube-proxy path optimisations can be applied conditionally. |
/assign @elankath |
And if not, we can accept @ScheererJ summary:
At least Istio issues are detected and that was a, if not the main motivation we had/have. |
If there is no simple way to avoid path optimisations and we do not want to accept the above, I actually like option 3 the most (new component in shoot, leases checked by DWD) as it most closely mimics what also the |
With respect to option 3 there was also an enhancement KEP for that purpose: kubernetes/enhancements#589
Example:
|
As discussed during our meeting, we considering using node leases but the actor (kubelet) that updates the node lease is not very reliable. Kubelet can go down with a lot of other reasons. We wanted a very light weight component which registers a lease. Semantically it is similar to having a node lease and is in-fact a duplicate object per node that we propose to create. If we wish to revisit that decision then we can. |
Is dependency watchdog's purpose not to prevent actions, e.g. node deletion, that are based on exactly this? Nodes are marked as |
We finalised on leveraging network-problem-detector as it already has functionality that probes using both pod and host network and we could enhance it by adding a Job (NWPD terminology, not a K8S Job) that periodically renews a lease. DWD listens for Node events and upon node registration it creates a However we learnt today that NWPD is deployed as an extension and that extensions can be disabled. Live issue#3891 suggests that one customer already did that. This makes it very hard to use NWPD as a component as for any cluster if this extension is disabled then renewals will never happen. This will be a false positive for DWD which will then scale down MCM, CA and KCM deployments to 0. I checked with @timuthy and he confirmed that today there is no way to enforce an extension. |
As discussed with @unmarshall and @ScheererJ, even if there was an option to enforce NWPD to be enabled for all shoots, we'd had a unwanted dependency from a Gardener core feature (DWD) and an extension (NWPD) that is not guaranteed to be installed in a landscape and theoretically exchangeable. We mainly see two options at the moment:
I personally prefer the second option, even though with a certain overhead but a clear segregation. Maybe the NWPD can even piggy-back in the DWD probes, once implemented?! (cc @MartinWeindel, @vlerenc) |
MOM
Conclusion: |
With gardener/gardener#8023, there is also the option to add a new controller to |
The Gardener project currently lacks enough active contributors to adequately respond to all issues.
You can:
/lifecycle stale |
/remove-lifecycle stale |
Reminder for a finding recently. Details missing here.
cc @unmarshall @ashwani2k
If memory serves, @ScheererJ and colleagues observed/noted that DWD will not actually reach out to/through the LB on all other infrastructures but AWS, which would make the test (API server reachable externally) not work as expected (reduced to an internal case, which cuts out problems like those for which we implemented the feature in the first place, e.g. a broken Istio ingress-gateway).
The text was updated successfully, but these errors were encountered: