Add annotation to allow use of service ClusterIP

[rajivm]

Equivalent to kubernetes/ingress-nginx#981

Added support for ingress.kubernetes.io/service-upstream Kubernetes ingress annotation that causes traefik to use ClusterIP as backend rather than endpoints API. Primary motivation is this allows for zero-downtime deployments as Kubernetes services respect readiness checks, which is lost when using the endpoints API directly.

From the doc update I wrote:

Use the service's clusterIP as the backend rather than resolve the pod IPs via the endpoints API and use them as individual backends. This will prevent sticky sessions from working and any specific loadbalancer algorithm as there will be only one backend for the service registered in traefik. The advantage is Kubernetes readiness checks will apply and rolling deployments to services can be graceful.

Use the service's clusterIP as the backend rather than resolve the pod IPs via the endpoints API and use them as individual backends. As this results in only a single backend, some features are lost:

• Prevents sticky session
• Specific load balancer algorithms
• Retries will not work (proxy_next_upstream directive will have no effect)

[dtomcej]

Retry may also break.

Wondering about this though:
The advantage is Kubernetes readiness checks will apply and rolling deployments to services can be zero-downtime.

Kubernetes only adds the endpoint to the service if the readiness checks pass. If you properly configure your readiness checks, traefik should have no issues with the rolling deployments.

[rajivm]

You are right, traefik is automatically reacting to the endpoints API changes based on readiness (unlike the nginx-ingress). I am doing further testing now because this version seemingly eliminated the brief 502s we were seeing during deploy. I am suspecting that maybe just the clusterIP kube-proxy event reaction timing is faster.

[aledbf]

@rajivm you forgot the most important part:

Known Issues

If the service-upstream annotation is specified the following things should be taken into consideration:

Sticky Sessions will not work as only round-robin load balancing is supported.
The proxy_next_upstream directive will not have any effect meaning on error the request will not be dispatched to another upstream.

so as @dtomcej said, retries are not possible.

[rajivm]

@aledbf did not miss that, wanted to update the docs once I verified whether there was any truth to the zero-downtime advantage claim.

[rajivm]

Updated with:

Use the service's clusterIP as the backend rather than resolve the pod IPs via the endpoints API and use them as individual backends. 
As this results in only a single backend, some features are lost:
 - Prevents sticky session
 - Specific load balancer algorithms
 - Retries will not work (`proxy_next_upstream` directive will have no effect)

[timoreimann]

Services do not guarantee you zero-downtime deployments or graceful termination automatically.

Please see this blog post (which was motivated by a discussion that took place at kubernetes-retired/contrib#1140) for details. In a nutshell, kube-proxy (the component powering Services under the hood) is just another controller watching the endpoints API for updates, which makes it susceptible just like any other controller (including Traefik's) to the effects of unpredictable event ordering inherent to the distributed nature of Kubernetes. Specifically, there's always a possibility for the scenario to happen in which a pod is first terminated by a SIGTERM before kube-proxy has had a chance to remove its IP address from rotation. Within this time window, requests sent through a Service would be lost and (in the case of kube-proxy's iptables mode, which is the norm these days) never retried.

The only strategy you have to execute deployments reliably is to orchestrate them properly.

[rajivm]

@timoreimann Yes, I realize this was not completely accurate. I removed the documentation text as a result. It would require making the backends live long enough while moved to the not-ready state. However, at least anecdotally, kube-proxy seems to process such events quicker than traefik today. I am looking into whether there are ways to make this more efficient in traefik's k8s event watch.

[timoreimann]

@rajivm it's possible that kube-proxy and Traefik diverge in update performance; either in favor of one or the other depending on a number of factors. If Traefik has any performance gaps in this regard, I certainly would be happy to see those getting closed.

I'm still not convinced of the general idea to route via Services though. As described above, there's no guarantee you will reach zero-downtime deployments. Even if it works for some clusters at some point in time, it may only take a load/memory shift inside your cluster to influence whether the SIGTERM or kube-proxy are going to win the race. In addition, it's also depended on the affected pods and how quickly they react to termination requests.

At best, this looks like a probabilistic mitigation to a problem that could be solved with significant better reliability using graceful termination handlers and/or pre-stop hooks. You usually want those anyway to cover all the other cases where Kubernetes needs to shut down pods gracefully (e.g., node draining, auto-scaling, pod evictions, taint updates, ConfigMap/Secrets changes).

[rajivm]

@timoreimann In full agreement at this point, if there is no demand for this annotation for other use cases, then we can certainly close this. I will continue to investigate ways to improve performance here.