Envoy does not remove endpoint after it's been removed from the Kubernetes Endpoint object when health checking is enabled

[alexbrand]

When active health checking is enabled, Envoy does not seem to remove endpoints from the ClusterLoadAssignment when endpoints are removed from the Kubernetes Endpoints object.

There is an Envoy Cluster option that I suspect is related to this:

drain_connections_on_host_removal
(bool) If this cluster uses EDS or STRICT_DNS to configure its hosts, immediately drain connections from any hosts that are removed from service discovery.

This only affects behavior for hosts that are being actively health checked. If this flag is not set to true, Envoy will wait until the hosts fail active health checking before removing it from the cluster.

(https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/cds.proto)

xref: projectcontour/gimbal#208

Steps to reproduce:

1. Create a deployment:

k run kuard --image=gcr.io/kuar-demo/kuard-amd64:1 --expose --port 8080

2. Create an ingressroute:

apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: kuard
spec:
  routes:
  - match: /
    services:
    - healthCheck:
        healthyThresholdCount: 5
        intervalSeconds: 5
        path: /
        timeoutSeconds: 2
        unhealthyThresholdCount: 3
      name: kuard
      port: 8080
  virtualhost:
    fqdn: example.com

3. Verify you can reach kuard via the ingressroute

4. Edit the kuard pod, and remove the run: kuard label to remove this pod from the Endpoints object. This will trigger the creation of a new pod, and the Kubernetes Endpoints object will be updated to contain the address of the new pod.

Expected result: Envoy removes the other endpoint from the ClusterLoadAssignment

Actual result: Envoy maintains the old endpoint in the ClusterLoadAssignment, and sends traffic to both the old pod and the new one.

[alexbrand]

Spent time looking into this issue. I am able to reproduce this consistently. I looked into setting drain_connections_on_host_removal to true, but it doesn't seem to have an effect. Starting to wonder if I am thinking about this wrong and I am missing something.

[rosskukulinski]

Ok, so drain_connections_on_host_removal was added in Envoy 1.7. I think we should enable this flag once we update to Envoy 1.7 (which is currently scheduled for Contour 0.7).

[davecheney]

Seconded. Let's push through the rest of 0.6 so we can start the 0.7 cycle with a fresh version of envoy

…

On 14 August 2018 at 06:47, Ross Kukulinski ***@***.***> wrote: Ok, so drain_connections_on_host_removal was added in Envoy 1.7. I think we should enable this flag once we update to Envoy 1.7 (which is currently scheduled for Contour 0.7). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#603 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA5ji2kqlDLYHu5_uc-sU_MmfzZeWks5uQeXKgaJpZM4VuhRl> .

[rosskukulinski]

@alexbrand would you have some time to dig into adding the drain_connections_on_host_removal?