Is there a more secure way to allow ALBs to health-check Envoy?

[kevincantu]

We have been happily using Contour+Envoy behind some NLBs for a while, now, but I recently spent some time exploring ALB usage: I'd like to set up some WAF ACLs. I've been able to get connections going after the hurdles of:

1. as for an NLB, setting up dnsPolicy: ClusterFirstWithHostNet and hostNetwork: true (and no k8s service for Envoy),
2. setting up a fallback certificate so HTTPS connections from the ALBs (which don't use SNI) can still work, and
3. exposing Envoy's port 8002 so I can have the ALB's make HTTP connections to /ready to health-check the nodes.

Now that works, but I'm not satisfied, because it would seem to expose a whole lot more things than /ready on that port! This would leave me one bad AWS Security Group rule away from giving Envoy's admin interface to the world.

Can you think of a safer way to expose Contour+Envoy for health-checking from ALBs?

Is there a way to define a default backend, so that when an ALB sends over a bogus (or just the IP) host header when health-checking, I can deploy a little toy app to respond 200 OK that has actually exercised Envoy?

[youngnick]

Envoy does not have a good way to expose only a health-check for the proxy itself at the moment, agreed.

As you suggest, you can fake this a bit by having a very simple service as a fallback and configure a default backend. Currently, the easiest way to do that is actually with the Ingress object, which allows you to specify an Ingress with no hostname match, and backends with no route match, making it the default.

We also have the option of a fallback certificate, for if you only wish to expose HTTPS, but your health checker doesn't support SNI (I have no idea if ALBs support this). See https://projectcontour.io/docs/v1.11.0/config/tls-termination/#fallback-certificate for details.

Sorry to not have a better answer for you right now, this is definitely an area which could use some further work.

[stevesloka]

@kevincantu so the listener 8002 is statically created by Contour to expose the /ready endpoint as well as the /stats endpoint in Envoy and that's it.

This static listener doesn't expose the other admin endpoints so should be good to use unless you don't want stats on the same endpoint.

[stevesloka]

The other benefit to using Envoy's built-in /ready endpoint is that it tells you when Envoy has started a draining process which is important to tell the downstream LB that Envoy is starting to shutdown and to stop sending new connections to it, where you wouldn't get that with just a dummy app sitting behind Envoy.

[kevincantu]

OK, that's great!

[youngnick]

Following up on old issues, @kevincantu, is there anything more on this one? Sorry to have led you astray, and thanks @stevesloka for the correction.

[kevincantu]

Well, I've changed companies and not quite yet talked the new team into migrating everything to Contour, so things are good on my end! My former team will probably be following up and can confirm when it all works!

[youngnick]

Thanks for that @kevincantu, I'll close this one for now then, we can reopen if needed.