ready: Use envoy listener to expose endpoint from worker for ambassador

[jfrabaute]

/ready endpoint used by emissary is using the admin port (8001 by
default).
This generates a problem during config reloads with large configs as the
admin thread is blocking so the /ready endpoint can be very slow to
answer (in the order of several seconds, even more).

The problem is described in this envoy issue:
envoyproxy/envoy#16425

This change is fixing the slowness of /ready endpoint.
The /ready endpoint is now exposed in the worker pool by adding a
listener+ health check http filter.
This way, the /ready endpoint is fast and it is not blocked by any
config reload or blocking admin operation as it depends on the worker
pool.

Related Issues

envoyproxy/envoy#16425

Testing

Manual tests
Deployed on a local k8s cluster.

[alexgervais]

Thanks for your contribution @jfrabaute!
I'm a bit curious about your testing strategy for this PR. How did you go about testing large configuration reloads? Could we automate any part of your manual tests?

[alexgervais]

Should the fallback ready port be 8001?
I see multiple occurrences of 8002 in the python code as well, yet the previously hardcoded ready-url was "http://localhost:8001/ready".

[jfrabaute]

I'm adding a new listener here, where the default port is 8002.
This is this port that I'm using as "default" port now for the readiness.
This new listener is just exposing the "/ready" endpoint with the special ready filter.
The default is 8002, and it can be changed to another value, in case port 8002 is already used by some customer config (to expose a TCPMapping for instance).

Now the new url for the readiness is "http://127.0.0.1:8002/ready".
If the user changes this port using the env var AMBASSADOR_READY_PORT, the value can be different.

I hope it's clear now.

[alexgervais]

Thanks for the clarification, I understand much better now.
I know a lot of installations and monitors are configured to ping :8001/ready for uptime and health check of course. I wonder how we can make a smooth transition to the new port... any thoughts @LanceEa ?

[jfrabaute]

:8001/ready endpoint is still working. This is an endpoint automatically exposed by the envoy admin thread, so installs using this endpoint should still work.

[LanceEa]

Thanks @jfrabaute for the PR. I will take a look this week and let you know if I have any questions.

I would also echo what @alexgervais mentioned around testing and how you tested the large configuration and whether we could add it to our test suite to validate the previous vs new behavior.

[jfrabaute]

Hi,

For testing, I did test on the previous PR: #3626
This PR is the exact same except that it also it also impacts the go process to check the readiness on this new endpoint rather then the old endpoint (which is slow).

In order to test manually, you can start an ambassador instance on a cluster(or a kind cluster for instance), create 1000 mappings.
When don, you can hit the stats endpoint on envoy (something like http://localhost:8001/stats?format=prometheus) and while running this hit, you can query http://localhost:8001/ready right after the stats endpoint has started(but not finished) yet.
You'll see that stats prometheus will take a while, like several seconds, and in the meantime, /ready endpoint will also hang.
Then when the stats endpoint ends, the ready endpoint will exit. So the ready endpoint execution on the envoy side is blocked by the stats endpoint (the admin thread is single threaded).
When moving the ready endpoint to a worker listener, the hit is always fast, just a efw milliseconds, regardless if a stats endpoint execution is in progress or not.

[jfrabaute]

Hi,

Any chance to have this PR reviewed and merged? or get feedback about what's missing for a merge?

Thanks.

[jfrabaute]

Superseded by: #4300