Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/contour: Retry Envoy healthcheck fail #2262

Closed
stevesloka opened this issue Feb 20, 2020 · 0 comments · Fixed by #3136
Closed

cmd/contour: Retry Envoy healthcheck fail #2262

stevesloka opened this issue Feb 20, 2020 · 0 comments · Fixed by #3136
Assignees
@stevesloka

Comments

@stevesloka
Copy link
Member

In the Envoy ShutdownManager, the first thing that happens is we tell Envoy to fail its healthchecks which causes the container to fail its readiness probe and start to drain connections.

It's possible that the initial call will fail and Envoy never starts to drain connections. If this occurs then the pod will wait it's total grace period seconds before terminating.

A best-effort could be applied to retry the healthcheck fail call if it doesn't succeed on the first attempt.

Originally posted by @jpeach in #2227 (comment)
@stevesloka stevesloka mentioned this issue Feb 20, 2020
Merged
@stevesloka stevesloka self-assigned this Nov 17, 2020
stevesloka added a commit to stevesloka/contour that referenced this issue Nov 17, 2020
@stevesloka
cmd/contour: Enable retry for shutdown-manager initiating Envoy healt…
fbe9deb 
…hcheck fail

The call in the shutdown-manager to intitiate the Envoy healthcheck fail HTTP request
did not retry in the event that the call fails. Adds the client-go retry.OnError method
which takes a backoff object allowing for a retry with backoff in the event the request
fails.

Fixes projectcontour#2262

Signed-off-by: Steve Sloka <[email protected]>
@stevesloka stevesloka mentioned this issue Nov 17, 2020
Merged
stevesloka added a commit that referenced this issue Dec 1, 2020
@stevesloka
cmd/contour: Enable retry for shutdown-manager initiating Envoy healt…
14f988e 
…hcheck fail (#3136)

The call in the shutdown-manager to intitiate the Envoy healthcheck fail HTTP request
did not retry in the event that the call fails. Adds the client-go retry.OnError method
which takes a backoff object allowing for a retry with backoff in the event the request
fails.

Fixes #2262

Signed-off-by: Steve Sloka <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stevesloka stevesloka

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cmd/contour: Enable retry for shutdown-manager initiating Envoy healthcheck fail stevesloka/contour
1 participant
@stevesloka