connect: deployments should wait for Connect sidecar checks #19334
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgross
added
theme/consul
type/bug
theme/consul/connect
When a Connect service is registered with Consul, Nomad includes the nested `Connect.SidecarService` field that includes health checks for the Envoy proxy. Because these are not part of the job spec, the alloc health tracker created by `health_hook` doesn't know to read the value of these checks. In many circumstances this won't be noticed, but if the Envoy health check happens to take longer than the `update.min_healthy_time` (perhaps because it's been set low), it's possible for a deployment to progress too early such that there will briefly be no healthy instances of the service available in Consul. Update the Consul service client to find the nested sidecar service in the service catalog and attach it to the results provided to the tracker. The tracker can then check the sidecar health checks. Fixes: #19269
tgross
force-pushed
the
b-gh19269-sidecar-checks
branch
from
f2810d9 to
d9c9dc5
Compare
This was referenced Dec 6, 2023
gulducat
approved these changes
Dec 6, 2023
|
I'll add the backport labels on this once 1.7.0 GA goes out. |
tgross
added
backport/1.5.x
This was referenced Dec 7, 2023
Merged
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a Connect service is registered with Consul, Nomad includes the nested
Connect.SidecarServicefield that includes health checks for the Envoy proxy. Because these are not part of the job spec, the alloc health tracker created byhealth_hookdoesn't know to read the value of these checks.In many circumstances this won't be noticed, but if the Envoy health check happens to take longer than the
update.min_healthy_time(perhaps because it's been set low), it's possible for a deployment to progress too early such that there will briefly be no healthy instances of the service available in Consul.Update the Consul service client to find the nested sidecar service in the service catalog and attach it to the results provided to the tracker. The tracker can then check the sidecar health checks.
Fixes: #19269