Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Service with Connect sidecar de-registered without removing sidecar first #10845

Closed
shoenig opened this issue Jul 2, 2021 · 1 comment · Fixed by #10873
Closed

Service with Connect sidecar de-registered without removing sidecar first #10845

shoenig opened this issue Jul 2, 2021 · 1 comment · Fixed by #10873
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/consul/connect Consul Connect integration type/bug

Comments

@shoenig
Copy link
Member

shoenig commented Jul 2, 2021

Nomad implicitly registers Connect sidecar services via the connect.sidecar_service stanza, and does not keep track of the sidecar service or its checks internally. When it's time to shut the task down, Nomad only de-registers the parent service (because that's all it knows about). This causes an ERROR and WARN in Consul logs - it's the root cause of hashicorp/consul#8657

Nomad needs to explicitly de-register the sidecar service before de-registering the parent service to avoid making Consul unhappy.

2021-07-02T10:14:55.473-0500 [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service '_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001' not in agent state" index=0
2021-07-02T10:14:55.473-0500 [ERROR] agent.proxycfg: Failed to handle update from watch: service_id=_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001-sidecar-proxy id=service-http-checks:_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001 error="error filling agent cache: Internal cache failure: service '_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001' not in agent state"
2021-07-02T10:14:55.474-0500 [INFO]  agent: Deregistered service: service=_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001
2021-07-02T10:14:55.474-0500 [WARN]  agent.cache: handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service '_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001' not in agent state" index=0
2021-07-02T10:14:55.474-0500 [ERROR] agent.proxycfg: Failed to handle update from watch: service_id=_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001-sidecar-proxy id=service-http-checks:_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001 error="error filling agent cache: Internal cache failure: service '_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001' not in agent state"
2021-07-02T10:14:55.474-0500 [INFO]  agent: Deregistered service: service=_nomad-task-4903435d-3694-9ebb-21ca-378815a1d366-group-api-count-api-9001-sidecar-proxy
@shoenig shoenig added type/bug theme/consul/connect Consul Connect integration stage/accepted Confirmed, and intend to work on. No timeline committment though. labels Jul 2, 2021
@shoenig shoenig mentioned this issue Jul 8, 2021
Merged
shoenig added a commit that referenced this issue Jul 8, 2021
@shoenig
consul/connect: Avoid assumption of parent service when filtering con…
86283ad 
…nect proxies

This PR uses regex-based matching for sidecar proxy services and checks when syncing
with Consul. Previously we would check if the parent of the sidecar was still being
tracked in Nomad. This is a false invariant - one which we must not depend when we
make #10845 work.

Fixes #10843
@shoenig shoenig mentioned this issue Jul 8, 2021
Merged
shoenig added a commit that referenced this issue Jul 8, 2021
@shoenig
consul/connect: remove sidecar proxy before removing parent service
8ad8e10 
This PR will have Nomad de-register a sidecar proxy service before
attempting to de-register the parent service. Otherwise, Consul will
emit a warning and an error.

Fixes #10845
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/consul/connect Consul Connect integration type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

consul/connect: remove sidecar proxy before removing parent service hashicorp/nomad
1 participant
@shoenig