Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pod recreation due to container.name tag missing in the jaeger.tags argument #1318

Closed
lujiajing1126 opened this issue Nov 27, 2020 · 1 comment · Fixed by #1319
Closed

Pod recreation due to container.name tag missing in the jaeger.tags argument #1318

lujiajing1126 opened this issue Nov 27, 2020 · 1 comment · Fixed by #1319
Labels
bug Something isn't working

Comments

@lujiajing1126
Copy link
Contributor

We have observed Pod recreation events in the production cluster very similar to this issue #1089

We find the following logs in the operator container,

...
time="2020-11-27T01:24:15Z" level=info msg="Injecting Jaeger Agent sidecar" deployment=hera-alertgateway-ack-deploy jaeger=jaeger-ack jaeger-namespace=observability namespace=sqb
time="2020-11-27T01:24:15Z" level=info msg="Injecting Jaeger Agent sidecar" deployment=kubernetes-ack-jack-ack-deploy jaeger=jaeger-ack jaeger-namespace=observability namespace=sqb
time="2020-11-27T01:24:15Z" level=info msg="Injecting Jaeger Agent sidecar" deployment=merchant-dmp-datacenter-ack-deploy jaeger=jaeger-ack jaeger-namespace=observability namespace=sqb
time="2020-11-27T01:24:15Z" level=info msg="Injecting Jaeger Agent sidecar" deployment=merchant-dmp-service-ack-deploy jaeger=jaeger-ack jaeger-namespace=observability namespace=sqb

These are the four deployments annotated with sidecar.jaegertracing.io/inject: 'true', and these event repeatedly occur recently. In the meantime, we find the following events from kubernetes, the times exactly coincide.

image

Thus, we compared two replicaSets generated and found the arguments of two agent containers are different,

$ diff merchant-dmp-datacenter-ack-deploy-6d5dc79477 merchant-dmp-datacenter-ack-deploy-68b5cc4bd6
...(not related)
120c120
<         - --jaeger.tags=cluster=undefined,deployment.name=merchant-dmp-datacenter-ack-deploy,pod.namespace=sqb,pod.name=${POD_NAME:},host.ip=${HOST_IP:},container.name=merchant-dmp-datacenter-ack
---
>         - --jaeger.tags=cluster=undefined,deployment.name=merchant-dmp-datacenter-ack-deploy,pod.namespace=sqb,pod.name=${POD_NAME:},host.ip=${HOST_IP:}

I suppose it is related to

if len(dep.Spec.Template.Spec.Containers) == 1 {
agentTags = fmt.Sprintf("%s,%s=%s", agentTags,
"container.name", dep.Spec.Template.Spec.Containers[0].Name,
)
}
since the agent container has already been injected.

@lujiajing1126 lujiajing1126 changed the title Pod recreation due to container.name tag is missing in the jaeger.tags argument Pod recreation due to container.name tag missing in the jaeger.tags argument Nov 27, 2020
@github-actions github-actions bot added the needs-triage New issues, in need of classification label Nov 27, 2020
@jpkrohling jpkrohling added bug Something isn't working and removed needs-triage New issues, in need of classification labels Nov 27, 2020
@jpkrohling
Copy link
Contributor

Thank you for finding the root cause and for the PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants