Injected Agent sidecar is never updated

[lorenzo-biava]

Describe the bug

The injected Agent sidecar is never updated, even if the container image changes.

To Reproduce
Steps to reproduce the behavior:

1. Setup the Jaeger CR to with Agent image = A
2. Create workload with injection annotation enabled and wait for it to get the Agent sidecar
3. Change the Jaeger CR with Agent image = B
4. Observe all Deployments/Pods will still use image A (restarting the Operator has no effect).

Expected behavior
The injected Agent image should be updated (as well as other changed properties pertaining to the Agent).

Screenshots
If applicable, add screenshots to help explain your problem.

Version (please complete the following information):

• Jaeger version: jaeger-operator:1.22.1
• Deployment: Kubernetes

What troubleshooting steps did you try?
Try to follow https://www.jaegertracing.io/docs/latest/troubleshooting/ and describe how far you were able to progress and/or which steps did not work.

Additional context
Looks like the reconciliation loop only checks whether the sidecar container exists, but doesn't check if the settings are the same as the ones in the Jaeger CR.
See:

• jaeger-operator/pkg/inject/sidecar.go

  Line 375 in e0c53a5

  func HasJaegerAgent(dep *appsv1.Deployment) (bool, int) {

• jaeger-operator/pkg/inject/sidecar.go

  Line 113 in e0c53a5

  if hasAgent {

[rubenvp8510]

@frzifus Do you think the new approach of using webhooks will fix this?

[frzifus]

yes, every change should lead to an reevaluation.

[AshutoshNirkhe]

yes, every change should lead to an reevaluation.

@frzifus we have existing user workloads (deployments) that are currently running with jaeger-agent sidecar (with v1.28).
We did upgrade the jaeger operator and jaeger custom resource to v1.34.1. But off course, that didn't trigger user workloads to be updated with new configs added for jaeger sidecar because of this issue.

Then I verified it worked fine for jaeger-query (that also has jaeger-agent as sidecar and was already running at v1.34.1), as it is showing all latest image/configs.

So do we need to ask users to update their jaeger sidecar agent image manually to v1.34 first and then it will trigger the webhook or we need to aks them to totally remove the auto-injection annotation and add it back to force the new jaeger updates (which is off course more disruptive) ?