Jaeger Agent - collector.host value from configmap not updating dynamically

[raz08]

I have a configmap and I'm passing these configmap values as ENV variables in Jaeger agent daemon set deployment file. But when i update collector.host in configmap , it is not getting reflected in Jaeger agent.

Configmap

apiVersion: v1
data:
  admin-http-port: "14271"
  collector.host-port: 1.1.1.1:14267
  metrics-backend: prometheus
  metrics-http-route: /metrics

Daemonset

      containers:
      - name: {{ template "jaeger.agent.name" . }}
        env:
        - name: COLLECTOR_HOST_PORT
          valueFrom:
            configMapKeyRef:
              name: {{ template "jaeger.agent.name" . }}
              key: collector.host-port
        - name: METRICS_BACKEND
          valueFrom:
            configMapKeyRef:
              name: {{ template "jaeger.agent.name" . }}
              key: metrics-backend
        - name: METRCIS_HTTP_ROUTE
          valueFrom:
            configMapKeyRef:
              name: {{ template "jaeger.agent.name" . }}
              key: metrics-http-route
        - name: ADMIN_HTTP_PORT
          valueFrom:
            configMapKeyRef:
              name: {{ template "jaeger.agent.name" . }}
              key: admin-http-port

How to make this works dynamically? If I update configmap it should load/reload configuration and start connecting to updated collector.host . Help is appreciated.

[Ashmita152]

I think what you're asking is a k8s limitation: kubernetes/kubernetes#22368

You need to trigger the deployment of jaeger-agent again once you update the configmap. You can find few workarounds mentioned in the above thread.

[jpkrohling]

In the v2 of the operator, we should be able to perform an automatic rollout whenever the configmap is updated. See open-telemetry/opentelemetry-operator#197

cc @rubenvp8510

[raz08]

Thanks @jpkrohling , @Ashmita152

Is there a way we can update the config without pod restart or rolling update. Lets say in Prometheus we have /-/reload API to reload the configuration which is mounted through configmap.

But here I see only way is to restart the pod because we setting config through ENVIROMENT variable. Is my understanding correct?

[jpkrohling]

Jaeger doesn't actually have a hot reload functionality for its main config file/options, so, even if we had hot-reload of env vars at Kubernetes level, it wouldn't help us that much. But any reasons to avoid a rollout? Given that Jaeger is quite lightweight, I don't see any disadvantages in that. Quite the opposite: if the new config has an error, the rollout would not kill the old pod with the working config.

[raz08]

Yes agree jpkrohlin,

Does Jaeger has plans to support hot reload functionality? (i.e. config reload API).

And what is the plan in Jaeger operator? How are we solving this issue ? Is it with stakater/reloader or have something inhouse similar to that? And when can we expect this to be released in Jaeger operator?

[jpkrohling]

Does Jaeger has plans to support hot reload functionality? (i.e. config reload API).

No, nor even for v2, I'm afraid.

And what is the plan in Jaeger operator? How are we solving this issue ? Is it with stakater/reloader or have something inhouse similar to that? And when can we expect this to be released in Jaeger operator?

Jaeger v2 will be based on OpenTelemetry Collector, and the Operator v2 will be based on the OpenTelemetry Operator, which has recently added support for issuing a new rollout when the config map backing the instance changes: open-telemetry/opentelemetry-operator#197

[jkowall]

Should be fixed in Jaeger v2 when we have moved to otel, feel free to re-open if not.