[Bug]: NoData for Latency and ErrorRate in Monitoring tab with Jaeger 1.46 and Opentelemetry SpanMetrics Connector

[Tyrion85]

What happened?

This issue added support for SpanMetrics connector (since SpanMetrics Processor is deprecated)

However, I'm not seeing latency and error rate in Jaeger's Monitoring tab - only Request rate is populated, and a "No data yet!" notification is displayed in UI.

Relevant metrics are written to Prometheus by OpenTelemetry Collector, as shown in the screenshots below. Jaeger metrics jaeger_query_requests_total{operation="get_latencies",result="ok"} and similar suggest everything is okay. There is also no error visible in Query logs.

Jaeger version 1.46.0, installed via helm chart (not a all-in-one)

Steps to reproduce

1. Install Jaeger via helm chart, image version 1.46.0
2. Setup OpenTelemetry collector with config from upstream
3. Connect Jaeger with Prometheus (in my case, Thanos, but it doesn't matter)
4. Inspect that Latency and Error Rate are missing

Expected behavior

Expected Latency and Error Rate to work, as related metrics are available in Prometheus

Relevant log output

No response

Screenshot

Additional context

Relevant OpenTelemetry Collector configuration:

receivers:
      otlp:
        protocols:
          grpc:
          http:
          ....
      zipkin:
      opencensus:
      jaeger:
        protocols:
          grpc:
          thrift_http:
          thrift_compact:
          thrift_binary:
    connectors:
      spanmetrics:
    processors:
      batch:
      span/to_attributes:
        ...
    extensions:
      health_check:
    exporters:
      jaeger:
        ....
      otlp/quickwit:
        ....
      prometheusremotewrite:
        endpoint: http://prometheus-operated.monitoring:9090/api/v1/write
        target_info:
          enabled: true
    service:
      extensions: [ health_check ]
      pipelines:
        traces:
          receivers: [ otlp,opencensus,zipkin,jaeger ]
          processors: [ batch,span/to_attributes ]
          exporters: [ jaeger,otlp/quickwit,spanmetrics ]
        metrics:
          receivers: [ spanmetrics ]
          exporters: [ prometheusremotewrite ]

Relevant Jaeger config (helm chart):

....
extraEnv:
    - name: GRPC_STORAGE_SERVER
      value: quickwit-searcher:7281
    - name: METRICS_STORAGE_TYPE
      value: prometheus
    - name: PROMETHEUS_SERVER_URL
      value: http://thanos-query-frontend.monitoring:9090

Jaeger backend version

v1.46.0

SDK

No response

Pipeline

No response

Stogage backend

GRPC_STORAGE_SERVER (Quickwit)

Operating system

No response

Deployment model

Kubernetes, helm chart

Deployment configs

No response

[Tyrion85]

--prometheus.query.support-spanmetrics-connector=true command line argument was missing from the config. This is not documented in Jaeger setup docs (it is in Migration docs, but you'd have to be aware that some kind of migration was going on). https://github.com/jaegertracing/jaeger/blob/main/docker-compose/monitor/README.md#migrating

Closing this issue & leaving the comment if someone else stumbles on the same

[yurishkuro]

command line argument was missing from the config. This is not documented in Jaeger setup docs

Which docs are you referring to, the README in the repo or the website?

Re-opening to track a need to update the docs.

[Tyrion85]

This one https://www.jaegertracing.io/docs/1.46/spm/

[Tyrion85]

Should probably be somewhere here:

[yurishkuro]

Thanks! Added to both places.

[hdong69]

Anyone can confirm that the Jaeger 1.46 works well with the tutorial https://github.com/jaegertracing/jaeger/blob/main/docker-compose/monitor/README.md#migrating ?

[Tyrion85]

It works well for me, but I was not migrating from OTEL SpanMetrics Processor config, I went straight into SpanMetrics Connector. Configuration was pretty simple except for the missing flag prometheus.query.support-spanmetrics-connector from the main docs page, which is now amended 👍🏼

[hdong69]

@Tyrion85 thanks for your feedback