Problems with integration with opentelemetry collector

[Ventura94]

Opentelemetry logs:

otelcol-contrib[33599]: 2024-09-04T18:05:49.789Z        error        exporterhelper/queue_sender.go:92        Exporting failed. Dropping data.        {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService", "dropped_items": 1}

I have the tempo services and the opentelemetry collector on the same local server with these configurations:

Opentelemetry:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:55680
      http:
        endpoint: 0.0.0.0:55681
exporters:
  prometheus:
    endpoint: "0.0.0.0:3300"

  logging:
  
  otlp:
    endpoint: "localhost:9095"
    tls:
      insecure: true

  loki:
    endpoint: http://localhost:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp, logging]

    metrics:
      receivers: [otlp]
      exporters: [prometheus, logging]

    logs:
      receivers: [otlp]
      exporters: [loki, logging]

Tempo:

stream_over_http_enabled: true
server:
  http_listen_port: 3200
  grpc_listen_port: 9095

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 5s

distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    otlp:
      protocols:
        http:
        grpc:
 
ingester:
  max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

compactor:
  compaction:
    block_retention: 1h                # overall Tempo trace retention. set for demo purposes

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces

storage:
  trace:
    backend: local                     # backend configuration to use
    wal:
      path: /var/tempo/wal             # where to store the wal locally
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks] # enables metrics generator
      generate_native_histograms: both

I don't understand what I'm doing wrong

[joe-elliott]

With this config Tempo will open trace ingestion on the standard otel ports:

distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    otlp:
      protocols:
        http:
        grpc:

For, GRPC that is 4318:

https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver#getting-started

This GRPC config:

server:
  grpc_listen_port: 9095

is for Tempo's internal GRPC communications and is unrelated to otel proto ingestion.

[polygbrl]

Hi joe-elliott!

My situation is very similar to OP's (only that Tempo and OpenTelemetry Collector are containerized). I've used 9095 to connect the OpenTelemetry Collector to Tempo because I had no success with 4317 nor 4318, but I realize 9095 won't work even if the connexion is accepted because of it being internal to Tempo as you mentioned here. This led to the "Not implemented" error, am I right?

Now, using 431x ports leads to a connection refused on the collector's side: transport: Error while dialing: dial tcp 192.20.0.3:4317: connect: connection refused. Do you have a lead to get me on the right track here? I wonder if it's a TLS issue, so I'm trying to set the receiver to accept insecure connexions but I'm a bit lost here, it looks like I can only specify insecure in a forwarder so that might not be the issue.

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

  forwarders:
    - name: otlp
      backend: otlpgrpc
      otlpgrpc:
        tls:
          insecure: true

Thank you!

[joe-elliott]

but I realize 9095 won't work even if the connexion is accepted because of it being internal to Tempo as you mentioned here. This led to the "Not implemented" error, am I right?

You are correct!

There was a change to behavior in Tempo 2.7 due to an upgrade of our OTel depedency. Now you need to specify the ips and ports you want the otel receivers to listen on:

https://grafana.com/docs/tempo/latest/release-notes/v2-7/#opentelemetry-collector-receiver-listens-on-localhost-by-default

[polygbrl]

Excellent, I failed to see this point in the 2.7 release notes, that's what got me fiddling around!