Loadbalancing Exporter Not Forwarding Incoming Request Headers to Downstream Collector

[bvsvas]

Component(s)

exporter/loadbalancing

What happened?

Description

Experiencing an issue with the loadbalancing exporter in OpenTelemetry Collector where incoming request headers (HTTP/gRPC) are not being forwarded to the downstream(second) collector. This behavior is critical for our use case, where headers such as Authorization need to be preserved throughout the request flow.

Steps to Reproduce

1. Configure the OpenTelemetry Collector with the loadbalancing exporter (Configure two collectors, first one with loadbalancing exporter).
2. Send a request with specific headers (e.g., Authorization) to the collector.
3. Observe that the downstream collector does not receive the headers.

Expected Result

The loadbalancing exporter should forward all incoming request headers to the downstream collector by default.

Actual Result

The request headers are not being forwarded to the downstream collector.

Collector version

0.103.0

Environment information

Environment

OS: CentOS
Kubernetes: 1.28
Deployment: Kubernetes Cluster
Image: otel/opentelemetry-collector-contrib:0.103.0

OpenTelemetry Collector configuration

extensions:
      health_check:
        endpoint: 0.0.0.0:13133
        path: "/health"
    exporters:
      loadbalancing:
        protocol:
          otlp:
            timeout: 1s
            tls:
              insecure: true
        resolver:
          k8s:
            service: otelcol-ingester-service
            ports:
            - 4318
            - 4317
    processors:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    service:
      telemetry:
        logs:
          level: debug
      extensions: [health_check]
      pipelines:
        metrics:
          exporters:
            - loadbalancing
          processors: []
          receivers:
            - otlp
        traces:
          exporters:
            - loadbalancing
          processors: []
          receivers:
            - otlp

Log output

No response

Additional context

We have attempted using the headerssetter extension, but the issue persists. Any guidance on how to ensure headers are forwarded would be appreciated.

    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
        path: "/health"
      headers_setter:
        headers:
        - action: insert
          key: authorization
          from_context: authorization
    exporters:
      loadbalancing:
        routing_key: "service"
        protocol:
          otlp:
            timeout: 1s
            tls:
              insecure: true
        resolver:
          k8s:
            service: otelcol-ingester-service
            ports:
            - 4318
            - 4317
    processors:
      resource:
        attributes:
        - key: authorization
          from_context: authorization
          action: insert
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            include_metadata: true
          http:
            endpoint: 0.0.0.0:4318
            include_metadata: true
    service:
      telemetry:
        logs:
          level: debug
      extensions: [headers_setter]
      pipelines:
        metrics:
          exporters:
            - loadbalancing
          processors: 
            - resource
          receivers:
            - otlp
        traces:
          exporters:
            - loadbalancing
          processors:
            - resource
          receivers:
            - otlp

[crobert-1]

I believe you may need to configure include_metadata: true in your OTLP receiver configuration.

OTLP receiver config example here, reference in the header setter extension README here.

[bvsvas]

Thanks @crobert-1 for your response. Yes, I tried it when I used the headers_setter extension, but it didn't work with the loadbalancing exporter. Probably it didn't work as the loadbalancing exporter aggregates multiple incoming requests into a group and sends different outbound requests??

I have updated the LB Collector configuration with the include_metadata and headers_setter extension in the Additional Context section. Could you review it and suggest if anything needs to be updated?

[crobert-1]

include_metadata looks good to me. Is it working for you?

[bvsvas]

No.

[github-actions]

Pinging code owners for exporter/loadbalancing: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

I need to try this out, I'm not sure this works at the moment.

[bvsvas]

Thanks @jpkrohling for looking into it. Your insights here are greatly appreciated.

[bvsvas]

@jpkrohling - I wanted to follow up and see if you had a chance to review the issue. We would greatly appreciate it if you could prioritize it when possible.

[jpkrohling]

No, sorry, I have other things in my queue. Given that this seems important to you, it would help if you could go ahead and send a PR with a test case for this.

[MilosMoravac]

Any updates, because I have the same setup and this issue is persistent. Otel collector version 0.114.
@bvsvas Did you find any workaround?
@jpkrohling Could we please get some info on this.