[processor/attributes] unable to add span custom attribute from custom header #5610

amoscatelli · 2022-06-30T12:45:21Z

Describe the bug
By enabling include_metadata and using attributes processor I don't see http headers from the POST request on the otlp http endpoint added as span attributes.

Steps to reproduce
I have set include_metadata to true in my http otlp receiver
I try to upsert a span custom attribute using attributes processor just like in this readme:

https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md

In my case the header is CloudFront-Viewer-Longitude but I don't think that's relevant :

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert

Anyway below you'll find my full configuration.

What did you expect to see?
CloudFront-Viewer-Longitude added as http.client_longitude span attribute.

What did you see instead?
http.client_longitude is not added among span attributes.

What version did you use?
otel/opentelemetry-collector-contrib:0.54.0

What config did you use?

receivers:
    otlp:
        protocols:
            http:
                include_metadata: true
                cors:
                    allowed_origins: [
                        ${RECEIVERS_OTLP_PROTOCOLS_HTTP_CORS_ALLOWED_ORIGINS}
                    ]
                    
    # Dummy receiver that's never used, because a pipeline is required to have one.
    otlp/spanmetrics:
        protocols:
            grpc:
                endpoint: "localhost:12345"

    prometheus/spanmetrics:
      config:
        scrape_configs:
          - job_name: 'ratelimiter'
            scrape_interval: 6m
            static_configs:
              - targets: ['localhost:8889']

processors:
    attributes/traces:
        actions:
          - key: http.client_latitude_2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude_2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: test
            value: 123
            action: upsert
          - key: origin2
            from_context: metadata.origin
            action: upsert
          - key: client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: client_latitude2
            from_context: cloudfront-viewer-latitude
            action: upsert
          - key: client_longitude2
            from_context: cloudfront-viewer-longitude
            action: upsert
          - key: client_latitude3
            from_context: metadata.cloudfront-viewer-latitude
            action: upsert
          - key: client_longitude3
            from_context: metadata.cloudfront-viewer-longitude
            action: upsert
          - key: client_latitude4
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: client_longitude4
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
    memory_limiter:
        check_interval: 5s
        limit_mib: 448
        spike_limit_mib: 64
    batch:
        send_batch_size: 48
        send_batch_max_size: 48
        timeout: 15s
    transform:
        logs:
            queries:
                - set(attributes["severity_text"], severity_text)
    filter:
        logs:
            exclude:
                match_type: regexp
                record_attributes:
                    - key: severity_text
                      value: "(TRACE|DEBUG)"
    spanmetrics:
        metrics_exporter: prometheus/spanmetrics
        dimensions:
            - name: enduser.id
            - name: http.status_code
            - name: http.client_ip
            - name: http.client_latitude
            - name: http.client_longitude
            - name: http.method
        aggregation_temporality: "AGGREGATION_TEMPORALITY_DELTA"

exporters:
    logging:
        loglevel: ${EXPORTERS_LOGGING_LOGLEVEL}
    otlp:
        endpoint: tempo-eu-west-0.grafana.net:443
        headers:
            authorization: ##########
            
    prometheusremotewrite:
        endpoint: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
        headers:
            authorization: ##########
            
    loki:
        endpoint: https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push
        headers:
            authorization: ##########
        format: json
        labels:
            attributes:
                container_name: ""
                source: ""
            resource:
                host.name: "hostname"
                
    prometheus/spanmetrics:
        endpoint: "localhost:8889"

extensions:
    health_check:
service:
    telemetry:
        logs:
            level: ${SERVICE_TELEMETRY_LOGS_LEVEL}
    extensions: [health_check]
    pipelines:
        traces:
            receivers: [otlp]
            processors: [attributes/traces, spanmetrics, memory_limiter, batch]
            exporters: [otlp, logging]
        metrics/spanmetrics:
            # This receiver is just a dummy and never used.
            # Added to pass validation requiring at least one receiver in a pipeline.
            receivers: [otlp/spanmetrics]
            # The metrics_exporter must be present in this list.
            exporters: [prometheus/spanmetrics]
        metrics:
            receivers: [otlp, prometheus/spanmetrics]
            processors: [memory_limiter, batch]
            exporters: [prometheusremotewrite]
        logs:
            receivers: [otlp]
            processors: [transform, filter, memory_limiter, batch]
            exporters: [loki]

Environment
AWS Beanstalk with Docker Linux V2

Additional context

I need this to inject AWS Cloudfront geolocation headers into my spans :
https://aws.amazon.com/it/about-aws/whats-new/2020/07/cloudfront-geolocation-headers/

I have put my otel collector behind a non caching Cloudfront, and it is properly receiving POST http requests on v1/traces with those headers containing latitude and logitude.

I am sure everything is received properly because I am running tcpdump on 4318 port on otel collector and I DO see those headers.

Now I need to propagate those values from the http request, containing the spans in its body, to the span themself.

This is what include_metadata is supposed to enable, right ? Did I understand correctly ?

I also tried different combinations, trying to read from metadata context (inspired by https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/testdata/config.yaml) but none worked.

The only attribute I see properly set is "test" to value 123, but that's the only key not using "from_context".

Please help.

The text was updated successfully, but these errors were encountered:

amoscatelli · 2022-07-05T13:33:21Z

@jpkrohling sorry to bother you
is the from_context: CloudFront-Viewer-Longitude supposed to work ?
am I doing something wrong or this is an issue indeed ?

miaw7 · 2022-07-06T08:48:24Z

+1
The same issue for me. Version 0.54.0

Can't get any header from the request, but hardcoded key as in example above works
this is my configuration

receivers:
  otlp:
    protocols:
      http:
        include_metadata: true
        endpoint: 0.0.0.0:4318
      grpc: 
        endpoint: 0.0.0.0:4317

processors:
  attributes/from_context:
    actions:
      - key: http.x_request_id
        from_context: x_request_id
        action: upsert
exporters:
  logging:
    loglevel: info
  awsxray:
    region: eu-west-1
  awsemf:
    region: eu-west-1

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers:
        - otlp
      exporters:
        - logging
        - awsxray
      processors: [attributes/from_context]
    metrics:
      receivers:
        - otlp
      exporters:
        - logging
        - awsemf

amoscatelli · 2022-07-06T10:43:41Z

I can't understand why it's not working, it seemed to be ok just a bunch of monthes ago :
#4901

fatsheep9146 · 2022-07-06T15:57:44Z

@amoscatelli I saw you have two kinds of setting,
one is with "metadata." prefix, for example

          - key: http.client_latitude_2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert

another is not with any prefix

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert

Do these two settings both not work?

amoscatelli · 2022-07-06T16:08:57Z

@fatsheep9146 by sending traces to logging exporter I could verify that neither http.client_latitude or http.client_latitude_2 were added.

Only 'test' key was present :

          - key: test
            value: 123
            action: upsert

Which one is supposed to work ? With or without metadata prefix ?

fatsheep9146 · 2022-07-06T16:14:28Z

@amoscatelli
I check about the doc, but not very carefully. So I'm not really sure. For now, I think the one with prefix should work. But apparently it doesn't. I will try to reproduce this on my mac and see if I can find some clues.

amoscatelli · 2022-07-06T16:15:27Z

@fatsheep9146 give me a minute, I'll give it another try

amoscatelli · 2022-07-06T16:25:43Z

@fatsheep9146

Using this setting :

processors:
    attributes/traces:
        actions:
          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude3
            value: 123
            action: upsert
          - key: http.client_longitude3
            value: 123
            action: upsert

As you can see my OTEL receives requests with CloudFront-Viewer-Latitude header :

Looking inside my logs :

[ec2-user@ip-172-31-36-36 ~]$ docker logs 3c336b99b2e1 2>&1 | grep latitude
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)
     -> http.client_latitude3: INT(123)

This confirms that every from_context key attribute are not propagated/properly set, while the hardcoded/valued ones work properly.

Please help !

jpkrohling · 2022-07-06T16:57:25Z

If you have include_metadata: true in the receiver, I would expect metadata.CloudFront-Viewer-Latitude to work. Are you able to prepare a curl call that would reproduce the problem? I'll then debug and see where the problem is.

amoscatelli · 2022-07-07T06:53:00Z

This is what I have :

receivers:
    otlp:
        protocols:
            http:
                include_metadata: true
                cors:
                    allowed_origins: [
                        "http://localhost:4200",
                        ###########
                    ]

I'll try to prepare a curl
Thank you @jpkrohling

amoscatelli · 2022-07-07T08:12:37Z

@jpkrohling here you are :
https://justpaste.it/4nzjd

I just added -H "CloudFront-Viewer-Latitude: 44.44444" and sending some spans generated by my application (copied directly from the payload of a traces request in browser DevTools).

Was this what you needed ?

amoscatelli · 2022-07-07T08:22:31Z

@jpkrohling this is the output of tcpdump for a real request coming from my application :

POST /v1/traces HTTP/1.1
X-Forwarded-For: 2.118.34.194, 130.176.221.45
X-Forwarded-Proto: http
X-Forwarded-Port: 80
Host: otel.visiontech.cloud
X-Amzn-Trace-Id: Root=1-62c6964e-7fe418027d9aa9667040a970
Content-Length: 2820
User-Agent: Amazon CloudFront
X-Amz-Cf-Id: xo3-xBbnzFR0YbHGDWW-3t7fLa0tvMcz-wjrwqcvG_LmtmxQpWtgTQ==
Via: 1.1 c2375420915a59a31c2575fcfd6cae7a.cloudfront.net (CloudFront)
Content-Type: application/json
CloudFront-Viewer-Latitude: 41.89040
CloudFront-Viewer-Longitude: 12.51260

As you can see CloudFront-Viewer-Latitude and CloudFront-Viewer-Longitude headers are present.
Should I test and try to read other headers with processor attributes ?

Maybe User-Agent ?

Is there any reason why metadata should not be filled with particular headers or in particular conditions ?

amoscatelli · 2022-07-07T09:32:32Z

New test :

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude3
            value: 123
            action: upsert
          - key: http.client_longitude3
            value: 123
            action: upsert
          - key: http.client_latitude4
            from_context: metadata.CloudFront-Viewer-Latitude
            action: insert
          - key: http.client_longitude4
            from_context: metadata.CloudFront-Viewer-Longitude
            action: insert
          - key: http.client_latitude5
            from_context: CloudFront-Viewer-Latitude
            action: insert
          - key: http.client_longitude5
            from_context: CloudFront-Viewer-Longitude
            action: insert
          - key: test1
            from_context: metadata.X-Forwarded-For
            action: upsert
          - key: test2
            from_context: X-Forwarded-For
            action: upsert
          - key: test3
            from_context: metadata.X-Forwarded-For
            action: insert
          - key: test4
            from_context: X-Forwarded-For
            action: insert
          - key: test5
            from_context: metadata.X-Amzn-Trace-Id
            action: upsert
          - key: test6
            from_context: X-Amzn-Trace-Id
            action: upsert
          - key: test7
            from_context: metadata.X-Amzn-Trace-Id
            action: insert
          - key: test8
            from_context: X-Amzn-Trace-Id
            action: insert
          - key: test11
            from_context: metadata.Via
            action: upsert
          - key: test12
            from_context: Via
            action: upsert
          - key: test13
            from_context: metadata.Via
            action: insert
          - key: test14
            from_context: Via
            action: insert

Then this is logged :

Span #5
    Trace ID       : 0275913834ce9a99b5cc0047ded211c9
    Parent ID      : 
    ID             : 28827a6522de56bb
    Name           : click
    Kind           : SPAN_KIND_INTERNAL
    Start time     : 2022-07-07 09:14:05.8470003 +0000 UTC
    End time       : 2022-07-07 09:14:05.8471002 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message : 
Attributes:
     -> component: STRING(user-interaction)
     -> event_type: STRING(click)
     -> target_element: STRING(SPAN)
     -> target_xpath: STRING(//html/body/div[3]/div[2]/div/div/div/mat-option[10]/span)
     -> http.url: STRING(https://calc.optoplus.cloud/#/factory/compute)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> http.client_latitude3: INT(123)
     -> http.client_longitude3: INT(123)
     -> test1: STRING(2.118.34.194, 130.176.221.40)
     -> test2: STRING(2.118.34.194, 130.176.221.40)
     -> test3: STRING(2.118.34.194, 130.176.221.40)
     -> test4: STRING(2.118.34.194, 130.176.221.40)
     -> test5: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test6: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test7: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test8: STRING(Root=1-62c6a3e0-7a2f843d26dac2c1205716cd)
     -> test11: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
     -> test12: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
     -> test13: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
     -> test14: STRING(2.0 ec2e016357b2a4b61d6fc1a2e7c0826a.cloudfront.net (CloudFront))
Resource SchemaURL: 
Resource labels:
     -> service.name: STRING(optoplus-calc-prod)
     -> telemetry.sdk.language: STRING(webjs)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(1.2.0)

So the problem is related ONLY to the specific CloudFront-Viewer-Latitude and CloudFront-Viewer-Longitude headers ...
How can this be even possible 😢 ?

fatsheep9146 · 2022-07-07T10:30:40Z

@jpkrohling this is the output of tcpdump for a real request coming from my application :
POST /v1/traces HTTP/1.1
X-Forwarded-For: 2.118.34.194, 130.176.221.45
X-Forwarded-Proto: http
X-Forwarded-Port: 80
Host: otel.visiontech.cloud
X-Amzn-Trace-Id: Root=1-62c6964e-7fe418027d9aa9667040a970
Content-Length: 2820
User-Agent: Amazon CloudFront
X-Amz-Cf-Id: xo3-xBbnzFR0YbHGDWW-3t7fLa0tvMcz-wjrwqcvG_LmtmxQpWtgTQ==
Via: 1.1 c2375420915a59a31c2575fcfd6cae7a.cloudfront.net (CloudFront)
Content-Type: application/json
CloudFront-Viewer-Latitude: 41.89040
CloudFront-Viewer-Longitude: 12.51260
As you can see CloudFront-Viewer-Latitude and CloudFront-Viewer-Longitude headers are present. Should I test and try to read other headers with processor attributes ?

Maybe User-Agent ?

Is there any reason why metadata should not be filled with particular headers or in particular conditions ?

I guess maybe http header value should not be float ? Can you set CloudFront-Viewer-Latitude value to 41 to give it a try?

amoscatelli · 2022-07-07T11:09:15Z

Config :

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude3
            from_context: metadata.X-CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude3
            from_context: metadata.X-CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude4
            from_context: X-CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude4
            from_context: X-CloudFront-Viewer-Longitude
            action: upsert
          - key: test
            from_context: Test
            action: upsert

Curl :

curl 'http://172.31.36.36:4318/v1/traces' \
  -H 'sec-ch-ua: " Not;A Brand";v="99", "Microsoft Edge";v="103", "Chromium";v="103"' \
  -H 'Referer: https://calc.optoplus.cloud/' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'Content-Type: application/json' \
  -H 'X-CloudFront-Viewer-Latitude: 44.44444' \
  -H 'X-CloudFront-Viewer-Longitude: pippo' \
  -H 'Test: 44.44444' \
  --data-raw '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"dummy"}},{"key":"telemetry.sdk.language","value":{"stringValue":"webjs"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.2.0"}}],"droppedAttributesCount":0},"instrumentationLibrarySpans":[{"spans":[{"traceId":"6e8bdcda0d523802ad34beab84f8f623","spanId":"265476d1a6cfa81f","name":"longtask","kind":1,"startTimeUnixNano":1657191217063300000,"endTimeUnixNano":1657191217245300200,"attributes":[{"key":"component","value":{"stringValue":"long-task"}},{"key":"longtask.name","value":{"stringValue":"self"}},{"key":"longtask.entry_type","value":{"stringValue":"longtask"}},{"key":"longtask.duration","value":{"intValue":182}},{"key":"longtask.attribution.name","value":{"stringValue":"unknown"}},{"key":"longtask.attribution.entry_type","value":{"stringValue":"taskattribution"}},{"key":"longtask.attribution.start_time","value":{"intValue":0}},{"key":"longtask.attribution.duration","value":{"intValue":0}},{"key":"longtask.attribution.container_type","value":{"stringValue":"window"}},{"key":"longtask.attribution.container_src","value":{"stringValue":""}},{"key":"longtask.attribution.container_id","value":{"stringValue":""}},{"key":"longtask.attribution.container_name","value":{"stringValue":""}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0},{"traceId":"532be1a850f11ba96980c49a75f921f8","spanId":"5553289df6522b13","name":"longtask","kind":1,"startTimeUnixNano":1657191217255300000,"endTimeUnixNano":1657191217518300200,"attributes":[{"key":"component","value":{"stringValue":"long-task"}},{"key":"longtask.name","value":{"stringValue":"self"}},{"key":"longtask.entry_type","value":{"stringValue":"longtask"}},{"key":"longtask.duration","value":{"intValue":263}},{"key":"longtask.attribution.name","value":{"stringValue":"unknown"}},{"key":"longtask.attribution.entry_type","value":{"stringValue":"taskattribution"}},{"key":"longtask.attribution.start_time","value":{"intValue":0}},{"key":"longtask.attribution.duration","value":{"intValue":0}},{"key":"longtask.attribution.container_type","value":{"stringValue":"window"}},{"key":"longtask.attribution.container_src","value":{"stringValue":""}},{"key":"longtask.attribution.container_id","value":{"stringValue":""}},{"key":"longtask.attribution.container_name","value":{"stringValue":""}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0}],"instrumentationLibrary":{"name":"@opentelemetry/instrumentation-long-task","version":"0.30.0"}},{"spans":[{"traceId":"9c62ddae5d5f1961bb919c80a8ccc1c2","spanId":"c3798a50ddbf1f89","name":"HTTP POST","kind":3,"startTimeUnixNano":1657191210572800300,"endTimeUnixNano":1657191217243000300,"attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.url","value":{"stringValue":"https://services.optoplus.cloud/optoplus-services-web/api/lens/compute"}},{"key":"http.response_content_length","value":{"intValue":0}},{"key":"http.status_code","value":{"intValue":200}},{"key":"http.status_text","value":{"stringValue":""}},{"key":"http.host","value":{"stringValue":"services.optoplus.cloud"}},{"key":"http.scheme","value":{"stringValue":"https"}},{"key":"http.user_agent","value":{"stringValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44"}}],"droppedAttributesCount":0,"events":[{"timeUnixNano":1657191210572800300,"name":"open","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210573500200,"name":"send","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210574900200,"name":"fetchStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"secureConnectionStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"requestStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"responseStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217063300000,"name":"responseEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217243000300,"name":"loaded","attributes":[],"droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0},{"traceId":"8c36567b254a0b87462c16829cf52d28","spanId":"d6e3fb36397a117d","name":"HTTP POST","kind":3,"startTimeUnixNano":1657191210577000200,"endTimeUnixNano":1657191217514400300,"attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.url","value":{"stringValue":"https://services.optoplus.cloud/optoplus-services-web/api/lens/compute"}},{"key":"http.response_content_length","value":{"intValue":0}},{"key":"http.status_code","value":{"intValue":200}},{"key":"http.status_text","value":{"stringValue":""}},{"key":"http.host","value":{"stringValue":"services.optoplus.cloud"}},{"key":"http.scheme","value":{"stringValue":"https"}},{"key":"http.user_agent","value":{"stringValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44"}}],"droppedAttributesCount":0,"events":[{"timeUnixNano":1657191210577000200,"name":"open","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210577800200,"name":"send","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191210579900200,"name":"fetchStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"domainLookupEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"secureConnectionStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"connectEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"requestStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657181768969600300,"name":"responseStart","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217136300300,"name":"responseEnd","attributes":[],"droppedAttributesCount":0},{"timeUnixNano":1657191217514400300,"name":"loaded","attributes":[],"droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0}],"instrumentationLibrary":{"name":"@opentelemetry/instrumentation-xml-http-request","version":"0.29.2"}}]}]}' \
  --compressed

Log :

Span #0
    Trace ID       : 6c8804c5066fa1207302e906b6637c25
    Parent ID      : 
    ID             : b4d6d5d41ebb75b6
    Name           : click
    Kind           : SPAN_KIND_INTERNAL
    Start time     : 2022-07-07 10:48:47.0784003 +0000 UTC
    End time       : 2022-07-07 10:48:47.0787003 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message : 
Attributes:
     -> component: STRING(user-interaction)
     -> event_type: STRING(click)
     -> target_element: STRING(MAT-ICON)
     -> target_xpath: STRING(//html/body/app-root/vt-mat-app-bar/mat-drawer-container/mat-drawer-content/div/app-compute/div/div/form/mat-card/mat-card-actions/button/span/mat-icon)
     -> http.url: STRING(https://calc.optoplus.cloud/#/factory/compute)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> test: STRING(44.44444)

For god sake it seems a matter of NAME of the header ??? 😠

amoscatelli · 2022-07-07T11:57:04Z

Yea, I confirm, the issue is related to the name of the header.

'Test' is ok,
'X-CloudFront-Viewer-Latitude' is not ok,
'CloudFront-Viewer-Latitude' is not ok.

@jpkrohling
How can I help on fixing this ?
I don't think I can change the header name from Cloudfront ...

jpkrohling · 2022-07-07T11:57:57Z

I think I have enough information here to debug the actual code. I'll check and get back to you soon.

jpkrohling · 2022-07-07T13:30:27Z

Looks like I solved the mystery: the attribute is stored as X-Cloudfront-Viewer-Latitude within the context. Note the lowercase "front" in "CloudFront". Not sure yet why this is so, but you can get it working with a configuration like this:

receivers:
  otlp:
    protocols:
      http:
        include_metadata: true

exporters:
  logging:
    loglevel: debug

processors:
  attributes:
    actions:
      - key: http.client_latitude2
        from_context: metadata.X-Cloudfront-Viewer-Latitude
        action: upsert
extensions:

service:
  extensions:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes]
      exporters: [logging]

Resulting in:

Attributes:
     -> http.method: STRING(POST)
     -> http.url: STRING(https://services.optoplus.cloud/optoplus-services-web/api/lens/compute)
     -> http.response_content_length: INT(0)
     -> http.status_code: INT(200)
     -> http.status_text: STRING()
     -> http.host: STRING(services.optoplus.cloud)
     -> http.scheme: STRING(https)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> http.client_latitude2: STRING(44.44444)

amoscatelli · 2022-07-07T13:45:16Z

THANK YOU very much @jpkrohling , going to test it right now ...

EDIT : Yea, this is working properly !

Anyway this should be fixed, the match with the header should be ignore case !!!

X-Cloudfront-Viewer-Latitude and x-cloudfront-viewer-latitude are the same http header, doing otherwise could lead to issues with libraries and sdk IMHO.

amoscatelli · 2022-07-07T13:56:24Z

I never worked with GO language, I don't know where to start, but if that's the case I can help of course

jpkrohling · 2022-07-07T14:09:27Z

We had a similar problem when it comes to the "authorization" header, and I thought about normalizing the keys either in lower or upper case: open-telemetry/opentelemetry-collector-contrib#8994 (comment)

Having an accessor somewhere in the path would make it easy to access the normalized version from the context.

Before this change, users would make references to headers such as "authorization" and "X-CloudFront-Viewer-Latitude", which would potentially fail, as they would be stored in the context as "X-Cloudfront-Viewer-Latitude" or "Authorization". This PR changes this behavior, to attempt a case insensitive lookup to the backing map in case the key wasn't found under the requested casing. Under the assumption that the lookup key specified by users won't change, on a subsequent lookup, we proactively copy the value to the looked up key, to make the next lookup faster. Fixes open-telemetry#5610 Fixes open-telemetry/opentelemetry-collector-contrib#8994 Signed-off-by: Juraci Paixão Kröhling <[email protected]>

jpkrohling · 2022-07-07T14:38:47Z

I just submitted a new PR that would fix this:

processors:
  attributes:
    actions:
      - key: http.client_latitude
        from_context: metadata.X-CloudFront-Viewer-Latitude
        action: upsert
      - key: http.client_latitude2
        from_context: metadata.X-Cloudfront-Viewer-Latitude
        action: upsert

Attributes:
     -> http.method: STRING(POST)
     -> http.url: STRING(https://services.optoplus.cloud/optoplus-services-web/api/lens/compute)
     -> http.response_content_length: INT(0)
     -> http.status_code: INT(200)
     -> http.status_text: STRING()
     -> http.host: STRING(services.optoplus.cloud)
     -> http.scheme: STRING(https)
     -> http.user_agent: STRING(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44)
     -> http.client_latitude: STRING(44.44444)
     -> http.client_latitude2: STRING(44.44444)

amoscatelli · 2022-07-07T15:10:03Z

Great !

Before this change, users would make references to headers such as "authorization" and "X-CloudFront-Viewer-Latitude", which would potentially fail, as they would be stored in the context as "X-Cloudfront-Viewer-Latitude" or "Authorization". This PR changes this behavior, to attempt a case insensitive lookup to the backing map in case the key wasn't found under the requested casing. Under the assumption that the lookup key specified by users won't change, on a subsequent lookup, we proactively copy the value to the looked up key, to make the next lookup faster. Fixes open-telemetry#5610 Fixes open-telemetry/opentelemetry-collector-contrib#8994 Signed-off-by: Juraci Paixão Kröhling <[email protected]>

* Change metadata lookup to be case insensitive Before this change, users would make references to headers such as "authorization" and "X-CloudFront-Viewer-Latitude", which would potentially fail, as they would be stored in the context as "X-Cloudfront-Viewer-Latitude" or "Authorization". This PR changes this behavior, to attempt a case insensitive lookup to the backing map in case the key wasn't found under the requested casing. Under the assumption that the lookup key specified by users won't change, on a subsequent lookup, we proactively copy the value to the looked up key, to make the next lookup faster. Fixes #5610 Fixes open-telemetry/opentelemetry-collector-contrib#8994 Signed-off-by: Juraci Paixão Kröhling <[email protected]> * Add changelog entry Signed-off-by: Juraci Paixão Kröhling <[email protected]>

amoscatelli added the bug Something isn't working label Jun 30, 2022

jpkrohling self-assigned this Jul 7, 2022

jpkrohling mentioned this issue Jul 7, 2022

Change metadata lookup to be case insensitive #5646

Merged

codeboten closed this as completed in #5646 Jul 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[processor/attributes] unable to add span custom attribute from custom header #5610

[processor/attributes] unable to add span custom attribute from custom header #5610

amoscatelli commented Jun 30, 2022

amoscatelli commented Jul 5, 2022

miaw7 commented Jul 6, 2022 •

edited

Loading

amoscatelli commented Jul 6, 2022

fatsheep9146 commented Jul 6, 2022

amoscatelli commented Jul 6, 2022 •

edited

Loading

fatsheep9146 commented Jul 6, 2022

amoscatelli commented Jul 6, 2022

amoscatelli commented Jul 6, 2022 •

edited

Loading

jpkrohling commented Jul 6, 2022

amoscatelli commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

amoscatelli commented Jul 7, 2022 •

edited

Loading

amoscatelli commented Jul 7, 2022

fatsheep9146 commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

amoscatelli commented Jul 7, 2022 •

edited

Loading

amoscatelli commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

[processor/attributes] unable to add span custom attribute from custom header #5610

[processor/attributes] unable to add span custom attribute from custom header #5610

Comments

amoscatelli commented Jun 30, 2022

amoscatelli commented Jul 5, 2022

miaw7 commented Jul 6, 2022 • edited Loading

amoscatelli commented Jul 6, 2022

fatsheep9146 commented Jul 6, 2022

amoscatelli commented Jul 6, 2022 • edited Loading

fatsheep9146 commented Jul 6, 2022

amoscatelli commented Jul 6, 2022

amoscatelli commented Jul 6, 2022 • edited Loading

jpkrohling commented Jul 6, 2022

amoscatelli commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

amoscatelli commented Jul 7, 2022 • edited Loading

amoscatelli commented Jul 7, 2022

fatsheep9146 commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

amoscatelli commented Jul 7, 2022 • edited Loading

amoscatelli commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

jpkrohling commented Jul 7, 2022

amoscatelli commented Jul 7, 2022

miaw7 commented Jul 6, 2022 •

edited

Loading

amoscatelli commented Jul 6, 2022 •

edited

Loading

amoscatelli commented Jul 6, 2022 •

edited

Loading

amoscatelli commented Jul 7, 2022 •

edited

Loading

amoscatelli commented Jul 7, 2022 •

edited

Loading