OTel array attributes not recorded as labels

[stuartnelson3]

APM Server version (apm-server version):

# ./apm-server version
apm-server version 7.14.0 (amd64), libbeat 7.14.0 [e369a0ef477ce27654ac3a401773ec7806bd3394 built 2021-07-08 06:28:07 +0000 UTC]

Description of the problem including expected versus actual behavior:

When sending traces using an attribute array, the values do not appear in the document in elasticsearch. When changing the code to send a single attribute value, the value appears as a label.

single attribute:

Steps to reproduce:

1. Start elasticsearch + kibana + apm-server build candidates for v7.14
2. Run this server: EXPORTER_ENDPOINT=localhost:8200 go run main.go
3. curl http://localhost:8080/users/123
4. Check the last document when querying elasticsearch GET apm-7.14.0-span-000001/_search

The gist linked above has a commented out line at the bottom setting a single string attribute.

[axw]

I did it for transactions, but forgot to do it for spans too. Also resource attributes.

[stuartnelson3]

Testing against #5704, array attributes at the resource level are correctly handled, but when I add them at the span level they still aren't appearing. Single value attributes added at the span level aren't appearing for me now, either.

version: apm-server version 7.14.0 (amd64), libbeat 7.14.0 [bea70f612b1d70561dbedd57b1cd21fe56120795 built 2021-07-19 05:24:50 +0000 UTC]

gist: https://gist.github.com/stuartnelson3/61b9e61bf4ab78955dc0a7844d26c626 (updated to add resource attributes)

[axw]

@stuartnelson3 I just ran the latest 7.14-SNAPSHOT image, and pointed your gist at it:

$ docker run --rm -p 8200:8200 docker.elastic.co/apm/apm-server:7.14.0-SNAPSHOT -E output.console.enabled=true -E output.elasticsearch.enabled=false

{"@timestamp":"2021-07-20T03:29:01.042Z","@metadata":{"beat":"apm-server","type":"_doc","version":"7.14.0","pipeline":"apm"},"processor":{"name":"onboarding","event":"onboarding"},"observer":{"type":"apm-server","hostname":"612749cc685d","version":"7.14.0","id":"bddf529d-1f19-42b5-af1d-90b39872f9bf","ephemeral_id":"ec2a9101-cc4a-48e1-94aa-ff81c5427d9b","listening":"[::]:8200","version_major":7},"ecs":{"version":"1.10.0"}}
{"@timestamp":"2021-07-20T03:29:43.058Z","@metadata":{"beat":"apm-server","type":"_doc","version":"7.14.0","pipeline":"apm"},"processor":{"name":"transaction","event":"transaction"},"trace":{"id":"d00913a9715a0db786faed16bb830744"},"http":{"version":"1.1","request":{"socket":{"remote_address":"127.0.0.1:40876"},"method":"GET"},"response":{"status_code":200}},"url":{"port":8080,"original":"/users/123","scheme":"http","full":"http://localhost:8080/users/123","domain":"localhost","path":"/users/123"},"service":{"name":"unknown","language":{"name":"unknown"},"framework":{"name":"go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux","version":"semver:0.20.0"}},"client":{"ip":"127.0.0.1","port":40876},"labels":{"unhandled_ids":["123","456"],"net_transport":"IP.TCP","http_route":"/users/{id:[0-9]+}"},"agent":{"name":"otlp","version":"unknown"},"source":{"ip":"127.0.0.1","port":40876},"transaction":{"type":"request","duration":{"us":146},"name":"/users/{id:[0-9]+}","result":"HTTP 2xx","sampled":true,"id":"8fad4671809d2b1b"},"user_agent":{"original":"curl/7.68.0"},"timestamp":{"us":1626751783058996},"event":{"outcome":"success"},"observer":{"version":"7.14.0","id":"bddf529d-1f19-42b5-af1d-90b39872f9bf","ephemeral_id":"ec2a9101-cc4a-48e1-94aa-ff81c5427d9b","version_major":7,"type":"apm-server","hostname":"612749cc685d"},"ecs":{"version":"1.10.0"}}
{"@timestamp":"2021-07-20T03:29:43.059Z","@metadata":{"beat":"apm-server","type":"_doc","version":"7.14.0","pipeline":"apm"},"agent":{"name":"otlp","version":"unknown"},"labels":{"other_ids":["778","097"],"unhandled_ids":["123","456"]},"trace":{"id":"d00913a9715a0db786faed16bb830744"},"parent":{"id":"8fad4671809d2b1b"},"timestamp":{"us":1626751783059088},"event":{"outcome":"success"},"ecs":{"version":"1.10.0"},"span":{"name":"getUser","type":"app","id":"4afd0ff2786b6792","duration":{"us":2}},"service":{"name":"unknown","language":{"name":"unknown"}},"observer":{"version":"7.14.0","id":"bddf529d-1f19-42b5-af1d-90b39872f9bf","ephemeral_id":"ec2a9101-cc4a-48e1-94aa-ff81c5427d9b","version_major":7,"type":"apm-server","hostname":"612749cc685d"},"processor":{"name":"transaction","event":"span"}}
{"@timestamp":"2021-07-20T03:30:01.041Z","@metadata":{"beat":"apm-server","type":"_doc","version":"7.14.0","pipeline":"apm"},"timeseries":{"instance":"unknown:/users/{id:[0-9]+}:a538892d8f033dbf"},"observer":{"hostname":"612749cc685d","version":"7.14.0","id":"bddf529d-1f19-42b5-af1d-90b39872f9bf","ephemeral_id":"ec2a9101-cc4a-48e1-94aa-ff81c5427d9b","version_major":7,"type":"apm-server"},"ecs":{"version":"1.10.0"},"transaction":{"type":"request","duration":{"histogram":{"counts":[1],"values":[146]}},"name":"/users/{id:[0-9]+}","result":"HTTP 2xx","root":true},"_doc_count":1,"agent":{"name":"otlp"},"processor":{"name":"metric","event":"metric"},"service":{"name":"unknown"},"event":{"outcome":"success"},"metricset.name":"transaction"}

There's 3 events here: transaction, span, and latency histogram metric. The transaction only has the resource attributes, while the span has both resource and span attributes. That's expected from the code, because only the span (getUser) has span attributes specified. Were you looking at the transaction doc and not span doc maybe?

[stuartnelson3]

correct, I was looking at the transactions index. thanks!