XRay traces not connected to log entries in Cloudwatch

[mpetrik-profinit]

Describe the bug
We've migrated from Java X-Ray SDK to AWS OTEL. And the thing we are facing now is the absence of log records while analyzing traces. This worked with X-Ray SDK:

However, with OTEL, the logs are not mapped (we use Fluentbit). The only difference we've found in the metrics/traces messages (compared to XRay SDK) is the absence of the following key:

...
Segments: [
 {
  ...
  "document": {
   ....
   "aws": {
     ...
     "cloudwatch_logs": {
         "log_group": "our log group"
     ....

The aws.cloudwatch_logs.log_group is missing when using OTEL. The documentation says it translates from aws.log.group.names (https://aws-otel.github.io/docs/getting-started/x-ray#otel-span-cw-logs-metadata-translation).

Steps to reproduce

• Use EKS Cluster with OTEL collector (Sidecar: public.ecr.aws/aws-observability/aws-otel-collector:latest) and the following configuration:

    extensions:
      health_check:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      awsxray:
        endpoint: 0.0.0.0:2000
        transport: udp
    processors:
      batch/traces:
        timeout: 1s
        send_batch_size: 50
      batch/metrics:
        timeout: 60s
    exporters:
      awsxray:
        region: eu-west-1   
      awsemf:
        region: eu-west-1
    service:
      pipelines:
        traces:
          receivers: [otlp,awsxray]
          processors: [batch/traces]
          exporters: [awsxray]
        metrics:
          receivers: [otlp]
          processors: [batch/metrics]
          exporters: [awsemf]
      extensions: [health_check]

• Use Fluentbit to send logs to CloudWatch
• Use SpringBoot application v2.7.0 with OTEL agent (1.17.0-aws)

Call a REST Endpoint (the endpoint logs to console with Log4J2). The log is visible in Cloudwatch, the log entry contains AWS_XRAY_TRACE_ID.

What did you expect to see?
I see a log entry in the trace detail (as with the previous XRay SDK)

What did you see instead?
I see traces without log information, however, the log entries are visible in Log groups in Cloudwatch.

Additional context
We've tried configuration using environment properties (AWS_LOG_GROUP_NAMES/ARNS), in OTEL_RESOURCE_ATTRIBUTES, in span attributes, however, with no luck. Are we missing some kind of configuration (pod roles?) here?

[willarmiros]

Hi @mpetrik-profinit, thanks for raising this! The problem appears to be that the Java EKS resource detector does not record the aws.log.group.names attribute. However the EKS Plugin in X-Ray SDK does with this logic.

We will add a backlog task to add this functionality. We are also open to a PR to copy in this functionality to the EKS Detector!

We've tried configuration using environment properties (AWS_LOG_GROUP_NAMES/ARNS), in OTEL_RESOURCE_ATTRIBUTES, in span attributes, however, with no luck. Are we missing some kind of configuration (pod roles?) here?

Could you elaborate more? What value did you set AWS_LOG_GROUP_NAMES to? Did the value show up on the X-Ray trace?

[rapphil]

OTEL_RESOURCE_ATTRIBUTES does not support arrays: open-telemetry/opentelemetry-specification#2857

I think our best option is to update the resource detectors while the issue with the spec is not solved. I will be working in adding the same logic that is used in the xray sdk to all sdks that implement resource detectors.

[rapphil]

Moved discussion to upstream open-telemetry/opentelemetry-java-contrib#564

[rapphil]

In the next release of the aws-otel-collector, v0.26.0 , we will add a couple of features that will make your life easier.

• Support for setting the log groups using the OTEL_RESOURCE_ATTRIBUTES environment variable: [exporter/awsxrayexporter]: Add support to string resource attributes open-telemetry/opentelemetry-collector-contrib#17503
• Add support to setting log group properties in the awsxray configuration in the collector [exporter/awsxray] Adding Log Group Names to X-Ray Exporter Config open-telemetry/opentelemetry-collector-contrib#16859

Unfortunately it is not possible to solve this issue in the resource detectors as there is no way to implement the log group detection in a stable and reproducible way.

[rapphil]

Hi,
As I mentioned in open-telemetry/opentelemetry-java-contrib#564 it is not possible to implement resource detectors that will reliably work across all environments, therefore that option will not be implemented.

However, collector v0.26.0 was released. It is now possible to set the log group property using the two mechanisms described above. https://github.com/aws-observability/aws-otel-collector/releases/tag/v0.26.0

Please use the OTEL_RESOURCE_ATTRIBUTES variable to set the aws.log.group.names resource attribute.

Please refer to this section:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awsxrayexporter#traces-and-logs-correlation

I'm going to close the ticket since the original problem was addressed. Please reopen if you are still having issues.