[exporter/datadog]: Failed to retrieve metadata

[inigohu]

What happened?

Some warning messages appear in our logs after upgrading to version v0.60.0.
We don't want to send host metrics to Datadog and it seems to be mandatory with the latest version.

Collector version

v0.60.0

Environment information

Gateway deployment mode on Google GKE autopilot.

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 500
  resource:
    attributes:
      - key: deployment.environment
        value: test
        action: upsert
      - key: cloud.region
        value: test
        action: upsert
  batch:
    send_batch_size: 256
    send_batch_max_size: 512
exporters:
  datadog:
    api:
      key: ${DD_API_KEY}
      site: datadoghq.eu
    metrics:
      sums:
        cumulative_monotonic_mode: to_delta
      histograms:
        mode: distributions
        send_count_sum_metrics: true
      resource_attributes_as_tags: true
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [datadog]

Log output

[Debug] Error fetching info for pid 1: open /etc/passwd: no such file or directory
warn	gohai/gohai.go:66	Failed to retrieve platform metadata	{"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}
warn	gohai/gohai.go:47	Failed to retrieve filesystem metadata	{"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "df failed to collect filesystem data: %!s(<nil>)"}

[Sunilwali679]

Hi @inigohu . I am interested to work on this could you please assign this to me ??

[inigohu]

Hi @inigohu . I am interested to work on this could you please assign this to me ??

Hi @Sunilwali679, I have no privileges to assign this issue to you, I am not a CODEOWNER.

[Sunilwali679]

@mx-psi Could you help on this issue assignment ?

[mx-psi]

Done! Also cc @dineshg13 to review this work

[husseinraoouf]

I think this issue should be with higher priority, the datadog exporter is essentially broken for metrics

at least there should be a workaround until the issue is fixed

[dineshg13]

@husseinraoouf sorry for delay in fixing this. I will be taking this on priority and fix same.

[husseinraoouf]

Thank you for the quick response

[dineshg13]

@husseinraoouf to clarify, this error message shouldn't stop datadog exporter from reporting metrics or have any other issues. This is happening as the docker image doesn't have required dependencies.

There are few routes we can take.

1. Classify the warning as info, with additional information that it doesn't impact the exporter. (preferred )
2. Add the dependencies to the official built , might be hard to get approval
3. Remove these metadata collector (least preferred )

@mx-psi let me know your thoughts on same.

[mx-psi]

@dineshg13 I think we should definitely do (1) now to fix the immediate issue. We can consider how to improve support in future PRs (possibly through (2) or through alternative options), but this was working fine before, so we should turn the error into a warning first

[ronenl3]

I'm running the Collector with v0.71.0 and see these log messages.
Is the issue going to be solved? I don't see any Infrastructure metrics in my spans since I updated my OTel Collector.

[mx-psi]

I don't see any Infrastructure metrics in my spans since I updated my OTel Collector.

Could you open a separate issue for this? I don't think your issue is related to the log messages here

[brunocrt]

I still see this same issue on messages running version 0.67.0,

2023-03-03T22:20:42.839Z        info    gohai/gohai.go:47       Failed to retrieve filesystem metadata  {"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "df failed to collect filesystem data: %!s(<nil>)"}
2023-03-03T22:20:42.840Z        info    gohai/gohai.go:66       Failed to retrieve platform metadata    {"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}

[regisverdin]

Still seeing this in version 0.72.0

gohai/gohai.go:47    Failed to retrieve filesystem metadata  {"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "df failed to collect filesystem data: %!s(<nil>)"}
gohai/gohai.go:66    Failed to retrieve platform metadata    {"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}

[mcatany]

Getting the same error above after upgrading to 0.77.0.

2023-05-18T12:47:49.516Z	info	gohai/gohai.go:47	Failed to retrieve filesystem metadata	{"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "df failed to collect filesystem data: %!s(<nil>)"}
2023-05-18T12:47:49.522Z	info	gohai/gohai.go:66	Failed to retrieve platform metadata	{"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}

[caromeva]

I have the same error in 0.78.0.
image: otel/opentelemetry-collector-contrib:0.78.0

info gohai/gohai.go:36 Failed to retrieve filesystem metadata {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "df failed to collect filesystem data: %!s(<nil>)"} info gohai/gohai.go:55 Failed to retrieve platform metadata {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}

[VinxItak]

Hi guys, did you found a solution for this ? I'm not able to retrieve metadata in DD (can see hosts and metrics, but without k8s data), i have this error

image: otel/opentelemetry-collector-contrib:0.84.0

info    gohai/gohai.go:35    Failed to retrieve filesystem metadata    {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "df failed to collect filesystem data: %!s(<nil>)"}
info    gohai/gohai.go:54    Failed to retrieve platform metadata    {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "exec: \"uname\": executable file not found in $PATH"}

[ringerc]

To silence this annoying noise, use

    exporters: 
      datadog:
        # ...
        host_metadata:
          enabled: false

[mx-psi]

Setting host_metadata::enable to false will reduce the amount of information you can see on the Infrastructure List and Host Map list and is not generally recommended. The log can be safely ignored.

[ringerc]

If it can be ignored, it shouldn't log at info so it's not shown by default.

Also, there's a bug where one of the messages

1698273439797527886 [Debug] Error fetching info for pid 1: open /etc/passwd: no such file or directory

ignores the .service.telemetry.logs.encoding: json configuration option and writes a plaintext log line anyway.

Why does DD need this if the hostmetrics collector is enabled anyway? Any extra info should really be collected as a separate receiver in a pipeline, not wedged into the output plugin.

[ringerc]

DataDog/gohai#37 is closed now, so I've re-raised this as DataDog/datadog-agent#21487

[ringerc]

I've raised #29741 to follow up for this