[receiver/hostmetrics] filesystem scraper doesn't respect root_path #35990

povilasv · 2024-10-25T06:04:25Z

Component(s)

receiver/hostmetrics

What happened?

Description

Looks like #35504 gave us a regression.

Now filesystem scraper ignores root_path and tries to just open /mounts althought root_path is set. Uses with root_path will start getting these errors, and no filesystem metrics:

2024-10-25T08:45:13.299+0300    error   scraperhelper/scrapercontroller.go:205  Error scraping metrics  {"kind": "receiver", "name": "hostmetrics", "data_type": "metrics", "error": "open /mounts: no such file or directory", "scraper": "filesystem"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:205
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:177

IMO this is an important regression, as we use this in opentelemetry-helm-charts hostMetrics preset -> https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/templates/_config.tpl#L71-L88

Steps to Reproduce

receivers:
  hostmetrics:
    root_path: /tmp
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.time:
            enabled: true
      disk:
        metrics:
          system.disk.io:
            enabled: true
          system.disk.operations:
            enabled: true
      filesystem:
        metrics:
          system.filesystem.usage:
            enabled: true
      load:
        metrics:
          system.cpu.load_average.1m:
            enabled: true
          system.cpu.load_average.5m:
            enabled: true
          system.cpu.load_average.15m:
            enabled: true
      memory:
        metrics:
          system.memory.usage:
            enabled: true
      network:
        metrics:
          system.network.connections:
            enabled: true
          system.network.io:
            enabled: true
      paging:
        metrics:
          system.paging.operations:
            enabled: true
          system.paging.usage:
            enabled: true
      process:
        metrics:
          process.cpu.time:
            enabled: true
          process.memory.usage:
            enabled: true

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [batch]
      exporters: [debug]

  telemetry:
    logs:
      level: "debug"

You will get:

2024-10-25T09:03:56.691+0300    error   scraperhelper/scrapercontroller.go:205  Error scraping metrics  {"kind": "receiver", "name": "hostmetrics", "data_type": "metrics", "error": "open /mounts: no such file or directory", "scraper": "filesystem"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:205
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:177

although it should try to open /tmp/mounts

Expected Result

Actual Result

Collector version

v0.102.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2024-10-25T06:04:42Z

Pinging code owners:

receiver/hostmetrics: @dmitryax @braydonk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

TylerHelmuth · 2024-10-25T13:16:48Z

@povilasv is there a work around?

povilasv · 2024-10-25T13:46:10Z

Workaround is to manually set the env var:

HOST_PROC_MOUNTINFO=/hostfs/proc/1

marcelaraujo · 2024-10-25T14:38:28Z

@povilasv, there is the same issue here after upgrading to the latest version.

{"level":"error","ts":1729866966.393247,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"open /mounts: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:181"}
{"level":"error","ts":1729866996.3939178,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"open /mounts: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:181"}
{"level":"error","ts":1729867026.3932698,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"open /mounts: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:181"}

povilasv · 2024-10-25T15:10:10Z

@marcelaraujo try setting the environment variable to HOST_PROC_MOUNTINFO=/hostfs/proc/1

it should workaround this issue

marcelaraujo · 2024-10-25T15:31:28Z

Hi @povilasv

It didn't work.

What should be the values for this case?

env:
   - name: HOST_PROC_MOUNTINFO
     value: /proc/1/self
volumes:
   - name: hostfs
     hostPath:
        path: /
volumeMounts:
   - name: hostfs
     mountPath: /hostfs
     readOnly: true
     mountPropagation: HostToContainer
config:
   receivers:
      hostmetrics:
          root_path: /hosts

When I tried using your environment variable, I got a different issue

{"level":"error","ts":1729870334.3869734,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"failed to read usage at /hostfs/conf: no such file or directory; failed to read usage at /hostfs/hostfs/run/containerd/io.containerd.runtime.v2.task/k8s.io/b9d436f09506a98c09ba162dc17d2692f70430adf0601dfc1cd2f676c0253b80/rootfs/conf: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:177"}

atoulme · 2024-10-25T16:06:02Z

Looking at this issue and going to attempt to reproduce ; the environment variable is used in 2 places and setting it as a workaround might not be the fix.

atoulme · 2024-10-25T16:11:00Z

Using your config file, on Mac, trying to reproduce:

docker run --rm -it -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml -v /:/tmp otel/opentelemetry-collector-contrib:latest

I don't see the error reported. I will try to reproduce on a Linux VM next.

atoulme · 2024-10-25T16:18:07Z

Reproducing on Linux now, taking it further.

atoulme · 2024-10-25T16:21:31Z

Adding -e HOST_PROC_MOUNTINFO=/tmp/proc/1 doesn't fix the issue.

2024-10-25T16:20:16.489Z	error	scraperhelper/scrapercontroller.go:205	Error scraping metrics	{"kind": "receiver", "name": "hostmetrics", "data_type": "metrics", "error": "failed to read usage at /tmp/tmp/boot: no such file or directory; failed to read usage at /tmp/tmp/boot/efi: no such file or directory; failed to read usage at /tmp/tmp/snap/snapd/21759: no such file or directory; failed to read usage at /tmp/tmp/snap/amazon-ssm-agent/7993: no such file or directory; failed to read usage at /tmp/tmp/snap/core18/2829: no such file or directory; failed to read usage at /tmp/tmp/snap/core22/1621: no such file or directory; failed to read usage at /tmp/tmp/snap/amazon-ssm-agent/9565: no such file or directory; failed to read usage at /tmp/tmp/snap/core18/2846: no such file or directory; failed to read usage at /tmp/tmp/snap/snapd/22991: no such file or directory; failed to read usage at /tmp/tmp/snap/core22/1663: no such file or directory; failed to read usage at /tmp/etc/otelcol-contrib/config.yaml: no such file or directory", "scraper": "hostmetrics"}

atoulme · 2024-10-25T16:39:29Z

Using -e HOST_PROC_MOUNTINFO="" fixes the issue, setting the correct default value for this env var. The fix is to not set the default value of HOST_PROC_MOUNTINFO in the envMap at all as any non-empty value is going to skew things. I will open a PR in a second with a fix.

atoulme · 2024-10-25T16:47:34Z

#36000 is a fix.

I am going to try and revive #32536 to make sure we have this fix tested.

marcelaraujo · 2024-10-25T17:26:52Z

@atoulme Confirming the suggestion worked.

…e system is mounted (open-telemetry#36000)  #### Description Do not set the default value of HOST_PROC_MOUNTINFO to respect root_path  #### Link to tracking issue Fixes open-telemetry#35990

povilasv added bug Something isn't working needs triage New item requiring triage labels Oct 25, 2024

github-actions bot added the receiver/hostmetrics label Oct 25, 2024

povilasv mentioned this issue Oct 25, 2024

[receiver/hostmetrics] fix filesystem scraper root_path support #35991

Closed

atoulme mentioned this issue Oct 25, 2024

[receiver/hostmetrics] Fix filesystems scraper behavior when root file system is mounted #36000

Merged

atoulme removed the needs triage New item requiring triage label Oct 28, 2024

github-actions bot mentioned this issue Oct 29, 2024

Weekly Report: 2024-10-22 - 2024-10-29 #36039

Closed

andrzej-stencel closed this as completed in #36000 Oct 30, 2024

andrzej-stencel closed this as completed in 1d864b6 Oct 30, 2024

rogercoll mentioned this issue Oct 31, 2024

fix: add filesystem mountinfo workaround elastic/opentelemetry#43

Merged

TylerHelmuth mentioned this issue Nov 4, 2024

Upgrade operator version to 112 open-telemetry/opentelemetry-helm-charts#1406

Closed

tosuke mentioned this issue Nov 6, 2024

fix(otelcol): downgrade hostmetrics receiver to v0.111.0 tosuke-homelab/containers#120

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[receiver/hostmetrics] filesystem scraper doesn't respect root_path #35990

[receiver/hostmetrics] filesystem scraper doesn't respect root_path #35990

povilasv commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024

TylerHelmuth commented Oct 25, 2024

povilasv commented Oct 25, 2024

marcelaraujo commented Oct 25, 2024

povilasv commented Oct 25, 2024

marcelaraujo commented Oct 25, 2024 •

edited

Loading

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

marcelaraujo commented Oct 25, 2024

[receiver/hostmetrics] filesystem scraper doesn't respect root_path #35990

[receiver/hostmetrics] filesystem scraper doesn't respect root_path #35990

Comments

povilasv commented Oct 25, 2024 • edited Loading

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Oct 25, 2024

TylerHelmuth commented Oct 25, 2024

povilasv commented Oct 25, 2024

marcelaraujo commented Oct 25, 2024

povilasv commented Oct 25, 2024

marcelaraujo commented Oct 25, 2024 • edited Loading

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

atoulme commented Oct 25, 2024

marcelaraujo commented Oct 25, 2024

povilasv commented Oct 25, 2024 •

edited

Loading

marcelaraujo commented Oct 25, 2024 •

edited

Loading