[attributesprocessor] The resource part of metrics include filtering config doesn't work

[chenlujjj]

Component(s)

processor/attributes

What happened?

Description

As you can see in the attached collector config file, I tried to use the attributes processor to add labels for the metric node_disk_write_time_seconds_total only when device=disk0

NOTE: this is just a minimum example to reproduce the issue.

Steps to Reproduce

Expected Result

Actual Result

All the metrics node_disk_write_time_seconds_total even with device!=disk0 were updated as well.

Collector version

0.79.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: "node_exporter"
            scrape_interval: 10s
            static_configs:
              - targets: ['0.0.0.0:9100']

exporters:
    file/metrics:
      path: /tmp/metrics

processors:
    batch:
    attributes/1:
      include:
        resources:
          - key: device
            value: disk0
        metric_names: [node_disk_write_time_seconds_total]
        match_type: strict
      actions:
        - key: name
          action: upsert
          value: zhangsan

extensions:
    health_check:
      endpoint: 0.0.0.0:13133
    pprof:
      endpoint: :1888
    zpages:
      endpoint: :55679

service:
    extensions: [pprof, zpages, health_check]
    telemetry:
      metrics:
        address: ":8888"
    pipelines:
      metrics:
        receivers: [prometheus]
        processors: [batch, attributes/1]
        exporters: [logging, file/metrics]

Log output

No response

Additional context

In the source code:

• https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.79.0/internal/filter/filtermetric/filtermetric.go#L49
• https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.79.0/internal/filter/filtermetric/name_matcher.go#L18

The ResourceAttributes of MatchProperties doesn't take effect at all. Only the metric names are used to filter.

[github-actions]

Pinging code owners:

• processor/attributes: @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[chenlujjj]

The readme doc says: "For metrics, one of metric_names or resources must be specified with a valid non-empty value for a valid configuration." So I think the resources can be used to filter as well.

[chenlujjj]

And when I just specified the resources like:

attributes/1:
      include:
        resources:
          - key: device
            value: disk0
     actions:
        - key: name
          action: upsert
          value: zhangsan

It will update labels for all metrics ...
The cause is at this line: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.79.0/internal/filter/filtermetric/filtermetric.go#L47

[crobert-1]

To clarify your minimum reproducible config file, you're saying that with this config all metrics named node_disk_write_time_seconds_total are coming through, even if their device resource isn't disk0, is that correct?

If that's the case, could you post some example metrics from your logging output?

[chenlujjj]

hi, @crobert-1, please see the attached file for the full list of node exporter metrics.
node_metrics.json

[crobert-1]

Okay, I'm seeing two main issues here.

1. The filter expression only accounts for the metric name from what I'm seeing, it drops the resource attribute filter from the check. This is what is causing part of your issue. Adding a reference to NewResourceSkipExprBridge to properly account for resource attribute filters would resolve this.
2. The attributes you're looking at are datapoint attributes, not resource attributes. However, this would still fail due to the bug referenced in my first point, attributes aren't properly accounted for either from what I can tell.

There is another more supported way to do this though, using the transform processor. If you use the datapoint context you can use the set function to add a key to your attributes map when specific conditions are met, as you're attempting to do here.

[github-actions]

Pinging code owners for processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[chenlujjj]

Hi @crobert-1 , thanks for your reply.
If I understand correctly, the device filter specified in the config above will not work even if the resource attribute filter is added to the attributes processor properly, because the device is a datapoint attribute, not resource attribute, is that right?

[chenlujjj]

And from the code here, the attributesprocessor processes the datapoint attribute, not resource attribute, too.

I'll give a try to transform processor, thanks.

[crobert-1]

If I understand correctly, the device filter specified in the config above will not work even if the resource attribute filter is added to the attributes processor properly, because the device is a datapoint attribute, not resource attribute, is that right?

That's correct, even if this bug is fixed the filter you've shared won't work with your data.

And from the code here, the attributesprocessor processes the datapoint attribute, not resource attribute, too.

Yes, the attributes processor only supports operations on datapoint attributes for metrics, it does not support operations on resource attributes.

[chenlujjj]

Thanks for your confirmation :)

[TylerHelmuth]

In the transformprocessor it would look like

transform:
  error_mode: ignore
  metric_statements:
    - context: datapoint
      statements:
        - set(name, "zhangsan") where metric.name == "node_disk_write_time_seconds_total" and resource.attributes["device"] == "disk0"

[TylerHelmuth]

@chenlujjj I am going to close this issue for now since that transformprocessor config solves the use case you wanted. Please ping me if it doesnt work and I'll reopen.

[chenlujjj]

Thank you @TylerHelmuth
After trial, I found the right config of transform processor for this case is:

transform:
      error_mode: ignore
      metric_statements:
        - context: datapoint
          statements:
            - set(attributes["name"], "zhangsan") where metric.name == "node_disk_write_time_seconds_total" and attributes["device"] == "disk0"

[TylerHelmuth]

You're right, good correction!