when the network interfaces come and go metric cardinality grows unbounded

[SergeyKanzhelev]

Calico creates network interfaces for each pod with the unique name. Collecting net metrics (like kubernetes.io/internal/node/guest/net/tx_packets) uses the interface name as a dimension interface_name. OpenCensus go library has no bounds for the metric cardinality and doesn't allow to clean up old values.

Need to find a best way to reset label values for net interfaces that don't exist any longer.

[mmiranda96]

For cases such as Calico, do we want to keep metrics for each individual interface? If not, a viable alternative would be to group these interfaces within the same dimension. We can use a config model like this:

"net": { 
  "interfaceGroupings": {
    [
      "name": "calico",
      "groupRegexp": "^cali[0-9a-f]+$"
    ]
  }
}

In the implementation, we would aggregate all interfaces that match the regex under the same name. Interfaces that don't match any grouping will be reported independently.

[mmiranda96]

/assign mmiranda96

[SergeyKanzhelev]

will the aggregated metric have any value for us? Instead of an aggregate we probably can simply disable the metric collection completely

[SergeyKanzhelev]

I'm not sure how useful this metric is. If we want to keep it - the best long term solution would be to switch to OpenTelemetry delta metrics and use this exporter: https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/tree/main/exporter/metric.

@dashpole do you think OpenTelemetry is in a good state to adopt it here for metrics?

[dashpole]

The Metrics SDK is currently being (entirely) rewritten in preparation for the first stable release, so I wouldn't recommend it right now. That effort is tracked in https://github.com/orgs/open-telemetry/projects/22. After that is published, we will adapt the GCM exporter and publish a stable release of that as well. After that, I would recommend switching.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mmiranda96]

This issue has been fixed via #675. This fix has been released in https://github.com/kubernetes/node-problem-detector/releases/tag/v0.8.11.

/remove-lifecycle stale
/close

[k8s-ci-robot]

@mmiranda96: Closing this issue.

In response to this:

This issue has been fixed via #675. This fix has been released in https://github.com/kubernetes/node-problem-detector/releases/tag/v0.8.11.

/remove-lifecycle stale
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.