Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter metrics by label keys and values #7822

Merged
merged 15 commits into from
Oct 28, 2020
Merged

Conversation

ChristineTChen
Copy link
Contributor

@ChristineTChen ChristineTChen commented Oct 20, 2020

What does this PR do?

This PR introduces a config option that allows users to ignore metrics with certain labels or label values.

in conf.yaml

init_config: {}
instances:
  prometheus_endpoint: http://localhost:8086/-/metrics
  ignore_metrics_by_labels:
  pid:
  - puma_master
  worker: *

The above will ignore metrics with labels pid:puma_master or any metric with label worker:<ANY>

Motivation

FR

Additional Notes

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • PR title must be written as a CHANGELOG entry (see why)
  • Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
  • PR must have changelog/ and integration/ labels attached

Copy link
Member

@mgarabed mgarabed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be a nice improvement!

mgarabed
mgarabed previously approved these changes Oct 23, 2020
xornivore
xornivore previously approved these changes Oct 27, 2020
Copy link
Contributor

@xornivore xornivore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a small comment about log with warning.

sample_labels = sample[self.SAMPLE_LABELS]
for label_key, label_values in ignore_metrics_by_label.items():
if not label_values:
self.log.warning(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious if this might be a bit noisy? maybe debug is better?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that sounds good, updated

@ChristineTChen ChristineTChen dismissed stale reviews from xornivore and mgarabed via c313716 October 27, 2020 18:48
xornivore
xornivore previously approved these changes Oct 27, 2020
@ChristineTChen ChristineTChen merged commit e9343e4 into master Oct 28, 2020
@ChristineTChen ChristineTChen deleted the cc/metric-filter branch October 28, 2020 17:19
github-actions bot pushed a commit that referenced this pull request Oct 28, 2020
* Add config option

* ignore samples with matching label exclusion

* filter all samples

* Fix label key check

* Fix style

* Use plural in option name

* Allow for wildcards

* Add test for wildcard

* Test invalid option value

* Update log for invalid label values to warning

* Use wildcard in list

* Test exact match

* Style

* Drop warning to debug

* Fix test e9343e4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants