[Metricbeat] Enhancement to add support for handling Openmetrics data #26308

premendrasingh · 2021-06-15T04:35:57Z

Enhancement - Enhanced Prometheus module and added openmetrics metricset to use Openmetrics parser.

What does this PR do?

This PR adds a new metricset in Prometheus module. It handles metrics endpoints returning data with
Content-Type: application/openmetrics-text
Message processing is similar to existing prometheus/collector.

Why is it important?

Many of our customers have data exposed in OpenMetrics format, but existing Prometheus module can process metrics only in expfmt, Openmetrics data is being ignored. Hence the need for Openmetrics support also.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Unit tests in prometheus/openmetricscollector can be used to test the metrics locally.

Setup a http server to serve Openmetrics data with HTTP header Content-Type: application/openmetrics-text

Change metricset in metricbeat.yml to use openmetricscollector as shown below

metricbeat.modules:
- module: prometheus
  hosts: ["http://localhost:8080/metrics"]
  metricsets: ["openmetricscollector"]

…-support

elasticmachine · 2021-06-15T04:40:24Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #26308 updated
Reason: The PR is not allowed to run in the CI yet
Start Time: 2021-06-17T01:58:42.886+0000
Duration: 5 min 57 sec
Commit: 3988843

Trends 🧪

Steps errors

Expand to view the steps failures

`Load a resource file from a shared library`

Took 0 min 0 sec . View more details on here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details on here
Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)

Log output

Expand to view the last 100 lines of log output

[2021-06-17T02:01:40.873Z]  > git config core.sparsecheckout # timeout=10
[2021-06-17T02:01:40.877Z]  > git checkout -f 3988843c0246a5e1a79d8fcb5156a39ef254192c # timeout=15
[2021-06-17T02:01:43.474Z] Merge succeeded, producing f4316eb958661b55ee4e34acc5593d19657e6aaa
[2021-06-17T02:01:43.475Z] Checking out Revision f4316eb958661b55ee4e34acc5593d19657e6aaa (PR-26308)
[2021-06-17T02:01:42.475Z]  > git remote # timeout=10
[2021-06-17T02:01:42.480Z]  > git config --get remote.origin.url # timeout=10
[2021-06-17T02:01:42.483Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-17T02:01:42.487Z]  > git merge 4684ac7ed3da20ecc533af492862cdb029f5cb3a # timeout=10
[2021-06-17T02:01:43.469Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-06-17T02:01:43.477Z]  > git config core.sparsecheckout # timeout=10
[2021-06-17T02:01:43.481Z]  > git checkout -f f4316eb958661b55ee4e34acc5593d19657e6aaa # timeout=15
[2021-06-17T02:01:47.061Z] Commit message: "Merge commit '4684ac7ed3da20ecc533af492862cdb029f5cb3a' into HEAD"
[2021-06-17T02:01:47.064Z]  > git rev-list --no-walk e82b1a7d38b877284c3814811211fdacf6156e7c # timeout=10
[2021-06-17T02:01:47.094Z] Cleaning workspace
[2021-06-17T02:01:48.103Z] Timeout set to expire in 3 hr 0 min
[2021-06-17T02:01:48.114Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-06-17T02:01:48.352Z] [INFO] Number of builds to be searched 10
[2021-06-17T02:01:49.106Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-06-17T02:01:49.117Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26308@tmp
[2021-06-17T02:01:49.130Z] [INFO] gitCheckout: Checkout SCM PR-26308 with default customisation from the Item.
[2021-06-17T02:01:49.151Z] [INFO] Override default checkout
[2021-06-17T02:01:49.182Z] Sleeping for 10 sec
[2021-06-17T02:01:47.096Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-17T02:01:47.099Z] Resetting working tree
[2021-06-17T02:01:47.099Z]  > git reset --hard # timeout=10
[2021-06-17T02:01:47.181Z]  > git clean -fdx # timeout=10
[2021-06-17T02:01:59.206Z] The recommended git tool is: git
[2021-06-17T02:01:59.254Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-06-17T02:01:59.258Z] Wiping out workspace first.
[2021-06-17T02:01:59.267Z] Cloning the remote Git repository
[2021-06-17T02:01:59.267Z] Using shallow clone with depth 10
[2021-06-17T02:01:59.267Z] Avoid fetching tags
[2021-06-17T02:01:59.280Z] Cloning repository [email protected]:elastic/beats.git
[2021-06-17T02:01:59.306Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-26308@tmp # timeout=10
[2021-06-17T02:01:59.313Z] Fetching upstream changes from [email protected]:elastic/beats.git
[2021-06-17T02:01:59.313Z]  > git --version # timeout=10
[2021-06-17T02:01:59.316Z]  > git --version # 'git version 2.17.1'
[2021-06-17T02:01:59.317Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-17T02:01:59.321Z]  > git fetch --no-tags --progress -- [email protected]:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-06-17T02:02:20.430Z] Cleaning workspace
[2021-06-17T02:02:20.444Z] Using shallow fetch with depth 10
[2021-06-17T02:02:20.444Z] Pruning obsolete local branches
[2021-06-17T02:02:20.414Z]  > git config remote.origin.url [email protected]:elastic/beats.git # timeout=10
[2021-06-17T02:02:20.419Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-06-17T02:02:20.426Z]  > git config remote.origin.url [email protected]:elastic/beats.git # timeout=10
[2021-06-17T02:02:20.432Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-17T02:02:20.435Z] No valid HEAD. Skipping the resetting
[2021-06-17T02:02:20.435Z]  > git clean -fdx # timeout=10
[2021-06-17T02:02:20.446Z] Fetching upstream changes from [email protected]:elastic/beats.git
[2021-06-17T02:02:20.447Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-17T02:02:20.450Z]  > git fetch --no-tags --progress --prune -- [email protected]:elastic/beats.git +refs/pull/26308/head:refs/remotes/origin/PR-26308 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-06-17T02:02:21.416Z] Merging remotes/origin/master commit 4684ac7ed3da20ecc533af492862cdb029f5cb3a into PR head commit 3988843c0246a5e1a79d8fcb5156a39ef254192c
[2021-06-17T02:02:21.418Z]  > git config core.sparsecheckout # timeout=10
[2021-06-17T02:02:21.421Z]  > git checkout -f 3988843c0246a5e1a79d8fcb5156a39ef254192c # timeout=15
[2021-06-17T02:02:24.701Z] Merge succeeded, producing 2b81b91ceb867e789ac0ad00b8c2286976f216e1
[2021-06-17T02:02:24.701Z] Checking out Revision 2b81b91ceb867e789ac0ad00b8c2286976f216e1 (PR-26308)
[2021-06-17T02:02:24.789Z] Commit message: "Merge commit '4684ac7ed3da20ecc533af492862cdb029f5cb3a' into HEAD"
[2021-06-17T02:02:22.968Z]  > git remote # timeout=10
[2021-06-17T02:02:22.972Z]  > git config --get remote.origin.url # timeout=10
[2021-06-17T02:02:22.978Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-17T02:02:22.982Z]  > git merge 4684ac7ed3da20ecc533af492862cdb029f5cb3a # timeout=10
[2021-06-17T02:02:24.696Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-06-17T02:02:24.703Z]  > git config core.sparsecheckout # timeout=10
[2021-06-17T02:02:24.706Z]  > git checkout -f 2b81b91ceb867e789ac0ad00b8c2286976f216e1 # timeout=15
[2021-06-17T02:02:24.792Z]  > git rev-list --no-walk e82b1a7d38b877284c3814811211fdacf6156e7c # timeout=10
[2021-06-17T02:02:24.812Z] Cleaning workspace
[2021-06-17T02:02:24.814Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-17T02:02:24.817Z] Resetting working tree
[2021-06-17T02:02:24.818Z]  > git reset --hard # timeout=10
[2021-06-17T02:02:24.903Z]  > git clean -fdx # timeout=10
[2021-06-17T02:02:26.065Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-06-17T02:02:26.563Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-06-17T02:03:34.337Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26308@tmp/.git
[2021-06-17T02:03:34.449Z] Archiving artifacts
[2021-06-17T02:03:35.081Z] + git rev-parse HEAD
[2021-06-17T02:03:35.397Z] + git rev-parse HEAD
[2021-06-17T02:03:35.695Z] + git rev-parse origin/pr/26308
[2021-06-17T02:03:35.732Z] [INFO] githubEnv: Found Git Build Cause: pr
[2021-06-17T02:03:36.108Z] Masking supported pattern matches of $GITHUB_TOKEN
[2021-06-17T02:03:37.095Z] [WARN] githubApiCall: The REST API call https://api.github.com/repos/elastic/beats/pulls/26308/reviews return 0 elements
[2021-06-17T02:03:37.123Z] [INFO] githubPrCheckApproved: Title: [Metricbeat] Enhancement to add support for handling Openmetrics data - User: premendrasingh - Author Association: CONTRIBUTOR
[2021-06-17T02:03:37.345Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet
[2021-06-17T02:03:37.345Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)
[2021-06-17T02:03:37.374Z] [INFO] Let's stop build #3. The PR is not allowed to run in the CI yet
[2021-06-17T02:03:37.385Z] Sleeping for 5 sec
[2021-06-17T02:03:38.330Z] Stage "Lint" skipped due to earlier failure(s)
[2021-06-17T02:03:38.356Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-06-17T02:03:38.382Z] Stage "Extended" skipped due to earlier failure(s)
[2021-06-17T02:03:38.409Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-17T02:03:38.435Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-17T02:03:38.479Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26308/src/github.com/elastic/beats
[2021-06-17T02:03:39.140Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26308
[2021-06-17T02:03:39.347Z] [INFO] getVaultSecret: Getting secrets
[2021-06-17T02:03:39.384Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-17T02:03:40.126Z] + chmod 755 generate-build-data.sh
[2021-06-17T02:03:40.126Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26308/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26308/runs/3 ABORTED 296978
[2021-06-17T02:03:40.126Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26308/runs/3/steps/?limit=10000 -o steps-info.json
[2021-06-17T02:03:40.377Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26308/runs/3/tests/?status=FAILED -o tests-errors.json
[2021-06-17T02:03:40.377Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-06-17T02:03:41.288Z] Retry 2/3 exited 22, retrying in 2 seconds...

elasticmachine · 2021-06-15T10:39:57Z

Pinging @elastic/integrations (Team:Integrations)

jsoriano · 2021-06-17T08:51:44Z

Hey @premendrasingh, thanks for opening this PR!

I see it adds a new metricset for openmetrics, but Metricbeat already has an openmetrics module. Have you considered improving the existing one to better support this data?

I also see that this PR includes changes for other things as azure or oracle, are these changes intended? If they are, I would suggest to open different PRs for them, as they seem to be unrelated.

premendrasingh · 2021-06-18T04:18:25Z

@jsoriano make update was failing for Metricbeat, that's why I had to change Azure. and Oracle related files.

mergify · 2021-06-29T16:12:47Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b openmetrics-support upstream/openmetrics-support
git merge upstream/master
git push upstream openmetrics-support

premendrasingh · 2021-08-06T05:44:37Z

Cancel PR

premendrasingh added 3 commits June 14, 2021 15:32

Initial drop for Openmetrics support

b19d7eb

Merge remote-tracking branch 'beats_upstream/master' into openmetrics…

59ac2b4

…-support

goimports reformatting

e5388cb

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2021

premendrasingh added 3 commits June 14, 2021 21:43

mage fmt reformatting

20c28d8

Remove commented code

e82b1a7

Add testcase

6790c26

ChrsMark added the Team:Integrations Label for the Integrations team label Jun 15, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2021

ChrsMark self-assigned this Jun 15, 2021

ChrsMark requested review from ChrsMark and jsoriano June 15, 2021 10:40

Remove commented code and clean up

3988843

ChrsMark mentioned this pull request Jun 17, 2021

Verify new Prometheus exposition formats are supported #24707

Closed

premendrasingh closed this Aug 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Enhancement to add support for handling Openmetrics data #26308

[Metricbeat] Enhancement to add support for handling Openmetrics data #26308

premendrasingh commented Jun 15, 2021

elasticmachine commented Jun 15, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Trends 🧪

`Load a resource file from a shared library`

`Error signal`

elasticmachine commented Jun 15, 2021

jsoriano commented Jun 17, 2021

premendrasingh commented Jun 18, 2021

mergify bot commented Jun 29, 2021

premendrasingh commented Aug 6, 2021

[Metricbeat] Enhancement to add support for handling Openmetrics data #26308

[Metricbeat] Enhancement to add support for handling Openmetrics data #26308

Conversation

premendrasingh commented Jun 15, 2021

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

elasticmachine commented Jun 15, 2021 • edited by jenkins-beats-ci bot Loading

❕ Build Aborted

Build stats

Trends 🧪

Steps errors

Load a resource file from a shared library

Error signal

Log output

elasticmachine commented Jun 15, 2021

jsoriano commented Jun 17, 2021

premendrasingh commented Jun 18, 2021

mergify bot commented Jun 29, 2021

premendrasingh commented Aug 6, 2021

elasticmachine commented Jun 15, 2021 •

edited by jenkins-beats-ci bot

Loading

`Load a resource file from a shared library`

`Error signal`