Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple duration fields for better elasticseach storage for controllermanager #32486

Conversation

MichaelKatsoulis
Copy link
Contributor

@MichaelKatsoulis MichaelKatsoulis commented Jul 25, 2022

What does this PR do?

Why is it important?

Due to the new fields introduced in controllermanager metricset in #32037

  1. kubernetes.controllermanager.client.request.duration.sec.bucket.*
  2. kubernetes.controllermanager.workqueue.work.duration.sec.bucket.*

there was a bug created.
Because of the histogram buckets of client.request.duration.sec the following fields where created.

kubernetes.controllermanager.client.request.duration.sec.bucket.0.128
kubernetes.controllermanager.client.request.duration.sec.bucket.0.256
...

The problem is that the field 0.128 is stored in elasticsearch encapsulated like

Screenshot 2022-07-25 at 5 35 41 PM

which then leads to the elastic package marking the client.request.duration.sec.bucket.0.128 as undefined.

FAILURE DETAILS:
kubernetes/controllermanager default:
[0] field "kubernetes.controllermanager.client.request.duration.sec.bucket.0.128" is undefined

For this reason the buckets are multiplied by 10^6 to be converted from seconds to microseconds , so that the field will be an integer.

Also the field kubernetes.controllermanager.workqueue.work.duration.sec.bucket.* will be deleted and reconsidered in the next release as the fields created like kubernetes.controllermanager.workqueue.work.duration.sec.bucket.0.000009999999999999999 cannot be meaningfully multiplied yet to become integers.

Dashboard is also updated.

@MichaelKatsoulis MichaelKatsoulis added bug Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team labels Jul 25, 2022
@MichaelKatsoulis MichaelKatsoulis requested a review from a team as a code owner July 25, 2022 16:27
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 25, 2022
@MichaelKatsoulis MichaelKatsoulis marked this pull request as draft July 25, 2022 16:28
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 25, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-07-26T07:45:55.849+0000

  • Duration: 59 min 45 sec

Test stats 🧪

Test Results
Failed 0
Passed 3547
Skipped 873
Total 4420

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@MichaelKatsoulis MichaelKatsoulis marked this pull request as ready for review July 26, 2022 07:45
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just make sure that the k8s package is also tuned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants