-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Metricbeat][elasticsearch] Use replica number in doc ID for shard metricset #33457
[Metricbeat][elasticsearch] Use replica number in doc ID for shard metricset #33457
Conversation
/test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@crespocarlos It's a legit lint error, I was also confused but you can see the errors in the code diff (or at least I can), and it's because of rules added but they only do something like "lint-staged" so this is the first time this file is changed since then so I get to fix them yay |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! 🚀
…tricset (#33457) (#33695) * [Metricbeat][elasticsearch] Use replica number in doc ID for shard metricset * "Fix" lint warnings * Fix lint errors (cherry picked from commit a5ffacc) Co-authored-by: Milton Hultgren <[email protected]>
## Summary This PR depends on Metricbeat changes introduced in this PR elastic/beats#33457 or the Internal collection changes introduced in elastic/elasticsearch#91153. The above PR fixes a bug that makes Metricbeat properly report on all the shards in an index, but our shard legend had the same kind of bug, it only displayed the first replica because it didn't account for the replica "id" in the function that tries to deduplicate the results. Since the above PR ensures we report each shard with a unique document ID, we change the Kibana code to use the document ID instead of trying to regenerate a unique ID. ### How to test Get the changes: `git fetch [email protected]:miltonhultgren/kibana.git sm-shard-allocations:sm-shard-allocations && git switch sm-shard-allocations` Start Elasticsearch, create an index like this: ``` PUT /some-index { "settings": { "index": { "number_of_shards": 3, "number_of_replicas": 3 } } } ``` Run Metricbeat with `xpack.enabled: true`. Visit the Index details page for `some-index` in the Stack Monitoring app and verify that on a single node cluster you have 12 shards, 9 of which are unassigned. <img width="753" alt="Screenshot 2022-10-25 at 17 19 25" src="https://user-images.githubusercontent.com/2564140/197819368-8ea45e1c-7472-4e15-9267-3b5d73378f2a.png"> Co-authored-by: Kibana Machine <[email protected]>
…ic#143963) ## Summary This PR depends on Metricbeat changes introduced in this PR elastic/beats#33457 or the Internal collection changes introduced in elastic/elasticsearch#91153. The above PR fixes a bug that makes Metricbeat properly report on all the shards in an index, but our shard legend had the same kind of bug, it only displayed the first replica because it didn't account for the replica "id" in the function that tries to deduplicate the results. Since the above PR ensures we report each shard with a unique document ID, we change the Kibana code to use the document ID instead of trying to regenerate a unique ID. ### How to test Get the changes: `git fetch [email protected]:miltonhultgren/kibana.git sm-shard-allocations:sm-shard-allocations && git switch sm-shard-allocations` Start Elasticsearch, create an index like this: ``` PUT /some-index { "settings": { "index": { "number_of_shards": 3, "number_of_replicas": 3 } } } ``` Run Metricbeat with `xpack.enabled: true`. Visit the Index details page for `some-index` in the Stack Monitoring app and verify that on a single node cluster you have 12 shards, 9 of which are unassigned. <img width="753" alt="Screenshot 2022-10-25 at 17 19 25" src="https://user-images.githubusercontent.com/2564140/197819368-8ea45e1c-7472-4e15-9267-3b5d73378f2a.png"> Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit b7debc8)
…143963) (#145520) # Backport This will backport the following commits from `main` to `8.6`: - [[Stack Monitoring] Use doc ID to deduplicate shard allocations (#143963)](#143963) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Milton Hultgren","email":"[email protected]"},"sourceCommit":{"committedDate":"2022-11-17T09:44:58Z","message":"[Stack Monitoring] Use doc ID to deduplicate shard allocations (#143963)\n\n## Summary\r\n\r\nThis PR depends on Metricbeat changes introduced in this PR\r\nhttps://github.com/elastic/beats/pull/33457 or the Internal collection\r\nchanges introduced in\r\nhttps://github.com/elastic/elasticsearch/pull/91153.\r\n\r\nThe above PR fixes a bug that makes Metricbeat properly report on all\r\nthe shards in an index, but our shard legend had the same kind of bug,\r\nit only displayed the first replica because it didn't account for the\r\nreplica \"id\" in the function that tries to deduplicate the results.\r\n\r\nSince the above PR ensures we report each shard with a unique document\r\nID, we change the Kibana code to use the document ID instead of trying\r\nto regenerate a unique ID.\r\n\r\n### How to test\r\nGet the changes: `git fetch [email protected]:miltonhultgren/kibana.git\r\nsm-shard-allocations:sm-shard-allocations && git switch\r\nsm-shard-allocations`\r\n\r\nStart Elasticsearch, create an index like this:\r\n```\r\nPUT /some-index\r\n{\r\n \"settings\": {\r\n \"index\": {\r\n \"number_of_shards\": 3, \r\n \"number_of_replicas\": 3\r\n }\r\n }\r\n}\r\n```\r\n\r\nRun Metricbeat with `xpack.enabled: true`. \r\n\r\nVisit the Index details page for `some-index` in the Stack Monitoring\r\napp and verify that on a single node cluster you have 12 shards, 9 of\r\nwhich are unassigned.\r\n\r\n<img width=\"753\" alt=\"Screenshot 2022-10-25 at 17 19 25\"\r\nsrc=\"https://user-images.githubusercontent.com/2564140/197819368-8ea45e1c-7472-4e15-9267-3b5d73378f2a.png\">\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"b7debc839ff6923923fe9a41355bd0318e04cdca","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Infra Monitoring UI","release_note:skip","Feature:Stack Monitoring","backport:prev-minor","v8.7.0"],"number":143963,"url":"https://github.com/elastic/kibana/pull/143963","mergeCommit":{"message":"[Stack Monitoring] Use doc ID to deduplicate shard allocations (#143963)\n\n## Summary\r\n\r\nThis PR depends on Metricbeat changes introduced in this PR\r\nhttps://github.com/elastic/beats/pull/33457 or the Internal collection\r\nchanges introduced in\r\nhttps://github.com/elastic/elasticsearch/pull/91153.\r\n\r\nThe above PR fixes a bug that makes Metricbeat properly report on all\r\nthe shards in an index, but our shard legend had the same kind of bug,\r\nit only displayed the first replica because it didn't account for the\r\nreplica \"id\" in the function that tries to deduplicate the results.\r\n\r\nSince the above PR ensures we report each shard with a unique document\r\nID, we change the Kibana code to use the document ID instead of trying\r\nto regenerate a unique ID.\r\n\r\n### How to test\r\nGet the changes: `git fetch [email protected]:miltonhultgren/kibana.git\r\nsm-shard-allocations:sm-shard-allocations && git switch\r\nsm-shard-allocations`\r\n\r\nStart Elasticsearch, create an index like this:\r\n```\r\nPUT /some-index\r\n{\r\n \"settings\": {\r\n \"index\": {\r\n \"number_of_shards\": 3, \r\n \"number_of_replicas\": 3\r\n }\r\n }\r\n}\r\n```\r\n\r\nRun Metricbeat with `xpack.enabled: true`. \r\n\r\nVisit the Index details page for `some-index` in the Stack Monitoring\r\napp and verify that on a single node cluster you have 12 shards, 9 of\r\nwhich are unassigned.\r\n\r\n<img width=\"753\" alt=\"Screenshot 2022-10-25 at 17 19 25\"\r\nsrc=\"https://user-images.githubusercontent.com/2564140/197819368-8ea45e1c-7472-4e15-9267-3b5d73378f2a.png\">\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"b7debc839ff6923923fe9a41355bd0318e04cdca"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/143963","number":143963,"mergeCommit":{"message":"[Stack Monitoring] Use doc ID to deduplicate shard allocations (#143963)\n\n## Summary\r\n\r\nThis PR depends on Metricbeat changes introduced in this PR\r\nhttps://github.com/elastic/beats/pull/33457 or the Internal collection\r\nchanges introduced in\r\nhttps://github.com/elastic/elasticsearch/pull/91153.\r\n\r\nThe above PR fixes a bug that makes Metricbeat properly report on all\r\nthe shards in an index, but our shard legend had the same kind of bug,\r\nit only displayed the first replica because it didn't account for the\r\nreplica \"id\" in the function that tries to deduplicate the results.\r\n\r\nSince the above PR ensures we report each shard with a unique document\r\nID, we change the Kibana code to use the document ID instead of trying\r\nto regenerate a unique ID.\r\n\r\n### How to test\r\nGet the changes: `git fetch [email protected]:miltonhultgren/kibana.git\r\nsm-shard-allocations:sm-shard-allocations && git switch\r\nsm-shard-allocations`\r\n\r\nStart Elasticsearch, create an index like this:\r\n```\r\nPUT /some-index\r\n{\r\n \"settings\": {\r\n \"index\": {\r\n \"number_of_shards\": 3, \r\n \"number_of_replicas\": 3\r\n }\r\n }\r\n}\r\n```\r\n\r\nRun Metricbeat with `xpack.enabled: true`. \r\n\r\nVisit the Index details page for `some-index` in the Stack Monitoring\r\napp and verify that on a single node cluster you have 12 shards, 9 of\r\nwhich are unassigned.\r\n\r\n<img width=\"753\" alt=\"Screenshot 2022-10-25 at 17 19 25\"\r\nsrc=\"https://user-images.githubusercontent.com/2564140/197819368-8ea45e1c-7472-4e15-9267-3b5d73378f2a.png\">\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"b7debc839ff6923923fe9a41355bd0318e04cdca"}}]}] BACKPORT--> Co-authored-by: Milton Hultgren <[email protected]>
…tricset (#33457) * [Metricbeat][elasticsearch] Use replica number in doc ID for shard metricset * "Fix" lint warnings * Fix lint errors
Summary
In the
shard
metricset, we want to report one document per shard allocation status. If you have an index with 3 shards (primaries) and 3 replicas (per primary, 9 total) you should have 12 documents that describe if the shards are assigned or not.To achieve this we query the routing_table of the cluster state at
/_cluster/state/version,nodes,master_node,routing_table
and loop over each index, where each shard is an array of the primary and all the replicas for that primary:We create a custom document ID for each shard allocation with the function
generateHashForEvent
but inmain
this function doesn't account for the "id" of the replica so it only reports the first replica (ignoring the other 2 in this example).This PR adds the array iterator to the ID if it's a replica, making sure that we generate a document for each replica of the shard.
How to test
Get the changes:
git fetch [email protected]:miltonhultgren/beats.git sm-shard-allocations:sm-shard-allocations && git switch sm-shard-allocations
Start Elasticsearch, create an index like this:
Run Metricbeat with
xpack.enabled: true
.Run the following query to verify that you get 12 documents back, 3 for the assigned primaries and 9 for the unassigned replicas:
Such as: