Skip to content

Commit

Permalink
[Stack Monitoring] Use doc ID to deduplicate shard allocations (elast…
Browse files Browse the repository at this point in the history
…ic#143963)

## Summary

This PR depends on Metricbeat changes introduced in this PR
elastic/beats#33457 or the Internal collection
changes introduced in
elastic/elasticsearch#91153.

The above PR fixes a bug that makes Metricbeat properly report on all
the shards in an index, but our shard legend had the same kind of bug,
it only displayed the first replica because it didn't account for the
replica "id" in the function that tries to deduplicate the results.

Since the above PR ensures we report each shard with a unique document
ID, we change the Kibana code to use the document ID instead of trying
to regenerate a unique ID.

### How to test
Get the changes: `git fetch [email protected]:miltonhultgren/kibana.git
sm-shard-allocations:sm-shard-allocations && git switch
sm-shard-allocations`

Start Elasticsearch, create an index like this:
```
PUT /some-index
{
  "settings": {
    "index": {
      "number_of_shards": 3,  
      "number_of_replicas": 3
    }
  }
}
```

Run Metricbeat with `xpack.enabled: true`. 

Visit the Index details page for `some-index` in the Stack Monitoring
app and verify that on a single node cluster you have 12 shards, 9 of
which are unassigned.

<img width="753" alt="Screenshot 2022-10-25 at 17 19 25"
src="https://user-images.githubusercontent.com/2564140/197819368-8ea45e1c-7472-4e15-9267-3b5d73378f2a.png">

Co-authored-by: Kibana Machine <[email protected]>
  • Loading branch information
miltonhultgren and kibanamachine authored Nov 17, 2022
1 parent 4608af4 commit b7debc8
Show file tree
Hide file tree
Showing 4 changed files with 8 additions and 4 deletions.
1 change: 1 addition & 0 deletions x-pack/plugins/monitoring/common/types/es.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ export interface ElasticsearchResponse {
}

export interface ElasticsearchResponseHit {
_id: string;
_index: string;
_source: ElasticsearchSource;
inner_hits?: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,11 @@ describe('get_shard_allocation', () => {
describe('handleResponse', () => {
it('deduplicates shards', () => {
const nextTimestamp = '2018-07-06T00:00:01.259Z';
const hits = shards.map((shard) => {
const hits = shards.map((shard, index) => {
return {
_id: `STATE_UUID:${shard.node}:${shard.index}:s${shard.shard}:${
shard.primary ? 'p' : `r${index}`
}`,
_source: {
...exampleShardSource,
shard,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,8 @@ export function handleResponse(response: ElasticsearchResponse) {
mbShard?.shard?.relocating_node?.id ?? legacyShard?.relocating_node ?? null;
const node = mbShard?.node?.id ?? legacyShard?.node;
// note: if the request is for a node, then it's enough to deduplicate without primary, but for indices it displays both
const shardId = `${index}-${shardNumber}-${primary}-${relocatingNode}-${node}`;

if (!uniqueShards.has(shardId)) {
if (!uniqueShards.has(hit._id)) {
// @ts-ignore
shards.push({
index,
Expand All @@ -48,7 +47,7 @@ export function handleResponse(response: ElasticsearchResponse) {
shard: shardNumber,
state: legacyShard?.state ?? mbShard?.shard?.state,
});
uniqueShards.add(shardId);
uniqueShards.add(hit._id);
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ jest.mock('../../static_globals', () => ({
// deletes, adds, or updates the properties based on a default object
function createResponseObjHit(params?: HitParams[]): ElasticsearchResponseHit {
const defaultResponseObj: ElasticsearchResponseHit = {
_id: '123123a',
_index: 'index',
_source: {
cluster_uuid: '123',
Expand Down

0 comments on commit b7debc8

Please sign in to comment.