Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Monitoring] Ensure Stack Monitoring UI works with metricbeat-* indices #96205

Merged
merged 103 commits into from
Apr 23, 2021
Merged
Changes from 1 commit
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
f5db63c
WIP
chrisronline Feb 9, 2021
2f5486c
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Feb 9, 2021
1992078
WIP
chrisronline Feb 9, 2021
6390c67
Remove unnecessary fields
chrisronline Feb 10, 2021
efefc3d
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Feb 10, 2021
cf2dae0
Work on node detail page
chrisronline Feb 11, 2021
2d1028c
Cluster overview looking good
chrisronline Feb 11, 2021
df0b593
Index page
chrisronline Feb 12, 2021
788e1cb
Fix types
chrisronline Feb 15, 2021
fb0c825
ML jobs
chrisronline Feb 15, 2021
fea4ada
CCR
chrisronline Feb 16, 2021
f1d429e
CCR
chrisronline Feb 16, 2021
0e4254e
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Feb 18, 2021
c7401ce
We just need total here
chrisronline Feb 18, 2021
5a386a7
Standalone cluster fix
chrisronline Feb 18, 2021
c24fdea
Re-enable logstash
chrisronline Feb 18, 2021
5fdc42d
FIx jest test
chrisronline Feb 19, 2021
4d63764
Fix tests
chrisronline Feb 19, 2021
0e5ad89
Fix tests
chrisronline Feb 19, 2021
d089d96
Fix unused import
chrisronline Feb 21, 2021
088c541
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Feb 21, 2021
bde898b
Add new MB-based archives
chrisronline Feb 25, 2021
152eb3d
Add actual archives
chrisronline Feb 25, 2021
bbc7d59
Fix types
chrisronline Feb 25, 2021
b10c37b
Add this file back in
chrisronline Feb 25, 2021
d893f49
Fix tests and add more
chrisronline Feb 26, 2021
a45d0ac
Update whitelist
chrisronline Feb 26, 2021
6b2ec11
Renames
chrisronline Feb 26, 2021
5a3e365
Renames
chrisronline Feb 26, 2021
ee1c1ce
Only do ccs if enabled
chrisronline Mar 2, 2021
40bed6b
Updates
chrisronline Mar 2, 2021
4e748e6
Comment out
chrisronline Mar 2, 2021
ffca414
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Mar 8, 2021
a0853a7
More tests passing
chrisronline Mar 11, 2021
cdc214f
Another passing test
chrisronline Mar 11, 2021
ade0c00
More passing, yay
chrisronline Mar 12, 2021
2bc04f3
Forgot to add the actual tests, wow
chrisronline Mar 12, 2021
50a2a45
CCR
chrisronline Mar 12, 2021
9063880
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Mar 12, 2021
f7484f1
Fix CI issues
chrisronline Mar 15, 2021
e89520a
Missed a field here
chrisronline Mar 15, 2021
b599f86
Kibana tests passing
chrisronline Mar 17, 2021
3553244
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Mar 17, 2021
e25e0db
Fix type issues
chrisronline Mar 17, 2021
9c600e0
Fix type
chrisronline Mar 18, 2021
37efea5
Fix types
chrisronline Mar 18, 2021
fac80fd
Good chunk of logstash work
chrisronline Mar 19, 2021
b1a2757
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Mar 19, 2021
aef1476
Fix types
chrisronline Mar 20, 2021
7d35323
Fix jest test
chrisronline Mar 20, 2021
b19a021
Fix issue with get usage in logstash code
chrisronline Mar 20, 2021
e45c4ae
Fix small bug here
chrisronline Mar 20, 2021
ffc8a9f
Update archives with proper mappings
chrisronline Mar 21, 2021
0a70b97
Handle both legacy and mb fields properly
chrisronline Mar 22, 2021
24c5f56
Fixes
chrisronline Mar 22, 2021
9ebf3d2
Fix jest test
chrisronline Mar 22, 2021
5f3405e
Fix issue
chrisronline Mar 22, 2021
8165c2a
Fixes assignment v comparison and null type check in metrics detail page
jasonrhodes Mar 22, 2021
0426643
Getting setup tests in a better state
chrisronline Mar 23, 2021
82bc344
Only beats is failing now, which is good
chrisronline Mar 24, 2021
b169b29
Progress on cluster listing
chrisronline Mar 24, 2021
b251ce5
Merge in master
chrisronline Mar 24, 2021
7cc3149
Functional tests passing!
chrisronline Mar 25, 2021
435a803
More progress on cluster tests
chrisronline Mar 25, 2021
8a0d713
More cluster work
chrisronline Mar 26, 2021
4825586
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Mar 26, 2021
983806f
Fix test
chrisronline Mar 26, 2021
85035fd
Last recovery working
chrisronline Mar 28, 2021
b53fbb6
Fix types
chrisronline Mar 28, 2021
6067420
Fix tests
chrisronline Mar 29, 2021
f867de4
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Mar 29, 2021
8a6cf95
More tweaks
chrisronline Mar 29, 2021
3b9bcdb
Fix snapshot
chrisronline Mar 29, 2021
78e645d
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Mar 30, 2021
db0fe36
Merge branch 'master' of github.com:elastic/kibana
jasonrhodes Mar 31, 2021
70d2638
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Apr 2, 2021
80e946a
Use stats instead of kibana_stats
chrisronline Apr 2, 2021
c1d505d
Use node and node_stats for logstash
chrisronline Apr 5, 2021
c09c72a
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Apr 12, 2021
3aaacff
Merge branch 'master' of github.com:elastic/kibana
jasonrhodes Apr 12, 2021
902ca09
Beats tests passing
chrisronline Apr 13, 2021
244315e
APM tests passing
chrisronline Apr 13, 2021
66a37a5
API tests passing!
chrisronline Apr 14, 2021
d820d86
Fix types
chrisronline Apr 14, 2021
ae1f606
Fix tests
chrisronline Apr 15, 2021
3485490
Merge remote-tracking branch 'elastic/master' into monitoring/ecs_fin…
chrisronline Apr 15, 2021
00e05e5
Renames beats-with-restarted-instance archive dirs
jasonrhodes Apr 19, 2021
75735d3
Renames logstash-pipeline dirs
jasonrhodes Apr 19, 2021
bf8b4a4
Renames multi-basic dirs
jasonrhodes Apr 19, 2021
6ed9dcb
Renames singlecluster-* dirs
jasonrhodes Apr 19, 2021
3588ff6
Merge branch 'master' into monitoring/ecs_final_es
kibanamachine Apr 19, 2021
5e29d34
Fixes inaccurate path change for archive setup
jasonrhodes Apr 19, 2021
0655fce
Merge branch 'monitoring/ecs_final_es' of github.com:elastic/kibana i…
jasonrhodes Apr 19, 2021
8854364
Reverts changes to rebuild_all script
jasonrhodes Apr 21, 2021
f2df4ad
Merge branch 'master' into monitoring/ecs_final_es
kibanamachine Apr 21, 2021
7166642
Merge branch 'master' into monitoring/ecs_final_es
kibanamachine Apr 22, 2021
9351da3
Merge branch 'master', remote-tracking branch 'upstream' into monitor…
jasonrhodes Apr 22, 2021
73d7397
Updated values for gc func test expectations
jasonrhodes Apr 23, 2021
da36354
Merge branch 'monitoring/ecs_final_es' of github.com:elastic/kibana i…
jasonrhodes Apr 23, 2021
15d50e7
Reverts accidental changes to a few files from another PR
jasonrhodes Apr 23, 2021
8cf58fd
Merge branch 'master' into monitoring/ecs_final_es
kibanamachine Apr 23, 2021
73d4d9d
Fixes broken func test archive references
jasonrhodes Apr 23, 2021
9e15b90
Merge branch 'monitoring/ecs_final_es' of github.com:elastic/kibana i…
jasonrhodes Apr 23, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix types
chrisronline committed Feb 15, 2021
commit 788e1cb75858f3be2d4d779349af149b2241e99c
11 changes: 8 additions & 3 deletions x-pack/plugins/monitoring/common/types/es.ts
Original file line number Diff line number Diff line change
@@ -100,6 +100,7 @@ export interface ElasticsearchNodeStats {

export interface ElasticsearchIndexStats {
index?: string;
name?: string;
primaries?: {
docs?: {
count?: number;
@@ -315,10 +316,14 @@ export interface ElasticsearchMetricbeatSource {
'@timestamp'?: string;
elasticsearch?: {
node?: ElasticsearchLegacySource['source_node'] & ElasticsearchMetricbeatNode;
index?: ElasticsearchIndexStats & {
name?: string;
index?: ElasticsearchIndexStats;
version?: string;
shard?: ElasticsearchLegacySource['shard'] & {
number?: string;
relocating_node?: {
uuid?: string;
};
};
shard?: ElasticsearchLegacySource['shard'];
cluster?: {
name?: string;
id?: string;
38 changes: 38 additions & 0 deletions x-pack/plugins/monitoring/common/types/filebeat.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

export interface FilebeatResponse {
hits?: {
hits: FilebeatResponseHit[];
total: {
value: number;
};
};
aggregations?: any;
}

export interface FilebeatResponseHit {
_source: {
message?: string;
log?: {
level?: string;
};
'@timestamp': string;
event?: {
dataset?: string;
};
elasticsearch?: {
component?: string;
index?: {
name?: string;
};
node?: {
name?: string;
};
};
};
}
36 changes: 24 additions & 12 deletions x-pack/plugins/monitoring/server/lib/cluster/get_cluster_status.ts
Original file line number Diff line number Diff line change
@@ -14,11 +14,11 @@ import { ElasticsearchSource } from '../../../common/types/es';
* @return top-level cluster summary data
*/
export function getClusterStatus(cluster: ElasticsearchSource, shardStats: unknown) {
const clusterStats = cluster.elasticsearch?.cluster?.stats ?? cluster.cluster_stats ?? {};
const clusterNodes = clusterStats.nodes ?? {};
const clusterIndices = clusterStats.indices ?? {};
const clusterStatsLegacy = cluster.cluster_stats;
const clusterStatsMB = cluster.elasticsearch?.cluster?.stats;

const clusterTotalShards = clusterIndices.shards?.count ?? clusterIndices.shards?.total ?? 0;
const clusterTotalShards =
clusterStatsLegacy?.indices?.shards?.total ?? clusterStatsMB?.indices?.shards?.count ?? 0;
let unassignedShardsTotal = 0;
const unassignedShards = get(shardStats, 'indicesTotals.unassigned');
if (unassignedShards !== undefined) {
@@ -31,17 +31,29 @@ export function getClusterStatus(cluster: ElasticsearchSource, shardStats: unkno
status:
cluster.elasticsearch?.cluster?.stats?.status ?? cluster.cluster_state?.status ?? 'unknown',
// index-based stats
indicesCount: clusterIndices.total ?? clusterIndices.count ?? 0,
documentCount: clusterIndices.docs?.total ?? clusterIndices.docs?.count ?? 0,
dataSize: clusterIndices.store?.size.bytes ?? clusterIndices.store?.size_in_bytes ?? 0,
indicesCount:
clusterStatsLegacy?.indices?.shards?.total ?? clusterStatsMB?.indices?.shards?.count ?? 0,
documentCount:
clusterStatsLegacy?.indices?.docs?.count ?? clusterStatsMB?.indices?.docs?.total ?? 0,
dataSize:
clusterStatsMB?.indices?.store?.size?.bytes ??
clusterStatsLegacy?.indices?.store?.size_in_bytes ??
0,
// node-based stats
nodesCount: clusterNodes.count?.total ?? clusterNodes.count ?? 0,
upTime: clusterNodes.jvm?.max_uptime?.ms ?? lusterNodes.jvm?.max_uptime_in_millis ?? 0,
version: clusterNodes.versions ?? null,
nodesCount: clusterStatsLegacy?.nodes?.count?.total ?? clusterStatsMB?.nodes?.count ?? 0,
upTime:
clusterStatsMB?.nodes?.jvm?.max_uptime?.ms ??
clusterStatsLegacy?.nodes?.jvm?.max_uptime_in_millis ??
0,
version: clusterStatsLegacy?.nodes?.versions ?? null,
memUsed:
clusterNodes.jvm?.memory?.heap?.used?.bytes ?? clusterNodes.jvm?.mem?.heap_used_in_bytes ?? 0,
clusterStatsMB?.nodes?.jvm?.memory?.heap?.used?.bytes ??
clusterStatsLegacy?.nodes?.jvm?.mem?.heap_used_in_bytes ??
0,
memMax:
clusterNodes.jvm?.memory?.heap?.max?.bytes ?? clusterNodes.jvm?.mem?.heap_max_in_bytes ?? 0,
clusterStatsMB?.nodes?.jvm?.memory?.heap?.max?.bytes ??
clusterStatsLegacy?.nodes?.jvm?.mem?.heap_max_in_bytes ??
0,
unassignedShards: unassignedShardsTotal,
totalShards,
};
Original file line number Diff line number Diff line change
@@ -10,7 +10,9 @@ import { getClustersSummary } from './get_clusters_summary';

const mockLog = jest.fn();
const mockServer = {
log: mockLog,
log: {
error: mockLog,
},
};

describe('getClustersSummary', () => {
@@ -34,7 +36,6 @@ describe('getClustersSummary', () => {

expect(() => getClustersSummary(mockServer, fakeClusters)).toThrow('Monitoring License Error');
expect(mockLog).toHaveBeenCalledWith(
['error', 'monitoring'],
"Could not find license information for cluster = 'Custom name'. " +
"Please check the cluster's master node server logs for errors or warnings."
);
Original file line number Diff line number Diff line change
@@ -6,11 +6,7 @@
*/

import { omit, get } from 'lodash';
import {
ElasticsearchModifiedSource,
ElasticsearchLegacySource,
ElasticsearchMetricbeatSource,
} from '../../../common/types/es';
import { ElasticsearchModifiedSource, ElasticsearchLegacySource } from '../../../common/types/es';
// @ts-ignore
import { calculateOverallStatus } from '../calculate_overall_status';
// @ts-ignore
@@ -44,11 +40,8 @@ export function getClustersSummary(
const license = cluster.license || cluster.elasticsearch?.cluster?.stats?.license;
const version = cluster.version || cluster.elasticsearch?.version;
const clusterUuid = cluster.cluster_uuid || cluster.elasticsearch?.cluster?.id;
const clusterStats:
| ElasticsearchLegacySource['cluster_stats']
| NonNullable<
NonNullable<ElasticsearchMetricbeatSource['elasticsearch']>['cluster']
>['stats'] = cluster.cluster_stats || cluster.elasticsearch?.cluster?.stats;
const clusterStatsLegacy = cluster.cluster_stats;
const clusterStatsMB = cluster.elasticsearch?.cluster?.stats;

const clusterName = get(clusterSettings, 'cluster.metadata.display_name', cluster.cluster_name);

@@ -71,22 +64,23 @@ export function getClustersSummary(
} = license;

const indices = {
count: clusterStats?.indices?.count ?? clusterStats?.indices?.total,
docs: clusterStats?.indices?.docs,
shards: clusterStats?.indices?.shards,
store: clusterStats?.indices?.store,
count: clusterStatsLegacy?.indices?.count ?? clusterStatsMB?.indices?.total,
docs: clusterStatsLegacy?.indices?.docs ?? clusterStatsMB?.indices?.docs,
shards: clusterStatsLegacy?.indices?.shards ?? clusterStatsMB?.indices?.shards,
store: clusterStatsLegacy?.indices?.store ?? clusterStatsMB?.indices?.store,
};

const jvm = {
max_uptime_in_millis:
clusterStats?.nodes?.jvm?.max_uptime_in_millis ?? clusterStats?.nodes?.jvm?.max_uptime?.ms,
mem: clusterStats?.nodes?.jvm?.mem ?? clusterStats?.nodes?.jvm?.memory,
clusterStatsLegacy?.nodes?.jvm?.max_uptime_in_millis ??
clusterStatsMB?.nodes?.jvm?.max_uptime?.ms,
mem: clusterStatsLegacy?.nodes?.jvm?.mem ?? clusterStatsMB?.nodes?.jvm?.memory,
};

const nodes = {
fs: clusterStats?.nodes?.fs,
fs: clusterStatsLegacy?.nodes?.fs ?? clusterStatsMB?.nodes?.fs,
count: {
total: clusterStats?.nodes?.count?.total ?? clusterStats?.nodes?.count,
total: clusterStatsLegacy?.nodes?.count?.total ?? clusterStatsMB?.nodes?.count,
},
jvm,
};
Original file line number Diff line number Diff line change
@@ -11,7 +11,11 @@ import { checkParam } from '../../error_missing_required';
import { createQuery } from '../../create_query';
// @ts-ignore
import { ElasticsearchMetric } from '../../metrics';
import { ElasticsearchResponse, ElasticsearchLegacySource } from '../../../../common/types/es';
import {
ElasticsearchResponse,
ElasticsearchLegacySource,
ElasticsearchMetricbeatSource,
} from '../../../../common/types/es';
import { LegacyRequest } from '../../../types';

export function handleResponse(response: ElasticsearchResponse) {
@@ -24,26 +28,38 @@ export function handleResponse(response: ElasticsearchResponse) {
const uniqueShards = new Set<string>();

// map into object with shard and source propertiesd
return hits.reduce((shards: Array<ElasticsearchLegacySource['shard']>, hit) => {
const shard = hit._source.shard ?? hit._source.elasticsearch;
return hits.reduce(
(
shards: Array<
| ElasticsearchLegacySource['shard']
| NonNullable<ElasticsearchMetricbeatSource['elasticsearch']>['shard']
>,
hit
) => {
const legacyShard = hit._source.shard;
const mbShard = hit._source.elasticsearch;

if (shard) {
const index = shard.index?.name ?? shard.index;
const shardNumber = shard.shard?.number ?? shard.shard;
const primary = shard.shard?.primary ?? shard.primary;
const relocatingNode = shard.shard?.relocating_node?.uuid ?? shard.relocating_node;
const node = shard.node?.name ?? shard.node;
// note: if the request is for a node, then it's enough to deduplicate without primary, but for indices it displays both
const shardId = `${index}-${shardNumber}-${primary}-${relocatingNode}-${node}`;
if (legacyShard || mbShard) {
const index = mbShard?.index?.name ?? legacyShard?.index;
const shardNumber = mbShard?.shard?.number ?? legacyShard?.shard;
const primary = mbShard?.shard?.primary ?? legacyShard?.primary;
const relocatingNode =
mbShard?.shard?.relocating_node?.uuid ?? legacyShard?.relocating_node;
const node = mbShard?.node?.name ?? legacyShard?.node;
// note: if the request is for a node, then it's enough to deduplicate without primary, but for indices it displays both
const shardId = `${index}-${shardNumber}-${primary}-${relocatingNode}-${node}`;

if (!uniqueShards.has(shardId)) {
shards.push(shard);
uniqueShards.add(shardId);
if (!uniqueShards.has(shardId)) {
// @ts-ignore
shards.push(legacyShard || mbShard);
uniqueShards.add(shardId);
}
}
}

return shards;
}, []);
return shards;
},
[]
);
}

export function getShardAllocation(
@@ -68,7 +84,7 @@ export function getShardAllocation(
},
{
term: {
'elasticsearch.cluster.stats.state.state_uuid': stateUuid,
'elasticsearch.cluster.state.id': stateUuid,
},
},
],
Original file line number Diff line number Diff line change
@@ -19,9 +19,14 @@ import { getShardAggs } from './get_shard_stat_aggs';
// @ts-ignore
import { calculateIndicesTotals } from './calculate_shard_stat_indices_totals';
import { LegacyRequest } from '../../../types';
import { ElasticsearchModifiedSource } from '../../../../common/types/es';
import { ElasticsearchResponse, ElasticsearchModifiedSource } from '../../../../common/types/es';

export function handleResponse(resp, includeNodes, includeIndices, cluster) {
export function handleResponse(
resp: ElasticsearchResponse,
includeNodes: boolean,
includeIndices: boolean,
cluster: ElasticsearchModifiedSource
) {
let indices;
let indicesTotals;
let nodes;
Loading