[Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] #125748

jasonrhodes · 2022-02-16T03:11:17Z

Summary

This PR replaces #125315 because I screwed that one up with a bad merge that resulted in tons of commits getting mixed into the PR.

@neptunian's original summary was:

resolves #125035. Fixes queries in Stack Monitoring rules to be compatible with metricbeat 8 mappings. Except for CCR, I tested these on main with metricbeat 8.1.0. CCR i tested with metricbeat 8.1.0 mappings but used my own data as I couldn't get a local CCR setup.

I tested these manually by looking at the response and the data it returned. Ideally we need a proper way to run functional tests on these rule queries.

TODO

test CCR with real metricbeat 8.0.0+ data - defer this until after this PR, need new issue
test all with internal collection data
[elasticsearch module] serialize shards properties beats#30408
[Stack Monitoring] add kibana_stats version alias to -mb template elasticsearch#83930

Backporting to 8.0.1 notes:

createDatasetFilter doesn't exist in 8.0.0 so will need add the metricset.name in a bool should along with type (legacy). no need to add data_stream.dataset term
no unit tests for each fetch in 8.0.0
esClient.search response change

x-pack/plugins/monitoring/server/lib/alerts/create_dataset_query_filter.ts

neptunian · 2022-02-16T04:48:39Z

x-pack/plugins/monitoring/server/lib/alerts/fetch_index_shard_size.ts

@@ -134,15 +135,20 @@ export async function fetchIndexShardSize(
      }
      const {
        _index: monitoringIndexName,
-        _source: { index_stats: indexStats },
+        _source: { index_stats: indexStats, elasticsearch },


if you just declare indexStats like this:
const indexStats = topHit._source.index_stats || topHit._source.elasticsearch?.index, I don't think you'd need to make any other changes below this line @klacabane

This will make the 8.0.1 backport a bit more painful in this file as well.

This reverts commit cdf8b24.

kibana-ci · 2022-02-16T12:24:17Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💛 Build #24523 was flaky 2ef5217
💔 Build #24511 failed bb1b0544f851c85532f60aea47046f2f31aac2b3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jasonrhodes · 2022-02-16T17:18:55Z

Thanks to the amazing new PR cloud deploy tool, I've just verified this PR fixes the SM CPU, Memory (JVM), and Disk Usage alerts in cloud! 🙌

kibanamachine · 2022-02-16T17:52:29Z

💔 All backports failed

Status	Branch	Result
❌	8.1	Backport failed because of merge conflicts
❌	8.0	Backport failed because of merge conflicts You might need to backport the following PRs to 8.0: - [REVERT THE REVERT] [Monitoring] Cluster state watch to Kibana alerting (#62793)

How to fix

Re-run the backport manually:

node scripts/backport --pr 125748

Questions ?

Please refer to the Backport tool documentation

…xed PR] (elastic#125748) * update queries to use metricset.name and fix paths * fix ccr query * fix cluster health query paths * update elasticsearch version mismatch * update fetchLicense paths * use ecs properties in fetch_index_shard_size * Clarified comment about createDatasetFilter * Re-editing a clarifying comment * Small comment edit * Revert "use ecs properties in fetch_index_shard_size" This reverts commit cdf8b24. * simplify fetch_index_shard_size document handling Co-authored-by: neptunian <[email protected]> Co-authored-by: klacabane <[email protected]> (cherry picked from commit 7126bc9) # Conflicts: # x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.ts

…xed PR] (elastic#125748) * update queries to use metricset.name and fix paths * fix ccr query * fix cluster health query paths * update elasticsearch version mismatch * update fetchLicense paths * use ecs properties in fetch_index_shard_size * Clarified comment about createDatasetFilter * Re-editing a clarifying comment * Small comment edit * Revert "use ecs properties in fetch_index_shard_size" This reverts commit cdf8b24. * simplify fetch_index_shard_size document handling Co-authored-by: neptunian <[email protected]> Co-authored-by: klacabane <[email protected]> (cherry picked from commit 7126bc9) # Conflicts: # x-pack/plugins/monitoring/server/lib/alerts/create_dataset_query_filter.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_ccr_read_exceptions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_ccr_read_exceptions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cluster_health.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cluster_health.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cpu_usage_node_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cpu_usage_node_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_disk_usage_node_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_disk_usage_node_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_elasticsearch_versions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_elasticsearch_versions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_index_shard_size.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_index_shard_size.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_kibana_versions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_kibana_versions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_licenses.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_licenses.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_logstash_versions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_logstash_versions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_memory_usage_node_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_memory_usage_node_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_missing_monitoring_data.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_missing_monitoring_data.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_nodes_from_cluster_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_nodes_from_cluster_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_thread_pool_rejections_stats.ts

….0 [fixed PR] (#125748) (#125856) * [Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] (#125748) * update queries to use metricset.name and fix paths * fix ccr query * fix cluster health query paths * update elasticsearch version mismatch * update fetchLicense paths * use ecs properties in fetch_index_shard_size * Clarified comment about createDatasetFilter * Re-editing a clarifying comment * Small comment edit * Revert "use ecs properties in fetch_index_shard_size" This reverts commit cdf8b24. * simplify fetch_index_shard_size document handling Co-authored-by: neptunian <[email protected]> Co-authored-by: klacabane <[email protected]> (cherry picked from commit 7126bc9) # Conflicts: # x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.ts * esClient still returns metadata Co-authored-by: Jason Rhodes <[email protected]>

….0 [fixed PR] (#125748) (#125877) * [Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] (#125748) * update queries to use metricset.name and fix paths * fix ccr query * fix cluster health query paths * update elasticsearch version mismatch * update fetchLicense paths * use ecs properties in fetch_index_shard_size * Clarified comment about createDatasetFilter * Re-editing a clarifying comment * Small comment edit * Revert "use ecs properties in fetch_index_shard_size" This reverts commit cdf8b24. * simplify fetch_index_shard_size document handling Co-authored-by: neptunian <[email protected]> Co-authored-by: klacabane <[email protected]> (cherry picked from commit 7126bc9) # Conflicts: # x-pack/plugins/monitoring/server/lib/alerts/create_dataset_query_filter.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_ccr_read_exceptions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_ccr_read_exceptions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cluster_health.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cluster_health.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cpu_usage_node_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_cpu_usage_node_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_disk_usage_node_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_disk_usage_node_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_elasticsearch_versions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_elasticsearch_versions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_index_shard_size.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_index_shard_size.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_kibana_versions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_kibana_versions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_licenses.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_licenses.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_logstash_versions.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_logstash_versions.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_memory_usage_node_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_memory_usage_node_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_missing_monitoring_data.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_missing_monitoring_data.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_nodes_from_cluster_stats.test.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_nodes_from_cluster_stats.ts # x-pack/plugins/monitoring/server/lib/alerts/fetch_thread_pool_rejections_stats.ts * import createDatasetFilter * target correct filter property Co-authored-by: Jason Rhodes <[email protected]>

neptunian and others added 6 commits February 15, 2022 21:43

update queries to use metricset.name and fix paths

4bd23aa

fix ccr query

93e309c

fix cluster health query paths

de5b26c

update elasticsearch version mismatch

6a0b7b4

update fetchLicense paths

6375950

use ecs properties in fetch_index_shard_size

cdf8b24

jasonrhodes added bug Fixes for quality problems that affect the customer experience v8.1.0 v8.2.0 v8.0.1 labels Feb 16, 2022

jasonrhodes requested a review from a team as a code owner February 16, 2022 03:11

jasonrhodes added the release_note:fix label Feb 16, 2022

jasonrhodes changed the title ~~125035 fix alerts mb 8.0~~ [Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] Feb 16, 2022

jasonrhodes mentioned this pull request Feb 16, 2022

[Stack Monitoring] update rules queries to support metricbeat 8.0 #125315

Closed

4 tasks

Clarified comment about createDatasetFilter

dcf4de6

jasonrhodes force-pushed the 125035-fix-alerts-mb-8.0 branch from bb1b054 to dcf4de6 Compare February 16, 2022 03:34

neptunian reviewed Feb 16, 2022

View reviewed changes

x-pack/plugins/monitoring/server/lib/alerts/create_dataset_query_filter.ts Show resolved Hide resolved

neptunian reviewed Feb 16, 2022

View reviewed changes

Re-editing a clarifying comment

21845af

jasonrhodes added the ci:deploy-cloud label Feb 16, 2022

jasonrhodes and others added 3 commits February 16, 2022 00:05

Small comment edit

2ef5217

Revert "use ecs properties in fetch_index_shard_size"

0208deb

This reverts commit cdf8b24.

simplify fetch_index_shard_size document handling

dd154c2

neptunian approved these changes Feb 16, 2022

View reviewed changes

klacabane approved these changes Feb 16, 2022

View reviewed changes

jasonrhodes merged commit 7126bc9 into elastic:main Feb 16, 2022

jasonrhodes added the auto-backport Deprecated - use backport:version if exact versions are needed label Feb 16, 2022

jasonrhodes deleted the 125035-fix-alerts-mb-8.0 branch February 16, 2022 18:11

jasonrhodes mentioned this pull request Feb 16, 2022

[8.0] [Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] #125845

Closed

klacabane mentioned this pull request Feb 16, 2022

[8.1] [Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] (#125748) #125856

Merged

klacabane mentioned this pull request Feb 16, 2022

[8.0] [Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] (#125748) #125877

Merged

KOTungseth added the Feature:Stack Monitoring label Feb 17, 2022

tylersmalley added ci:cloud-deploy Create or update a Cloud deployment and removed ci:deploy-cloud labels Aug 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] #125748

[Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] #125748

jasonrhodes commented Feb 16, 2022 •

edited by klacabane

Loading

neptunian Feb 16, 2022

neptunian Feb 16, 2022 •

edited

Loading

kibana-ci commented Feb 16, 2022

jasonrhodes commented Feb 16, 2022

kibanamachine commented Feb 16, 2022

[Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] #125748

[Stack Monitoring] update rules queries to support metricbeat 8.0 [fixed PR] #125748

Conversation

jasonrhodes commented Feb 16, 2022 • edited by klacabane Loading

Summary

TODO

neptunian Feb 16, 2022

Choose a reason for hiding this comment

neptunian Feb 16, 2022 • edited Loading

Choose a reason for hiding this comment

kibana-ci commented Feb 16, 2022

💚 Build Succeeded

Metrics [docs]

History

jasonrhodes commented Feb 16, 2022

kibanamachine commented Feb 16, 2022

💔 All backports failed

How to fix

Questions ?

jasonrhodes commented Feb 16, 2022 •

edited by klacabane

Loading

neptunian Feb 16, 2022 •

edited

Loading