[Stack Monitoring] Uncaught exception attempting to load standalone logstash nodes page

[matschaffer]

Kibana version: b2462a6 (8.0 branch head)

Elasticsearch version: 8.0 head

Server OS version: macos

Browser version: chrome

Browser OS version: macos

Original install method (e.g. download page, yum, from source, etc.): source

Describe the bug:

This error is shown when attempting to navigate to standalone logstash nodes page.

Cannot read properties of undefined (reading 'uuid')

Additionally once in this state, navigating via breadcrumbs or back button doesn't help. You need to navigate back to the top of the monitoring app via sidebar or kibana search.

Steps to reproduce:

1. Start a stack with stack monitoring (metricbeat colleciton)
2. Run logstash in a standalone configuration (while :; do echo 'standalone hello' ; sleep 1; done | bin/logstash -e 'input { stdin { } } output { stdout {} }')
3. Navigate to the "Nodes" section of the standalone logstash cluster in stack monitoring

Expected behavior:

Similar node listing/details that we'd get with an elasticsearch output

Screenshots (if relevant):

Cluster overview is good

Logstash overview too

"Nodes" tab explodes

Errors in browser console (if relevant):

listing.js:57 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'uuid')
    at render (listing.js:57:1)
    at EuiBasicTable.renderItemFieldDataCell (basic_table.js:1232:1)
    at basic_table.js:1087:1
    at Array.forEach (<anonymous>)
    at EuiBasicTable.renderItemRow (basic_table.js:1081:1)
    at basic_table.js:1020:1
    at Array.map (<anonymous>)
    at EuiBasicTable.renderTableBody (basic_table.js:1017:1)
    at EuiBasicTable.renderTable (basic_table.js:739:1)
    at EuiBasicTable.render (basic_table.js:715:1)

Provide logs and/or server output (if relevant):

Console output with monitoring.ui.debug_mode: true

[2022-01-26T11:06:20.529+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:filebeat-*,filebeat-*","size":0,"filter_path":["aggregations.levels.buckets","aggregations.types.buckets"],"ignore_unavailable":true,"body":{"sort":{"@timestamp":{"order":"desc","unmapped_type":"long"}},"query":{"bool":{"filter":[{"term":{"service.type":"elasticsearch"}},{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"term":{"elasticsearch.cluster.uuid":"__standalone_cluster__"}}]}},"aggs":{"types":{"terms":{"field":"event.dataset"},"aggs":{"levels":{"terms":{"field":"log.level"}}}}}}},"result":{"aggregations":{"types":{"buckets":[]}}}}}
[2022-01-26T11:06:20.534+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"body":[{"index":"*:filebeat-*,filebeat-*"},{"size":0},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"term":{"service.type":"elasticsearch"}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"term":{"service.type":"elasticsearch"}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"term":{"service.type":"elasticsearch"}},{"exists":{"field":"elasticsearch.cluster"}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"term":{"service.type":"elasticsearch"}},{"term":{"elasticsearch.cluster.uuid":"__standalone_cluster__"}}]}}}]},"result":{"took":1,"responses":[{"took":1,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":897,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":12,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":897,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":12,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]},"status":200}]}}}
[2022-01-26T11:06:20.542+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-logstash-*,.monitoring-logstash-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"node_stats"}},{"term":{"type":"logstash_stats"}},{"term":{"metricset.name":"node_stats"}},{"term":{"metricset.name":"logstash_stats"}}]}},{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"logstash_uuids":{"terms":{"field":"logstash_stats.logstash.uuid","size":10000},"aggs":{"latest_report":{"terms":{"field":"logstash_stats.timestamp","size":1,"order":{"_key":"desc"}},"aggs":{"memory_used":{"max":{"field":"logstash_stats.jvm.mem.heap_used_in_bytes"}},"memory":{"max":{"field":"logstash_stats.jvm.mem.heap_max_in_bytes"}},"events_in_total":{"max":{"field":"logstash_stats.events.in"}},"events_out_total":{"max":{"field":"logstash_stats.events.out"}}}},"memory_used_per_node":{"max_bucket":{"buckets_path":"latest_report>memory_used"}},"memory_per_node":{"max_bucket":{"buckets_path":"latest_report>memory"}},"events_in_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_in_total"}},"events_out_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_out_total"}}}},"logstash_versions":{"terms":{"field":"logstash_stats.logstash.version","size":10000}},"pipelines_nested":{"nested":{"path":"logstash_stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash_stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash_stats.pipelines.id"}}}}}},"pipelines_nested_mb":{"nested":{"path":"logstash.node.stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash.node.stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash.node.stats.pipelines.id"}}}}}},"events_in_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_in_total_per_node"}},"events_out_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_out_total_per_node"}},"memory_used":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_used_per_node"}},"memory":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_per_node"}},"max_uptime":{"max":{"field":"logstash_stats.jvm.uptime_in_millis"}}}}},"result":{"took":4,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":2,"successful":2,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":43,"relation":"eq"},"max_score":0,"hits":[]},"aggregations":{"max_uptime":{"value":345234},"logstash_uuids":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","doc_count":34,"latest_report":{"doc_count_error_upper_bound":0,"sum_other_doc_count":33,"buckets":[{"key":1643162697408,"key_as_string":"2022-01-26T02:04:57.408Z","doc_count":1,"events_in_total":{"value":346},"memory":{"value":1037959168},"events_out_total":{"value":346},"memory_used":{"value":423274752}}]},"memory_used_per_node":{"value":423274752,"keys":["2022-01-26T02:04:57.408Z"]},"memory_per_node":{"value":1037959168,"keys":["2022-01-26T02:04:57.408Z"]},"events_in_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]},"events_out_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]}}]},"pipelines_nested":{"doc_count":0,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]},"pipelines":{"value":0}},"pipelines_nested_mb":{"doc_count":34,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"persisted","doc_count":34,"num_pipelines":{"value":1}}]},"pipelines":{"value":1}},"logstash_versions":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"8.0.0","doc_count":34}]},"events_in_total":{"value":346},"events_out_total":{"value":346},"memory_used":{"value":423274752},"memory":{"value":1037959168}}}}}
[2022-01-26T11:06:20.548+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-logstash-*,.monitoring-logstash-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.nest.id.buckets","aggregations.nest_mb.id.buckets"],"body":{"query":{"bool":{"filter":[{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"nest":{"nested":{"path":"logstash_stats.pipelines"},"aggs":{"id":{"terms":{"field":"logstash_stats.pipelines.id","size":1},"aggs":{"unnest":{"reverse_nested":{},"aggs":{"nodes":{"terms":{"field":"logstash_stats.logstash.uuid","size":1}}}}}}}},"nest_mb":{"nested":{"path":"logstash.node.stats.pipelines"},"aggs":{"id":{"terms":{"field":"logstash.node.stats.pipelines.id","size":1},"aggs":{"unnest_mb":{"reverse_nested":{},"aggs":{"nodes":{"terms":{"field":"logstash.node.stats.logstash.uuid","size":1}}}}}}}}}}},"result":{"aggregations":{"nest":{"id":{"buckets":[]}},"nest_mb":{"id":{"buckets":[{"key":"main","doc_count":34,"unnest_mb":{"doc_count":34,"nodes":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","doc_count":34}]}}}]}}}}}}
[2022-01-26T11:06:20.552+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-beats-*,.monitoring-beats-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.total","aggregations.types.buckets.key","aggregations.types.buckets.uuids.buckets.doc_count","aggregations.min_events_total.value","aggregations.max_events_total.value","aggregations.min_bytes_sent_total.value","aggregations.max_bytes_sent_total.value"],"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"stats"}},{"term":{"type":"beats_stats"}},{"term":{"metricset.name":"stats"}},{"term":{"metricset.name":"beats_stats"}}]}},{"range":{"beats_stats.timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"bool":{"must_not":{"term":{"beats_stats.beat.type":"apm-server"}}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"types":{"terms":{"field":"beats_stats.beat.type","size":1000},"aggs":{"uuids":{"terms":{"field":"beats_stats.beat.uuid","size":10000}}}},"total":{"cardinality":{"field":"beats_stats.beat.uuid","precision_threshold":10000}},"ephemeral_ids":{"terms":{"field":"beats_stats.metrics.beat.info.ephemeral_id","size":10000},"aggs":{"min_events":{"min":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"max_events":{"max":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"min_bytes_sent":{"min":{"field":"beats_stats.metrics.libbeat.output.write.bytes"}},"max_bytes_sent":{"max":{"field":"beats_stats.metrics.libbeat.output.write.bytes"}}}},"min_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_events"}},"max_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_events"}},"min_bytes_sent_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_bytes_sent"}},"max_bytes_sent_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_bytes_sent"}}}}},"result":{"aggregations":{"total":{"value":0},"min_events_total":{"value":0},"max_events_total":{"value":0},"min_bytes_sent_total":{"value":0},"max_bytes_sent_total":{"value":0}}}}}
[2022-01-26T11:06:20.557+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-beats-*,.monitoring-beats-*,metricbeat-*,*:metricbeat-*","size":1,"ignore_unavailable":true,"body":{"_source":["beats_stats.timestamp","@timestamp"],"sort":[{"timestamp":{"order":"desc","unmapped_type":"long"}}],"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"stats"}},{"term":{"type":"beats_stats"}},{"term":{"metricset.name":"stats"}},{"term":{"metricset.name":"beats_stats"}}]}},{"range":{"beats_stats.timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"range":{"beats_stats.metrics.libbeat.output.events.acked":{"gt":0}}},{"bool":{"must":{"term":{"beats_stats.beat.type":"apm-server"}}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}}}},"result":{"took":1,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":2,"successful":2,"skipped":1,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]}}}}
[2022-01-26T11:06:20.558+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-beats-*,.monitoring-beats-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.total","aggregations.min_events_total.value","aggregations.max_events_total.value","aggregations.min_mem_total.value","aggregations.max_mem_total.value","aggregations.versions.buckets"],"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"stats"}},{"term":{"type":"beats_stats"}},{"term":{"metricset.name":"stats"}},{"term":{"metricset.name":"beats_stats"}}]}},{"range":{"beats_stats.timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"bool":{"must":{"term":{"beats_stats.beat.type":"apm-server"}}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"total":{"cardinality":{"field":"beats_stats.beat.uuid","precision_threshold":10000}},"versions":{"terms":{"field":"beats_stats.beat.version"}},"ephemeral_ids":{"terms":{"field":"beats_stats.metrics.beat.info.ephemeral_id","size":10000},"aggs":{"min_events":{"min":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"max_events":{"max":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"min_mem":{"min":{"field":"beats_stats.metrics.beat.memstats.rss"}},"max_mem":{"max":{"field":"beats_stats.metrics.beat.memstats.rss"}}}},"min_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_events"}},"max_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_events"}},"min_mem_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_mem"}},"max_mem_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_mem"}}}}},"result":{"aggregations":{"total":{"value":0},"versions":{"buckets":[]},"min_events_total":{"value":0},"max_events_total":{"value":0},"min_mem_total":{"value":0},"max_mem_total":{"value":0}}}}}
[2022-01-26T11:06:20.563+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-ent-search-*,.monitoring-ent-search-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.total","aggregations.versions.buckets","aggregations.app_search_engines.value","aggregations.workplace_search_org_sources.value","aggregations.workplace_search_private_sources.value","aggregations.uptime.value","aggregations.cluster_heap_used.value","aggregations.cluster_heap_total.value","aggregations.cluster_heap_committed.value"],"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"health"}},{"term":{"type":"stats"}},{"term":{"metricset.name":"health"}},{"term":{"metricset.name":"stats"}}]}},{"term":{"enterprisesearch.cluster_uuid":"__standalone_cluster__"}},{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"bool":{"should":[{"term":{"event.dataset":"enterprisesearch.health"}},{"term":{"event.dataset":"enterprisesearch.stats"}}]}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"total":{"cardinality":{"field":"agent.id","precision_threshold":10000}},"versions":{"terms":{"field":"enterprisesearch.health.version.number"}},"latest_report":{"terms":{"field":"@timestamp","size":2,"order":{"_key":"desc"}},"aggs":{"app_search_engines":{"max":{"field":"enterprisesearch.stats.product_usage.app_search.total_engines"}},"workplace_search_org_sources":{"max":{"field":"enterprisesearch.stats.product_usage.workplace_search.total_org_sources"}},"workplace_search_private_sources":{"max":{"field":"enterprisesearch.stats.product_usage.workplace_search.total_private_sources"}}}},"ephemeral_ids":{"terms":{"field":"agent.ephemeral_id","size":10000},"aggs":{"uptime_max":{"max":{"field":"enterprisesearch.health.process.uptime.sec"}},"heap_used":{"max":{"field":"enterprisesearch.health.jvm.memory_usage.heap_used.bytes"}},"heap_total":{"max":{"field":"enterprisesearch.health.jvm.memory_usage.heap_max.bytes"}},"heap_committed":{"max":{"field":"enterprisesearch.health.jvm.memory_usage.heap_committed.bytes"}}}},"app_search_engines":{"max_bucket":{"buckets_path":"latest_report>app_search_engines"}},"workplace_search_org_sources":{"max_bucket":{"buckets_path":"latest_report>workplace_search_org_sources"}},"workplace_search_private_sources":{"max_bucket":{"buckets_path":"latest_report>workplace_search_private_sources"}},"uptime":{"max_bucket":{"buckets_path":"ephemeral_ids>uptime_max"}},"cluster_heap_used":{"sum_bucket":{"buckets_path":"ephemeral_ids>heap_used"}},"cluster_heap_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>heap_total"}},"cluster_heap_committed":{"sum_bucket":{"buckets_path":"ephemeral_ids>heap_committed"}}}}},"result":{"aggregations":{"total":{"value":0},"versions":{"buckets":[]},"app_search_engines":{"value":null},"workplace_search_org_sources":{"value":null},"workplace_search_private_sources":{"value":null},"uptime":{"value":null},"cluster_heap_used":{"value":0},"cluster_heap_total":{"value":0},"cluster_heap_committed":{"value":0}}}}}
[2022-01-26T11:06:20.567+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-es-*,.monitoring-es-*,metricbeat-*,*:metricbeat-*","size":1,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"cluster_stats"}},{"term":{"metricset.name":"cluster_stats"}}]}},{"range":{"timestamp":{"format":"epoch_millis","gte":1643161880503,"lte":1643162780503}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"sort":[{"timestamp":{"order":"desc","unmapped_type":"long"}}]},"filter_path":["hits.hits._source.stack_stats.xpack.ccr","hits.hits._source.elasticsearch.cluster.stats.stack.xpack.ccr"]},"result":{}}}
[2022-01-26T11:06:21.592+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:filebeat-*,filebeat-*","size":0,"filter_path":["aggregations.levels.buckets","aggregations.types.buckets"],"ignore_unavailable":true,"body":{"sort":{"@timestamp":{"order":"desc","unmapped_type":"long"}},"query":{"bool":{"filter":[{"term":{"service.type":"elasticsearch"}},{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"term":{"elasticsearch.cluster.uuid":"__standalone_cluster__"}}]}},"aggs":{"types":{"terms":{"field":"event.dataset"},"aggs":{"levels":{"terms":{"field":"log.level"}}}}}}},"result":{"aggregations":{"types":{"buckets":[]}}}}}
[2022-01-26T11:06:21.625+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"body":[{"index":"*:filebeat-*,filebeat-*"},{"size":0},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"term":{"service.type":"elasticsearch"}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"term":{"service.type":"elasticsearch"}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"term":{"service.type":"elasticsearch"}},{"exists":{"field":"elasticsearch.cluster"}}]}}},{"index":"*:filebeat-*,filebeat-*"},{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"term":{"service.type":"elasticsearch"}},{"term":{"elasticsearch.cluster.uuid":"__standalone_cluster__"}}]}}}]},"result":{"took":1,"responses":[{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":897,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":12,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":897,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":12,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]},"status":200},{"took":0,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]},"status":200}]}}}
[2022-01-26T11:06:21.637+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-logstash-*,.monitoring-logstash-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"node_stats"}},{"term":{"type":"logstash_stats"}},{"term":{"metricset.name":"node_stats"}},{"term":{"metricset.name":"logstash_stats"}}]}},{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"logstash_uuids":{"terms":{"field":"logstash_stats.logstash.uuid","size":10000},"aggs":{"latest_report":{"terms":{"field":"logstash_stats.timestamp","size":1,"order":{"_key":"desc"}},"aggs":{"memory_used":{"max":{"field":"logstash_stats.jvm.mem.heap_used_in_bytes"}},"memory":{"max":{"field":"logstash_stats.jvm.mem.heap_max_in_bytes"}},"events_in_total":{"max":{"field":"logstash_stats.events.in"}},"events_out_total":{"max":{"field":"logstash_stats.events.out"}}}},"memory_used_per_node":{"max_bucket":{"buckets_path":"latest_report>memory_used"}},"memory_per_node":{"max_bucket":{"buckets_path":"latest_report>memory"}},"events_in_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_in_total"}},"events_out_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_out_total"}}}},"logstash_versions":{"terms":{"field":"logstash_stats.logstash.version","size":10000}},"pipelines_nested":{"nested":{"path":"logstash_stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash_stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash_stats.pipelines.id"}}}}}},"pipelines_nested_mb":{"nested":{"path":"logstash.node.stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash.node.stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash.node.stats.pipelines.id"}}}}}},"events_in_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_in_total_per_node"}},"events_out_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_out_total_per_node"}},"memory_used":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_used_per_node"}},"memory":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_per_node"}},"max_uptime":{"max":{"field":"logstash_stats.jvm.uptime_in_millis"}}}}},"result":{"took":5,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":2,"successful":2,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":44,"relation":"eq"},"max_score":0,"hits":[]},"aggregations":{"max_uptime":{"value":345234},"logstash_uuids":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","doc_count":34,"latest_report":{"doc_count_error_upper_bound":0,"sum_other_doc_count":33,"buckets":[{"key":1643162697408,"key_as_string":"2022-01-26T02:04:57.408Z","doc_count":1,"events_in_total":{"value":346},"memory":{"value":1037959168},"events_out_total":{"value":346},"memory_used":{"value":423274752}}]},"memory_used_per_node":{"value":423274752,"keys":["2022-01-26T02:04:57.408Z"]},"memory_per_node":{"value":1037959168,"keys":["2022-01-26T02:04:57.408Z"]},"events_in_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]},"events_out_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]}}]},"pipelines_nested":{"doc_count":0,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]},"pipelines":{"value":0}},"pipelines_nested_mb":{"doc_count":34,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"persisted","doc_count":34,"num_pipelines":{"value":1}}]},"pipelines":{"value":1}},"logstash_versions":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"8.0.0","doc_count":34}]},"events_in_total":{"value":346},"events_out_total":{"value":346},"memory_used":{"value":423274752},"memory":{"value":1037959168}}}}}
[2022-01-26T11:06:21.651+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-logstash-*,.monitoring-logstash-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.nest.id.buckets","aggregations.nest_mb.id.buckets"],"body":{"query":{"bool":{"filter":[{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"nest":{"nested":{"path":"logstash_stats.pipelines"},"aggs":{"id":{"terms":{"field":"logstash_stats.pipelines.id","size":1},"aggs":{"unnest":{"reverse_nested":{},"aggs":{"nodes":{"terms":{"field":"logstash_stats.logstash.uuid","size":1}}}}}}}},"nest_mb":{"nested":{"path":"logstash.node.stats.pipelines"},"aggs":{"id":{"terms":{"field":"logstash.node.stats.pipelines.id","size":1},"aggs":{"unnest_mb":{"reverse_nested":{},"aggs":{"nodes":{"terms":{"field":"logstash.node.stats.logstash.uuid","size":1}}}}}}}}}}},"result":{"aggregations":{"nest":{"id":{"buckets":[]}},"nest_mb":{"id":{"buckets":[{"key":"main","doc_count":34,"unnest_mb":{"doc_count":34,"nodes":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","doc_count":34}]}}}]}}}}}}
[2022-01-26T11:06:21.657+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-beats-*,.monitoring-beats-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.total","aggregations.types.buckets.key","aggregations.types.buckets.uuids.buckets.doc_count","aggregations.min_events_total.value","aggregations.max_events_total.value","aggregations.min_bytes_sent_total.value","aggregations.max_bytes_sent_total.value"],"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"stats"}},{"term":{"type":"beats_stats"}},{"term":{"metricset.name":"stats"}},{"term":{"metricset.name":"beats_stats"}}]}},{"range":{"beats_stats.timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"bool":{"must_not":{"term":{"beats_stats.beat.type":"apm-server"}}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"types":{"terms":{"field":"beats_stats.beat.type","size":1000},"aggs":{"uuids":{"terms":{"field":"beats_stats.beat.uuid","size":10000}}}},"total":{"cardinality":{"field":"beats_stats.beat.uuid","precision_threshold":10000}},"ephemeral_ids":{"terms":{"field":"beats_stats.metrics.beat.info.ephemeral_id","size":10000},"aggs":{"min_events":{"min":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"max_events":{"max":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"min_bytes_sent":{"min":{"field":"beats_stats.metrics.libbeat.output.write.bytes"}},"max_bytes_sent":{"max":{"field":"beats_stats.metrics.libbeat.output.write.bytes"}}}},"min_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_events"}},"max_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_events"}},"min_bytes_sent_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_bytes_sent"}},"max_bytes_sent_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_bytes_sent"}}}}},"result":{"aggregations":{"total":{"value":0},"min_events_total":{"value":0},"max_events_total":{"value":0},"min_bytes_sent_total":{"value":0},"max_bytes_sent_total":{"value":0}}}}}
[2022-01-26T11:06:21.664+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-beats-*,.monitoring-beats-*,metricbeat-*,*:metricbeat-*","size":1,"ignore_unavailable":true,"body":{"_source":["beats_stats.timestamp","@timestamp"],"sort":[{"timestamp":{"order":"desc","unmapped_type":"long"}}],"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"stats"}},{"term":{"type":"beats_stats"}},{"term":{"metricset.name":"stats"}},{"term":{"metricset.name":"beats_stats"}}]}},{"range":{"beats_stats.timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"range":{"beats_stats.metrics.libbeat.output.events.acked":{"gt":0}}},{"bool":{"must":{"term":{"beats_stats.beat.type":"apm-server"}}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}}}},"result":{"took":1,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":2,"successful":2,"skipped":1,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]}}}}
[2022-01-26T11:06:21.665+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-beats-*,.monitoring-beats-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.total","aggregations.min_events_total.value","aggregations.max_events_total.value","aggregations.min_mem_total.value","aggregations.max_mem_total.value","aggregations.versions.buckets"],"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"stats"}},{"term":{"type":"beats_stats"}},{"term":{"metricset.name":"stats"}},{"term":{"metricset.name":"beats_stats"}}]}},{"range":{"beats_stats.timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"bool":{"must":{"term":{"beats_stats.beat.type":"apm-server"}}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"total":{"cardinality":{"field":"beats_stats.beat.uuid","precision_threshold":10000}},"versions":{"terms":{"field":"beats_stats.beat.version"}},"ephemeral_ids":{"terms":{"field":"beats_stats.metrics.beat.info.ephemeral_id","size":10000},"aggs":{"min_events":{"min":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"max_events":{"max":{"field":"beats_stats.metrics.libbeat.pipeline.events.total"}},"min_mem":{"min":{"field":"beats_stats.metrics.beat.memstats.rss"}},"max_mem":{"max":{"field":"beats_stats.metrics.beat.memstats.rss"}}}},"min_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_events"}},"max_events_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_events"}},"min_mem_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>min_mem"}},"max_mem_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>max_mem"}}}}},"result":{"aggregations":{"total":{"value":0},"versions":{"buckets":[]},"min_events_total":{"value":0},"max_events_total":{"value":0},"min_mem_total":{"value":0},"max_mem_total":{"value":0}}}}}
[2022-01-26T11:06:21.672+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-ent-search-*,.monitoring-ent-search-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.total","aggregations.versions.buckets","aggregations.app_search_engines.value","aggregations.workplace_search_org_sources.value","aggregations.workplace_search_private_sources.value","aggregations.uptime.value","aggregations.cluster_heap_used.value","aggregations.cluster_heap_total.value","aggregations.cluster_heap_committed.value"],"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"health"}},{"term":{"type":"stats"}},{"term":{"metricset.name":"health"}},{"term":{"metricset.name":"stats"}}]}},{"term":{"enterprisesearch.cluster_uuid":"__standalone_cluster__"}},{"range":{"@timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"bool":{"should":[{"term":{"event.dataset":"enterprisesearch.health"}},{"term":{"event.dataset":"enterprisesearch.stats"}}]}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"total":{"cardinality":{"field":"agent.id","precision_threshold":10000}},"versions":{"terms":{"field":"enterprisesearch.health.version.number"}},"latest_report":{"terms":{"field":"@timestamp","size":2,"order":{"_key":"desc"}},"aggs":{"app_search_engines":{"max":{"field":"enterprisesearch.stats.product_usage.app_search.total_engines"}},"workplace_search_org_sources":{"max":{"field":"enterprisesearch.stats.product_usage.workplace_search.total_org_sources"}},"workplace_search_private_sources":{"max":{"field":"enterprisesearch.stats.product_usage.workplace_search.total_private_sources"}}}},"ephemeral_ids":{"terms":{"field":"agent.ephemeral_id","size":10000},"aggs":{"uptime_max":{"max":{"field":"enterprisesearch.health.process.uptime.sec"}},"heap_used":{"max":{"field":"enterprisesearch.health.jvm.memory_usage.heap_used.bytes"}},"heap_total":{"max":{"field":"enterprisesearch.health.jvm.memory_usage.heap_max.bytes"}},"heap_committed":{"max":{"field":"enterprisesearch.health.jvm.memory_usage.heap_committed.bytes"}}}},"app_search_engines":{"max_bucket":{"buckets_path":"latest_report>app_search_engines"}},"workplace_search_org_sources":{"max_bucket":{"buckets_path":"latest_report>workplace_search_org_sources"}},"workplace_search_private_sources":{"max_bucket":{"buckets_path":"latest_report>workplace_search_private_sources"}},"uptime":{"max_bucket":{"buckets_path":"ephemeral_ids>uptime_max"}},"cluster_heap_used":{"sum_bucket":{"buckets_path":"ephemeral_ids>heap_used"}},"cluster_heap_total":{"sum_bucket":{"buckets_path":"ephemeral_ids>heap_total"}},"cluster_heap_committed":{"sum_bucket":{"buckets_path":"ephemeral_ids>heap_committed"}}}}},"result":{"aggregations":{"total":{"value":0},"versions":{"buckets":[]},"app_search_engines":{"value":null},"workplace_search_org_sources":{"value":null},"workplace_search_private_sources":{"value":null},"uptime":{"value":null},"cluster_heap_used":{"value":0},"cluster_heap_total":{"value":0},"cluster_heap_committed":{"value":0}}}}}
[2022-01-26T11:06:21.677+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-es-*,.monitoring-es-*,metricbeat-*,*:metricbeat-*","size":1,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"cluster_stats"}},{"term":{"metricset.name":"cluster_stats"}}]}},{"range":{"timestamp":{"format":"epoch_millis","gte":1643161881193,"lte":1643162781193}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"sort":[{"timestamp":{"order":"desc","unmapped_type":"long"}}]},"filter_path":["hits.hits._source.stack_stats.xpack.ccr","hits.hits._source.elasticsearch.cluster.stats.stack.xpack.ccr"]},"result":{}}}
[2022-01-26T11:06:22.616+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"method":"POST","path":"/_security/user/_has_privileges","body":{"index":[{"names":[".monitoring-*"],"privileges":["read"]}]},"ignore_unavailable":true},"result":{"username":"elastic","has_all_requested":true,"cluster":{},"index":{".monitoring-*":{"read":true}},"application":{}}}}
[2022-01-26T11:06:22.623+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-es-*,.monitoring-es-*,metricbeat-*,*:metricbeat-*","size":10000,"ignore_unavailable":true,"filter_path":["hits.hits._index","hits.hits._source.cluster_uuid","hits.hits._source.elasticsearch.cluster.id","hits.hits._source.cluster_name","hits.hits._source.elasticsearch.cluster.name","hits.hits._source.version","hits.hits._source.elasticsearch.version","hits.hits._source.elasticsearch.cluster.node.version","hits.hits._source.license.status","hits.hits._source.elasticsearch.cluster.stats.license.status","hits.hits._source.license.type","hits.hits._source.elasticsearch.cluster.stats.license.type","hits.hits._source.license.issue_date","hits.hits._source.elasticsearch.cluster.stats.license.issue_date","hits.hits._source.license.expiry_date","hits.hits._source.elasticsearch.cluster.stats.license.expiry_date","hits.hits._source.license.expiry_date_in_millis","hits.hits._source.elasticsearch.cluster.stats.license.expiry_date_in_millis","hits.hits._source.cluster_stats","hits.hits._source.elasticsearch.cluster.stats","hits.hits._source.cluster_state","hits.hits._source.elasticsearch.cluster.stats.state","hits.hits._source.cluster_settings.cluster.metadata.display_name"],"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"cluster_stats"}},{"term":{"metricset.name":"cluster_stats"}}]}},{"range":{"timestamp":{"format":"epoch_millis","gte":1643161882555,"lte":1643162782555}}}]}},"collapse":{"field":"cluster_uuid"},"sort":{"timestamp":{"order":"desc","unmapped_type":"long"}}}},"result":{"hits":{"hits":[{"_index":".ds-.monitoring-es-8-mb-2022.01.25-000001","_source":{"elasticsearch":{"cluster":{"name":"main","id":"28iKoFsvTJ6HEyXbdLL-PQ","stats":{"state":{"nodes_hash":-1356093597,"master_node":"tc3nhgC0SFCKfwwy6jCmkw","state_uuid":"h3JOYXKaRsyUmJGVZsQIzg","nodes":{"tc3nhgC0SFCKfwwy6jCmkw":{"name":"matschaffer-mbp2019.lan","ephemeral_id":"iP5aOy1DRciwbc4pLTOxHQ","transport_address":"127.0.0.1:9300","attributes":{"xpack.installed":"true","ml.max_jvm_size":"33285996544","ml.machine_memory":"68719476736"},"roles":["data","data_cold","data_content","data_frozen","data_hot","data_warm","ingest","master","ml","remote_cluster_client","transform"]}}},"status":"yellow","nodes":{"versions":["8.0.0"],"count":1,"master":1,"fs":{"total":{"bytes":1000240963584},"available":{"bytes":457802162176}},"jvm":{"max_uptime":{"ms":37533046},"memory":{"heap":{"max":{"bytes":33285996544},"used":{"bytes":3934425312}}}}},"indices":{"shards":{"primaries":87,"count":87},"store":{"size":{"bytes":548800966}},"fielddata":{"memory":{"bytes":1688}},"docs":{"total":513215},"total":87},"stack":{"xpack":{"ccr":{"enabled":true,"available":true}}},"license":{"issue_date_in_millis":1643123650031,"expiry_date":"2022-02-24T15:14:10.031Z","cluster_needs_tls":true,"max_nodes":1000,"start_date_in_millis":-1,"expiry_date_in_millis":1645715650031,"status":"active","uid":"6622e805-8d4a-4fc6-ab8e-faacc05f910f","issue_date":"2022-01-25T15:14:10.031Z","issuer":"elasticsearch","type":"trial","issued_to":"secondary"}}},"version":481}}},{"_index":".ds-.monitoring-es-8-mb-2022.01.25-000001","_source":{"elasticsearch":{"cluster":{"id":"PuRGGPeVR-akKtACT2yTCQ","stats":{"license":{"status":"active","issued_to":"secondary","issue_date_in_millis":1643123650031,"issuer":"elasticsearch","expiry_date_in_millis":1645715650031,"type":"trial","issue_date":"2022-01-25T15:14:10.031Z","cluster_needs_tls":true,"expiry_date":"2022-02-24T15:14:10.031Z","max_nodes":1000,"uid":"6622e805-8d4a-4fc6-ab8e-faacc05f910f","start_date_in_millis":-1},"state":{"nodes":{"vN1RzXDqQwWdZfURDETxDQ":{"name":"matschaffer-mbp2019.lan","ephemeral_id":"ZbJyrZExTGybyOZR7w0Ixg","transport_address":"127.0.0.1:9301","attributes":{"ml.machine_memory":"68719476736","xpack.installed":"true","ml.max_jvm_size":"33285996544"},"roles":["data","data_cold","data_content","data_frozen","data_hot","data_warm","ingest","master","ml","remote_cluster_client","transform"]}},"nodes_hash":-1573159685,"master_node":"vN1RzXDqQwWdZfURDETxDQ","state_uuid":"bubbIo8KSF6QNdsYabeP6A"},"status":"yellow","nodes":{"versions":["8.0.0"],"count":1,"master":1,"fs":{"available":{"bytes":457803210752},"total":{"bytes":1000240963584}},"jvm":{"memory":{"heap":{"used":{"bytes":5127408288},"max":{"bytes":33285996544}}},"max_uptime":{"ms":37471176}}},"indices":{"shards":{"count":5,"primaries":5},"store":{"size":{"bytes":42470185}},"fielddata":{"memory":{"bytes":0}},"docs":{"total":59},"total":5},"stack":{"xpack":{"ccr":{"enabled":true,"available":true}}}},"name":"secondary"},"version":69}}}]}}}}
[2022-01-26T11:06:22.626+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-logstash-*,.monitoring-logstash-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"node_stats"}},{"term":{"type":"logstash_stats"}},{"term":{"metricset.name":"node_stats"}},{"term":{"metricset.name":"logstash_stats"}}]}},{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161882555,"lte":1643162782555}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"logstash_uuids":{"terms":{"field":"logstash_stats.logstash.uuid","size":10000},"aggs":{"latest_report":{"terms":{"field":"logstash_stats.timestamp","size":1,"order":{"_key":"desc"}},"aggs":{"memory_used":{"max":{"field":"logstash_stats.jvm.mem.heap_used_in_bytes"}},"memory":{"max":{"field":"logstash_stats.jvm.mem.heap_max_in_bytes"}},"events_in_total":{"max":{"field":"logstash_stats.events.in"}},"events_out_total":{"max":{"field":"logstash_stats.events.out"}}}},"memory_used_per_node":{"max_bucket":{"buckets_path":"latest_report>memory_used"}},"memory_per_node":{"max_bucket":{"buckets_path":"latest_report>memory"}},"events_in_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_in_total"}},"events_out_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_out_total"}}}},"logstash_versions":{"terms":{"field":"logstash_stats.logstash.version","size":10000}},"pipelines_nested":{"nested":{"path":"logstash_stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash_stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash_stats.pipelines.id"}}}}}},"pipelines_nested_mb":{"nested":{"path":"logstash.node.stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash.node.stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash.node.stats.pipelines.id"}}}}}},"events_in_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_in_total_per_node"}},"events_out_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_out_total_per_node"}},"memory_used":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_used_per_node"}},"memory":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_per_node"}},"max_uptime":{"max":{"field":"logstash_stats.jvm.uptime_in_millis"}}}}},"result":{"took":2,"timed_out":false,"num_reduce_phases":3,"_shards":{"total":2,"successful":2,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":44,"relation":"eq"},"max_score":0,"hits":[]},"aggregations":{"max_uptime":{"value":345234},"logstash_uuids":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","doc_count":34,"latest_report":{"doc_count_error_upper_bound":0,"sum_other_doc_count":33,"buckets":[{"key":1643162697408,"key_as_string":"2022-01-26T02:04:57.408Z","doc_count":1,"events_in_total":{"value":346},"memory":{"value":1037959168},"events_out_total":{"value":346},"memory_used":{"value":423274752}}]},"memory_used_per_node":{"value":423274752,"keys":["2022-01-26T02:04:57.408Z"]},"memory_per_node":{"value":1037959168,"keys":["2022-01-26T02:04:57.408Z"]},"events_in_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]},"events_out_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]}}]},"pipelines_nested":{"doc_count":0,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]},"pipelines":{"value":0}},"pipelines_nested_mb":{"doc_count":34,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"persisted","doc_count":34,"num_pipelines":{"value":1}}]},"pipelines":{"value":1}},"logstash_versions":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"8.0.0","doc_count":34}]},"events_in_total":{"value":346},"events_out_total":{"value":346},"memory_used":{"value":423274752},"memory":{"value":1037959168}}}}}
[2022-01-26T11:06:22.628+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-es-*,.monitoring-es-*,metricbeat-*,*:metricbeat-*","size":2,"ignore_unavailable":true,"filter_path":["hits.hits._source.cluster_uuid","hits.hits._source.elasticsearch.cluster.id","hits.hits._source.cluster_state","hits.hits._source.elasticsearch.cluster.stats.state"],"body":{"query":{"bool":{"filter":[{"term":{"type":"cluster_state"}},{"terms":{"cluster_uuid":["28iKoFsvTJ6HEyXbdLL-PQ","PuRGGPeVR-akKtACT2yTCQ"]}}]}},"collapse":{"field":"cluster_uuid"},"sort":{"timestamp":{"order":"desc","unmapped_type":"long"}}}},"result":{}}}
[2022-01-26T11:06:22.632+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":["*:.monitoring-logstash-*",".monitoring-logstash-*","metricbeat-*","*:metricbeat-*","*:.monitoring-beats-*",".monitoring-beats-*","metricbeat-*","*:metricbeat-*","*:.monitoring-beats-*",".monitoring-beats-*","metricbeat-*","*:metricbeat-*","*:.monitoring-ent-search-*",".monitoring-ent-search-*","metricbeat-*","*:metricbeat-*"],"body":{"size":0,"terminate_after":1,"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}},{"bool":{"should":[{"terms":{"type":["logstash_stats","logstash_state","beats_stats","beats_state"]}},{"terms":{"event.dataset":["logstash.node.stats","logstash.node","beat.stats","beat.state"]}}]}},{"range":{"timestamp":{"format":"epoch_millis","gte":1643161882555,"lte":1643162782555}}}]}}}},"result":{"took":0,"timed_out":false,"terminated_early":true,"num_reduce_phases":3,"_shards":{"total":3,"successful":3,"skipped":0,"failed":0},"_clusters":{"total":2,"successful":2,"skipped":0},"hits":{"total":{"value":3,"relation":"eq"},"max_score":0,"hits":[]}}}}
[2022-01-26T11:06:22.634+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-logstash-*,.monitoring-logstash-*,metricbeat-*,*:metricbeat-*","size":0,"ignore_unavailable":true,"filter_path":["aggregations.nest.id.buckets","aggregations.nest_mb.id.buckets"],"body":{"query":{"bool":{"filter":[{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161882555,"lte":1643162782555}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"nest":{"nested":{"path":"logstash_stats.pipelines"},"aggs":{"id":{"terms":{"field":"logstash_stats.pipelines.id","size":1},"aggs":{"unnest":{"reverse_nested":{},"aggs":{"nodes":{"terms":{"field":"logstash_stats.logstash.uuid","size":1}}}}}}}},"nest_mb":{"nested":{"path":"logstash.node.stats.pipelines"},"aggs":{"id":{"terms":{"field":"logstash.node.stats.pipelines.id","size":1},"aggs":{"unnest_mb":{"reverse_nested":{},"aggs":{"nodes":{"terms":{"field":"logstash.node.stats.logstash.uuid","size":1}}}}}}}}}}},"result":{"aggregations":{"nest":{"id":{"buckets":[]}},"nest_mb":{"id":{"buckets":[{"key":"main","doc_count":34,"unnest_mb":{"doc_count":34,"nodes":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","doc_count":34}]}}}]}}}}}}
[2022-01-26T11:06:22.639+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-es-*,.monitoring-es-*,metricbeat-*,*:metricbeat-*","size":1,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"cluster_stats"}},{"term":{"metricset.name":"cluster_stats"}}]}},{"range":{"timestamp":{"format":"epoch_millis","gte":1643161882555,"lte":1643162782555}}}]}},"sort":[{"timestamp":{"order":"desc","unmapped_type":"long"}}]},"filter_path":["hits.hits._source.stack_stats.xpack.ccr","hits.hits._source.elasticsearch.cluster.stats.stack.xpack.ccr"]},"result":{"hits":{"hits":[{"_source":{"elasticsearch":{"cluster":{"stats":{"stack":{"xpack":{"ccr":{"enabled":true,"available":true}}}}}}}}]}}}}
[2022-01-26T11:06:22.640+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":"*:.monitoring-es-*,.monitoring-es-*,metricbeat-*,*:metricbeat-*","size":1,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"cluster_stats"}},{"term":{"metricset.name":"cluster_stats"}}]}},{"range":{"timestamp":{"format":"epoch_millis","gte":1643161882555,"lte":1643162782555}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"sort":[{"timestamp":{"order":"desc","unmapped_type":"long"}}]},"filter_path":["hits.hits._source.stack_stats.xpack.ccr","hits.hits._source.elasticsearch.cluster.stats.stack.xpack.ccr"]},"result":{}}}
[2022-01-26T11:06:22.704+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}/logstash/nodes","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":".monitoring-logstash-*,metricbeat-*","size":0,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"node_stats"}},{"term":{"type":"logstash_stats"}},{"term":{"metricset.name":"node_stats"}},{"term":{"metricset.name":"logstash_stats"}}]}},{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161882657,"lte":1643162782657}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"aggs":{"logstash_uuids":{"terms":{"field":"logstash_stats.logstash.uuid","size":10000},"aggs":{"latest_report":{"terms":{"field":"logstash_stats.timestamp","size":1,"order":{"_key":"desc"}},"aggs":{"memory_used":{"max":{"field":"logstash_stats.jvm.mem.heap_used_in_bytes"}},"memory":{"max":{"field":"logstash_stats.jvm.mem.heap_max_in_bytes"}},"events_in_total":{"max":{"field":"logstash_stats.events.in"}},"events_out_total":{"max":{"field":"logstash_stats.events.out"}}}},"memory_used_per_node":{"max_bucket":{"buckets_path":"latest_report>memory_used"}},"memory_per_node":{"max_bucket":{"buckets_path":"latest_report>memory"}},"events_in_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_in_total"}},"events_out_total_per_node":{"max_bucket":{"buckets_path":"latest_report>events_out_total"}}}},"logstash_versions":{"terms":{"field":"logstash_stats.logstash.version","size":10000}},"pipelines_nested":{"nested":{"path":"logstash_stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash_stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash_stats.pipelines.id"}}}}}},"pipelines_nested_mb":{"nested":{"path":"logstash.node.stats.pipelines"},"aggs":{"pipelines":{"sum_bucket":{"buckets_path":"queue_types>num_pipelines"}},"queue_types":{"terms":{"field":"logstash.node.stats.pipelines.queue.type","size":10000},"aggs":{"num_pipelines":{"cardinality":{"field":"logstash.node.stats.pipelines.id"}}}}}},"events_in_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_in_total_per_node"}},"events_out_total":{"sum_bucket":{"buckets_path":"logstash_uuids>events_out_total_per_node"}},"memory_used":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_used_per_node"}},"memory":{"sum_bucket":{"buckets_path":"logstash_uuids>memory_per_node"}},"max_uptime":{"max":{"field":"logstash_stats.jvm.uptime_in_millis"}}}}},"result":{"took":3,"timed_out":false,"_shards":{"total":2,"successful":2,"skipped":0,"failed":0},"hits":{"total":{"value":44,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"max_uptime":{"value":345234},"logstash_uuids":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","doc_count":34,"latest_report":{"doc_count_error_upper_bound":0,"sum_other_doc_count":33,"buckets":[{"key":1643162697408,"key_as_string":"2022-01-26T02:04:57.408Z","doc_count":1,"events_in_total":{"value":346},"memory":{"value":1037959168},"events_out_total":{"value":346},"memory_used":{"value":423274752}}]},"memory_used_per_node":{"value":423274752,"keys":["2022-01-26T02:04:57.408Z"]},"memory_per_node":{"value":1037959168,"keys":["2022-01-26T02:04:57.408Z"]},"events_in_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]},"events_out_total_per_node":{"value":346,"keys":["2022-01-26T02:04:57.408Z"]}}]},"pipelines_nested_mb":{"doc_count":34,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"persisted","doc_count":34,"num_pipelines":{"value":1}}]},"pipelines":{"value":1}},"pipelines_nested":{"doc_count":0,"queue_types":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]},"pipelines":{"value":0}},"logstash_versions":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"8.0.0","doc_count":34}]},"events_in_total":{"value":346},"events_out_total":{"value":346},"memory_used":{"value":423274752},"memory":{"value":1037959168}}}}}
[2022-01-26T11:06:22.704+09:00][INFO ][plugins.monitoring.monitoring] {"api_path":"/api/monitoring/v1/clusters/{clusterUuid}/logstash/nodes","referer_url":"http://kibana.127.0.0.1.ip.es.io:5601/app/monitoring","query":{"params":{"index":".monitoring-logstash-*,metricbeat-*","size":10000,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"type":"node_stats"}},{"term":{"type":"logstash_stats"}},{"term":{"metricset.name":"node_stats"}},{"term":{"metricset.name":"logstash_stats"}}]}},{"range":{"logstash_stats.timestamp":{"format":"epoch_millis","gte":1643161882657,"lte":1643162782657}}},{"bool":{"should":[{"term":{"cluster_uuid":{"value":""}}},{"bool":{"must_not":[{"exists":{"field":"cluster_uuid"}}]}}]}}]}},"collapse":{"field":"logstash_stats.logstash.uuid"},"sort":[{"timestamp":{"order":"desc","unmapped_type":"long"}}],"_source":["timestamp","@timestamp","logstash_stats.process.cpu.percent","logstash.node.stats.process.cpu.percent","logstash_stats.jvm.mem.heap_used_percent","logstash.node.stats.jvm.mem.heap_used_percent","logstash_stats.os.cpu.load_average.1m","logstash.node.stats.os.cpu.load_average.1m","logstash_stats.events.out","logstash.node.stats.events.out","logstash_stats.logstash.http_address","logstash.node.stats.logstash.http_address","logstash_stats.logstash.name","logstash.node.stats.logstash.name","logstash_stats.logstash.host","logstash.node.stats.logstash.host","logstash_stats.logstash.uuid","logstash.node.stats.logstash.uuid","logstash_stats.logstash.status","logstash.node.stats.logstash.status","logstash_stats.logstash.pipeline","logstash.node.stats.logstash.pipeline","logstash_stats.reloads","logstash.node.stats.reloads","logstash_stats.logstash.version","logstash.node.stats.logstash.version"]}},"result":{"took":3,"timed_out":false,"_shards":{"total":2,"successful":2,"skipped":0,"failed":0},"hits":{"total":{"value":44,"relation":"eq"},"max_score":null,"hits":[{"_index":".ds-metricbeat-8.0.0-2022.01.25-000001","_id":"EVYilH4BEep1J2kYOhlb","_score":null,"_source":{"@timestamp":"2022-01-26T02:06:17.403Z"},"sort":[1643162777403]},{"_index":".ds-.monitoring-logstash-8-mb-2022.01.25-000001","_id":"U1YhlH4BEep1J2kYAQXh","_score":null,"_source":{"logstash":{"node":{"stats":{"jvm":{"mem":{"heap_used_percent":40}},"logstash":{"pipeline":{"batch_size":125,"workers":16},"http_address":"127.0.0.1:9600","name":"matschaffer-mbp2019.lan","host":"matschaffer-mbp2019.lan","version":"8.0.0","uuid":"3a0d44cb-cbe7-46be-88fd-b46eaac6941a","status":"green"},"process":{"cpu":{"percent":0}},"os":{"cpu":{"load_average":{"1m":3.44140625}}},"events":{"out":346},"reloads":{"failures":0,"successes":0}}}},"@timestamp":"2022-01-26T02:04:57.408Z"},"fields":{"logstash_stats.logstash.uuid":["3a0d44cb-cbe7-46be-88fd-b46eaac6941a"]},"sort":[1643162697408]}]}}}}

Any additional context:

Need to check if this also happens on main.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

Looks like main works, so labeling this as 8.0.0

[matschaffer]

Bummer. 7.16.3 works so this is a new bug effecting just 8.0

[matschaffer]

Ah. 8.0 works on internal collection (the above 7.16.3 test was internal, the initial bug used metricbeat).

Will try to reproduce it on 7.16.3 with metricbeat collection. Sort of wondering now if it was some data issue in my test setup as well.

[matschaffer]

Re-tested on 7.16.3 from source and it worked fine. So this is some sort of 8.x regression.

[matschaffer]

Okay, I see what's happening here. The logstash nodes query is hitting .monitoring-logstash-*,metricbeat-* for standalone (no uuid) which is picking up docs like these:

{
  "_index" : ".ds-metricbeat-8.0.0-2022.01.31-000001",
  "_id" : "H_zqrn4Bkd71XbVhraxx",
...
  "_source" : {
    "@timestamp" : "2022-01-31T06:55:21.475Z",
...
    "error" : {
      "message" : """error making http request: Get "http://localhost:9600/": dial tcp 127.0.0.1:9600: connect: connection refused"""
    },
...
  }
}

They're not logstash docs so they understandably don't have a UUID.

[matschaffer]

main doesn't have this problem since we've removed the metricbeat-* part of the query