Stack monitoring - standalone metricbeat 8.0 compatibility

[klacabane]

This ticket tracks the action items necessary to make the Stack monitoring UI compatible with metricbeat standalone 8.x.

Steps to setup an environment with the new templates:

• Start an 8.0 elastic snapshot and a local kibana instance
• clone this repo and run npm run push-templates. this will setup the new index templates in the running elasticsearch. This step is only necessary if Adding default templates for Metricbeat ECS data elasticsearch#81744 isn't available yet or if you want to quickly iterate over mapping changes
• start metricbeat with appropriate modules enabled, make sure your build includes these changes (1, 2)
• kibana needs to be patched, I've put the changes here
• update stack monitoring to read from the new index patterns .monitoring-{product}-8-*

Action items:

• Adding default templates for Metricbeat ECS data elasticsearch#81744
• Update code relying on hardcoded .monitoring template versions #121251
• Stack monitoring.elasticsearch - 8.0 compatibility #121282
• Stack monitoring.beats - 8.0 compatibility #121518
• Stack monitoring.kibana - 8.0 compatibility #121289
• Stack monitoring.logstash - 8.0 compatibility #121346
• Stack monitoring.entsearch - 8.0 compatibility #121975
• [Stack Monitoring] Uncaught exception attempting to load standalone logstash nodes page #123785
  • Ensure logstash getNodes always contains a uuid #124201
  • [8.0] Ensure logstash getNodes always contains a uuid (#124201) #124319

Related items:

• [metricbeat] beats xpack module creates a single datastream per service monitored beats#29657 - we should probably avoid releasing while daily data streams are being created per advice from @jakelandis
• New .monitoring-8 templates need to be automatically cleaned over time elasticsearch#81839 - we need some kind of clean up process (probably an ILM policy) for monitoring-8 indices
  • Upgrade documentation for xpack.monitoring.history.duration elasticsearch#82295
• Unable to ingest json logs using elasticsearch filebeat module beats#29880

AC:

• stack monitoring UI is powered by data written in .monitoring-{product}-8 data streams
• UI parity with data generated by standalone metricbeat 7.x and metricbeat 8.x
• existing _mb functional tests should be updated to write their respective archives to the new .monitoring-{product}-8 datastreams
• Reasonable coverage on Test Plan

---

Quick test of the metricbeat 8.0 standalone mode that naively merges the individual elasticsearch integration mappings into a new index template.

Observations:

• UI loads in main branch but the datastream-compatible changes shows a No Data screen. This is most likely because we removed filter terms for metricbeat-* documents and documents inserted in standalone mode don't have the data_stream.dataset, or the legacy type property
• Indices tab does not load any data which is most likely related to this issue
• Nodes overview displays N/A for CPU usage and CPU throttling when it shows data with the documents inserted by the elastic agent
• Node details page fail to load with

  TypeError: Cannot read properties of undefined (reading 'key')
      at reduce (/Users/lacabane/workspace/kibana/x-pack/plugins/monitoring/server/lib/elasticsearch/shards/normalize_shard_objects.ts:51:44)
      at Array.reduce (<anonymous>)
      at handleResponse (/Users/lacabane/workspace/kibana/x-pack/plugins/monitoring/server/lib/elasticsearch/shards/get_shard_stats.ts:46:46)
  

The last two points are probably related to the naive property merge, I'll have a look.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[klacabane]

Regarding

Node details page fail to load with

TypeError: Cannot read properties of undefined (reading 'key')
    at reduce (/Users/lacabane/workspace/kibana/x-pack/plugins/monitoring/server/lib/elasticsearch/shards/normalize_shard_objects.ts:51:44)
    at Array.reduce (<anonymous>)
    at handleResponse (/Users/lacabane/workspace/kibana/x-pack/plugins/monitoring/server/lib/elasticsearch/shards/get_shard_stats.ts:46:46)

The query returns an empty bucket for the node_names and that's, I assume, because the source_node.name which aliases to elasticsearch.node.name is not set in the shard documents produced by metricbeat 8, only the source_node.uuid. I confirmed that documents produced by metricbeat 7 did have the property set.

---

We fixed this in the datastream compatibility PR.

Applying the same fix to main branch solves the issues related to Node's missing CPU data and the Node details page not loading. So the merge was actually successful as far as the smoke test goes

[klacabane]

I ran a diff of the template generated from the datastream mappings and the one built manually. One thing to consider is that we don't have a beats integrations package so we're missing the relevant properties in the template constructed from datastreams mappings, namely beat, beats_state and beats_stats.

There are other differences, like missing index_stats which was already known, the event root property and some property types like elasticsearch.cluster.pending_task.priority being a long or a keyword.

Full diff

[klacabane]

Created the templates here and currently going through some manual tests:

• Elasticsearch data shows up
• Kibana data does not show up while documents are being indexed. It also fails with the manual mappings. It could be related to the elasticsearch.cluster.id not being set. The indexed stats documents have the property under kibana.elasticsearch.cluster.id which looks like a definition error in metricbeat ?
  Should the property be assigned to RootFields instead of ModuleFields here ? cc @sayden
• Logstash - not tested yet
• Beats - similar to kibana where it could be caused by documents missing elasticsearch.cluster.id property. Also it seems that we never had _mb tests for beats,

[klacabane]

Going through Kibana views powered by the new templates there are missing aliases (on both agent and metricbeat-generated template) which fail queries, and wrong assumptions about the data structure of the metricbeat-generated data. MB queries expect a nested kibana object to retrieve details but it does not exist in main version (a similar property was deleted ~8months ago from metricbeat which is possibly the cause).

[klacabane]

Got Beats to show up after fixing missing elasticsearch.cluster.id property on both stats/state metricsets and fixing some aliases as well. The list starts to grow, I'll cut a separate issue to track the local patches. These patches impact the SM server code, metricbeat and index templates.
The good thing about working backwards from the datastreams templates is that we'll also fix the integration packages by testing metricbeat standalone, that'll give us more time to implement functional tests :)

[klacabane]

Regarding Logstash section, we're looking for stats or logstash_stats metricset names but the metricbeat's logstash module only produces node and node_stats, did that also change since we initially implemented mb support ?

[jasonrhodes]

@klacabane can we close this or convert it to "type: epic" if it's main goal is to track other work?

[matschaffer]

I think maybe it's time for a general Stack Monitoring 8.0 testing epic?

I have two issues (added to the description) that are still outstanding.

These aren't specifically related to the 7->8 metricbeat changes (I think), but it's hard to say "pass" until we get a clean manual test which we haven't had yet (sheet).

[klacabane]

Let's close this once #123785 is fixed and move to a clean epic for the followup testing

[matschaffer]

Cool. I opened #124312 for follow up testing.