[Stack monitoring] Add agent-compatible data stream patterns to data fetching patterns

[jasonrhodes]

We currently search for data in .monitoring-* and metricbeat-*, but once packages are an option, we'll need to search against the correct metrics-{dataset}-* data stream pattern(s). And because we no longer plan to support metricbeat-* indices, we should remove that pattern from the search.

AC:

• metricbeat-* is removed from the search pattern for Stack Monitoring queries
• metrics-* is added to the search pattern for Stack Monitoring queries -- there are two options for how to do this correctly
  • Option 1: we use the correct datasets for the patterns (metrics-{dataset}-*)
  • Option 2: we search metrics-* but use a "constant_keyword" field value as part of the query
• Understand what changes are needed for functional tests and fix or log a ticket to update (note: esarchiver has had issues with data streams in the past)

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[weltenwort]

Do we need to add logs-* anywhere?

The ui.logs.index config setting defaults to filebeat-*, which is apparently used in the registration of the internal log source:

kibana/x-pack/plugins/monitoring/server/config.ts

Line 32 in e5954a6

index: schema.string({ defaultValue: 'filebeat-*' }),

[simianhacker]

I started digging into this issue, there are a few things I think we need to decide before moving forward...

1. Should we remove the appendMetricbeatIndex functions? I think it's unnecessary complexity when we should just add metrics-monitoring-* to the current constants that define the .monitoring-* index patterns.
2. Should we remove the mbSafeQuery? The main issue I have with this function is that it's original intent was to deal with mapping issue from legacy metricbeat-* indices, which we are no longer supporting. This function hides mapping errors when customer's mappings get messed up in .monitoring-* indices.
3. What should we do with all the _mb tests? These are loading into metricbeat-* indices, which we are no longer supporting.
4. Should we remove the UI config for the Metricbeat indices? Probably

[matschaffer]

@jasonrhodes just a heads up that this seems to be a theme in SDHs. Guessing we'll have to address https://github.com/elastic/observability-dev/issues/1718 before we work on moving to metrics-* and removing metricbeat-* from the queries.

[jasonrhodes]

@jasonrhodes just a heads up that this seems to be a theme in SDHs. Guessing we'll have to address elastic/observability-dev#1718 before we work on moving to metrics-* and removing metricbeat-* from the queries.

@matschaffer not sure I follow? We can remove metricbeat-* anytime, but yes we will have to solve the issue you mention before we can add metrics-*