[Observability discussion] what's the DEAL with index patterns

[jasonrhodes]

Following on from #39115 (and related to the ongoing discussion in #35481), we are running into more and more index pattern related issues inside observability. This latest one makes links from metrics explorer to TSVB/visualize fail because visualize as an app requires at least one (any) index pattern to be present or it redirects you to the index pattern creation screen.

TSVB, however, allows you to specify an Elasticsearch "index pattern" (i.e. a string representing the pattern of an ES index) for your query, without needing to make use of a Kibana index pattern (i.e. the persisted field mapping abstraction).

What we've tried

We considered auto-creating an index pattern similar to how APM does here: https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/apm/server/lib/index_pattern/index.ts but APM is storing their index pattern config in Kibana because the APM setup instructions also create the index pattern from within Kibana. The metricbeat index pattern, however, is created in Go inside the beats repo:
https://github.com/elastic/beats/blob/master/libbeat/cmd/export/index_pattern.go#L57
https://github.com/elastic/beats/blob/master/libbeat/kibana/index_pattern_generator.go

If we tried to create the same index pattern in Kibana, we would need a way to keep those configs in sync so that the index pattern created during setup is the same as the one created by us in the UI -- they would each also need to be careful not to create duplicate index patterns (not sure if this is actually possible by ID? overwrite: false may solve it if the same ID is used, to prevent an update).

What next?

Short term: for now we have opted to leave the experience as-is in the metrics explorer (#39115 (comment)), so if a user ingests data but hasn't run the "setup" command in metricbeat, when they click a graph's "View in visualize" action, they'll be redirected to the "create index pattern" page. We should decide on the below questions before investing time in fully solving this, I think.

Longer term: We need to either a) solve the problem of creating an index pattern, or b) work with the Kibana team to remove the restriction around requiring an index pattern. Thoughts on how to proceed?

Support tickets regarding index patterns
#48674

[elasticmachine]

Pinging @elastic/apm-ui

[elasticmachine]

Pinging @elastic/uptime

[elasticmachine]

Pinging @elastic/infra-logs-ui

[graphaelli]

APM uses this script hooked up to CI to ensure the index patterns stay in sync.

[simianhacker]

How about "Dymanic Index Patterns" for Kibana? Something we can inject in the URL so it appears like it has an index pattern but doesn't really exist.

[sorenlouv]

Longer term: We need to either a) solve the problem of creating an index pattern, or b) work with the Kibana team to remove the restriction around requiring an index pattern. Thoughts on how to proceed?

Has it been explored whether the index pattern can be derived from the elasticsearch mapping on the fly? If the index pattern is dynamically generated and not persisted we won't have problems with it coming out of sync as the underlying data and mapping changes.

[sorenlouv]

Btw. really glad you bring up this issue. I've spend an inordinate amount of time fixing problems related to index patterns and I would be very happy to see them being dropped as a foundational requirement in Kibana.

[jasonrhodes]

@elastic/apm-ui + @elastic/infra-logs-ui can any of you elaborate on the issues and problems caused by index patterns and why you'd prefer to remove the requirement? It'd be nice to list our concerns on the main Kibana index pattern issue (#35481) so I'd like to understand them and summarize them over there. Thanks!

[weltenwort]

I'll start with a small list from my perspective here before we push it upstream to the other issue:

• lack of additional metadata on the index and field level: We store additional metadata with the index patterns, e.g. about the semantics of certain fields. The index patterns don't allow for that.
• index patterns are hard to keep in sync with the mappings: The index patterns need to be manually refreshed and don't automatically adapt to changes in the indices and their mappings.
• implementation detail exposed to the user: The "internal" index patterns are exposed to the user. They can therefore break the solutions in various ways by messing with them.
• life-cycle mangement/migrations very difficult: Performing migrations from one version to the next is nearly impossible, because the user might have re-purposed them for other visualizations or dashboards.

[sorenlouv]

I think @weltenwort put it really well.

On APM we've mostly run into the problem with keeping the index patterns in sync with mappings. This can happen when a user upgrades their stack. Since index patterns are not automatically migrated they'll not longer match the mappings of the new/migrated data. Last time I checked we didn't have any good ways of solving this.

APIs that required index patterns
Certain Kibana endpoints require index patterns to be present. An example is the endpoint for creating ML jobs. To get ML to analyze data the user must therefore first create an index pattern that matches the specific indices before calling the endpoint. It would be great to avoid requirements like these.

Naming
The name "index pattern" is both ambiguous and misleading. Naming is often neglected but it is difficult to communicate about these things when we don't have accurate names to identify them.
AFAICT outside Kibana an "index pattern" describes a string, that matches one or more indices eg "apm-*".
However, in Kibana an index pattern is an object that lists the mappings of one or more indices. This object happens to have a title property which is a string that acts like what an outsider would call an "index pattern" but in Kibana should be referred to as an index pattern title 😩
A more accurate name for kibana index patterns would probably be "cached index mappings" or "index metadata object" - or really just anything else than "index pattern".

[sorenlouv]

Oops, did I close this issue? Sorry, re-opening!
(moving things around in Zube sometimes has unintended consequences)

[sorenlouv]

Perhaps of interest to people here: APM currently has a hard dependency on index patterns due to the way kql works. I've started an issue around creating a "fake index pattern" to avoid the dependency on actual index patterns, and gain some more flexibility: #44487

[sorenlouv]

Fyi: I've finally bit the bullet and created an issue to rename Kibana Index Patterns: #44955

[andrewvc]

Removed the uptime label since we no longer use index patterns.