Update of component templates can trigger data stream mapping updates

[ruflin]

If an Elasticsearch component template is updated, a rollover of any data stream that this template applies to is needed to get the new mappings and settings. In the case of Fleet with the data stream naming scheme updates to templates happen each time a package is upgraded. Instead of just rolling over indices each time, Fleet checks if the mapping of the index that is written to can be updated as in most scenarios only additional mappings are added. This makes sure less indices / shards are created.

One of the challenges is that with the namespace concept Fleet has to find all data streams first and update each one individually. It would be nice if this could be done by Elasticsearch directly.

Use cases

Above the use case of a package upgrade is described but there are more I want to list here:

Custom template updates

In 7.14 Fleet adds a {type}-{dataset}@custom component template foreach each type/dataset combination. This allows users to add their own settings or additional mappings. As these templates are modified by users, Fleet does not know when to update the data streams. Instead the user has to do it manually which is not ideal.

Global settings

Currently when a user modifies a component template which might be used by multiple index templates this has no direct effect. All data streams that use this component templates need to be rolled over. An example here is if the ILM policy for all logs should be changed, there is no easy way yet to apply this.

Base templates

When the base templates of Elasticsearch are updated the new mappings / settings / ingest pipelines are only applied on rollover. Instead it would be nice if this would happen automatically.

How it could work

In the following I want to share some ideas on how it could work, this is not meant as a proposal on implementation but to better describe expected behaviour.

As this "auto update" feature is a bit magical I think there should be an option to opt-in or opt-out. Either a setting on the index template should exist or the data stream if updates should be applied automatically. It could also exist in both places where it can be set in the template itself first but overwritten per data stream too to disable it.

Let's assume we have an index template logs-nginx.access with a component template logs-nginx.access@mappings. If the component template is updated and in the index template logs-nginx.access auto-update is enabled, Elasticsearch will search for all the data streams that use this template. At first, Elasticsearch will try to apply the new mapping to the data streams and in case this fails, it will trigger a rollover of the data streams.

Additional thoughts

Currently Fleet treats all updates to mappings or settings the same. But in reality these are not necessarily the same. An update to the number of shards or ILM might require a rollover, other settings don't. Having this logic built into Elasticsearch directly, Elasticsearch could decided what the right logic is.

[dgieselaar]

For RAC, we'd love to be able to use this for aliases as well - we can't use data streams due to an update use case. See the discussion in elastic/kibana#102586.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)