[7.x] Prefer hot data tiers

[jakelandis]

Starting in 7.16 Beats started using a composable index templates by default. Composable index templates will always take precedence over legacy index templates, meaning that an index patterns that match both composable and legacy templates, the legacy templates will be ignored.

Normally this is fine and generally desired. However, prior to this change it was possible to declare a legacy index template to match on * (i.e. all indices) to default index settings to ensure that indices were first allocated on the hot node. Since Beats are now using composable index templates this type of pattern will no longer work. For systems which relied on that * pattern for hot allocation, they will now loose the ability to default to hot nodes.

I would like to request the following index setting be the default for Beats 7.x :

"index.routing.allocation.include._tier_preference" : "data_hot,data_content"

This says prefer data_hot if present, else use data_content. If no node roles are configured for data tiers this is still OK and won't break anything since every node will logically get every data_* node role. However, if node roles are present then with this change new indices will prefer hot nodes.

This is not necessary for 8.x defaults since 8.x Beats default to use data streams and data streams have an implicit default of data_hot.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

@nimarezainia @cmacknz do you think we should make it land in 7.17?

[cmacknz]

The change seems trivial to make, it just a question of having some available to do it.

Most of the effort would be in testing to confirm that the setting is applied as expected, and perhaps confirming that there are no backwards compatibility concerns as the description states.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[VimCommando]

This is still relevant, especially given 7.17 will be supported until 9.x is released; plus 7.17 is the recommended upgrade path to 8.x. This update helps smooth that transition.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!