ILM: Change of setup.ilm.pattern is not applied even with setup.ilm.overwrite set.

[adriansr]

For confirmed bugs, please report:

• Version: 7.10.2
• Operating System: -
• Discuss Forum URL: -
• Steps to Reproduce:

Contrary to our documentation, changes to setup.ilm.pattern doesn't apply if the alias already exists. It keeps using the previous pattern.

Run a Beat with:

setup.ilm.enabled: true
setup.ilm.rollover_alias: "myindex5"
setup.ilm.overwrite: true
setup.ilm.pattern: "000001"

This works and the alias myindex5 => myindex5-000001 gets created.

Now replace the pattern:

  setup.ilm.enabled: true
  setup.ilm.rollover_alias: "myindex5"
  setup.ilm.overwrite: true
- setup.ilm.pattern: "000001"
+ setup.ilm.pattern: "{now/M{yyyy.MM}}-000001"

Run beatname setup --index-management and then ingest more documents.

The Beat is still indexing using the previous pattern, and the write alias is still myindex5 => myindex5-000001.

If the alias is removed before running setup, then the correct alias is created, myindex5 => myindex5-2021.02-000001.

Documentation states:

[adriansr]

In the code I can see that the overwrite setting only affects the installation of a new ILM policy (EnsurePolicy at libbeat/idxmgmt/ilm/std.go).

The code that handles the alias creation (EnsureAlias in the same file), will only create a new alias if it doesn't exists already, the actual pointed-to index is not taken into account.

I'm not sure it's easy to fix this, because the original pattern used to create an alias is lost, in my tests if the alias is replaced for a pattern, it will reset the counts, starting with -000001 every time a Beat is restarted.

/cc @urso @simitt

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[urso]

An alias can not be created independently. For ILM setup we first need to create the ILM policy, then upload the template, and finally create the first index with the associated alias. The indices are now under control of Elasticsearch ILM and should not be updated. The code only checks if the alias exist, not that the current associated index matches the given pattern (which is somewhat Elasticsearch specific).

I'm not sure it's easy to fix this, because the original pattern used to create an alias is lost, in my tests if the alias is replaced for a pattern, it will reset the counts, starting with -000001 every time a Beat is restarted.

Correct. One must create the first index manually. The setup code assumes that your "setup" is new. We do not read the old associated index name. There is also no logic to check if the current name matches the index naming pattern (which syntax is specific to ES).
Changing the pattern is more similar to creating a new set of indices. If someone wants to change the pattern, there might be a need to rename existing indices as well.

The setup command aims at providing an initial setup to have the stack prepared before publishing events. But there are limits to it. setup is no full stack management solution.

Something we might want to consider is to reduce the "burden" by allowing users to create datastreams instead of indices + aliases (which is somewhat fragile at times). Datastreams kinda unifies the experience by making sure that Elasticsearch handles associated resource internally. One can use ES API or Kibana to apply changes to a Datastream. For the Beats setup this means that we would not need to setup an alias at all. Instead we just add "datastream": {} to the template.

Note: The ILM pattern configures the index naming, not the alias. At worst changing the ILM pattern might interfere with ILM, such that old indices are not correctly handled anymore. Once the alias exists, ILM is in full control of the indices.

[adriansr]

Thanks for chiming in, @urso.

Then I understand the best we can do at this time is to update the docs to make this more clear? Instead of

NOTE: If you modify this setting after loading the index template, you must overwrite the template to apply the changes.

can we suggest to remove the existing aliases and remind the user that old indices under the previous pattern won't we subject to ILM anymore?

[urso]

can we suggest to remove the existing aliases and remind the user that old indices under the previous pattern won't we subject to ILM anymore?

Sure.

s/won't/might/