ILM reports error illegal_argument_exception

[slawekww]

Elasticsearch version : 7.2.0
Installed by eck-operator: 0.9.0-rc7

Plugins installed: as installed by eck-operator version 0.9.0-rc7
docker.elastic.co/elasticsearch/elasticsearch:7.2.0

JVM version (java -version): as used in elasticsearch docker image

OS version (uname -a if on a Unix-like system): Linux sworkstation 4.15.0-55-generic

Description of the problem including expected versus actual behavior:
The following options are used for deploying metricbeat helm chart - overwritten values below.

ILM reports the following error:
illegal_argument_exception: index.lifecycle.rollover_alias [metricbeat] does not point to index [metricbeat-nfs-provisioner-2019.08.09]

Steps to reproduce:

1. install elasticsearch 7.2.0 using eck-operator at elasticsearch namespace.
   Default install with 2 nodes.

2. install metricbeat using helm chart: stable/metricbeat with the following overwrite myvalues.yml:
   helm install -f myvalues.yml --name metric --namespace elasticsearch stable/metricbeat

myvalues.yml - > rename txt to yml
myvalues.txt

3. Wait 2-5min to see error on kibana. It is not seen immediately.

[elasticmachine]

Pinging @elastic/es-core-features

[gwbrown]

This line right here is the problem:

        - index: "metricbeat-%{[kubernetes.pod._module.namespace]}-%{+yyyy.MM.dd}"

That setting in Beats generally doesn't play well with ILM - this is something that should probably be addressed in Beats. When Beats configures ILM, it does so based on the ILM configuration (see below). When the output index is manually set to something other than the alias that Beats set up, it will cause errors similar to this.

The proper way to change the Beats output when using ILM can be found in the Configure index lifecycle management page in the Beats documentation.

I've answered this here because this is something that looks on the surface like it could be a bug in our products, but for more help on this problem please post to the forums, where we typically provide configuration assistance and troubleshooting, as we try to keep GitHub issues reserved for bugs and feature requests.

Because this is not an Elasticsearch bug (although it is arguably a Beats usability bug), I hope you don't mind that I close this issue.