Beats require unnecessary privileges at startup on 7.13

[iorfix]

Using version 7.13.1 Beats force to set ILM policies, even when configuration explicitly set them to don't set them.
This may cause an error, if beats credentials are reduced to minimum, as in: https://www.elastic.co/guide/en/beats/filebeat/current/privileges-to-publish-events.html

• Version: 7.13.1 (it doesn't affect 7.10.2)
• Operating System: CentOs / Redhat
• Discuss Forum URL: https://discuss.elastic.co/t/heartbeat-ask-for-unnecessary-cluster-privileges-at-startup/275814
• Steps to Reproduce:

1. create a user with this privileges, and use it in a metricbeat.yml configuration:

{
  "cluster": ["monitor" ,"cluster:admin/ingest/pipeline/get"],
  "indices": [
    {
      "names": [ "cloud-metrics*", "cloud-logs*", "metricbeat-*", "filebeat-*", "heartbeat-*" ],
      "privileges": ["create_doc"]
    }
  ],
  "metadata" : { 
    "version" : 1
  }
}

2. execute metricbeat setup with proper superuser credentials.

metricbeat setup -E setup.ilm.overwrite=true -E setup.template.enabled=true -E output.elasticsearch.username=elastic -E output.elasticsearch.password=***

3. configure metricbeat.yml with these settings:

setup.template.enabled: false
setup.ilm.check_exists: false
setup.ilm.overwrite: false
setup.ilm.enabled: true

4. start metricbeat and check for logs. It should appear something similar to:

{"log.level":"error","@timestamp":"2021-06-14T11:06:35.785+0200","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":166},"message":"ILM policy heartbeat creation failed: 403 Forbidden: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ilm/put] is unauthorized for API key id [0e0c9nkBdKUEp4Eqeb4y] of user [apikeywriteruser], this action is granted by the cluster privileges [manage_ilm,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ilm/put] is unauthorized for API key id [0e0c9nkBdKUEp4Eqeb4y] of user [apikeywriteruser], this action is granted by the cluster privileges [manage_ilm,manage,all]\"},\"status\":403}","ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ph]

@iorfix This was working as is in 7.10.2?

[iorfix]

It works as expected with 7.10.2, but it doesn’t work in 7.13.1

[4urb4nm0nk]

I still have this issue on 7.14 with all Beats when using API for authentication. I followed this guide. I know it's a open issue and I'll be watching for any changes.
I even get it when I change the authentication to the elastic user. So it builds the index and then goes back to API authentication. But I don't want to do this for every Beats every time I update or when the index reaches 50GB or 30 days.

[ruflin]

@kvch This seems to be related to #27270 ?

[kvch]

Have you tried setting setup.ilm.enabled to false so ILM is not loaded and output.elasticsearch.index to your index name so the Beat knows what is the target index?

[MrMMorris]

still seeing this issue with metricbeat 7.17.0.

used this to create api key:
https://www.elastic.co/guide/en/beats/metricbeat/7.17/beats-api-keys.html

get this error:
ILM policy metricbeat creation failed: 403 Forbidden: {"error":{"root_cause"[{"type":"security_exception","reason":"action [cluster:admin/ilm/put] is unauthorized for API key id [xxxxxxxxxxxxxx] of user [elastic], this action is granted by the cluster privileges [manage_ilm,manage,all]"}]

EDIT: even after updating the api key permissions to add the above, I now get this error:
action [indices:admin/index_template/put] is unauthorized for API key id [xxxxxxxxxxxx] of user [elastic], this action is granted by the cluster privileges [manage_index_templates,manage,all]