Functionbeat requires manage_index_templates cluster permission with Elasticsearch 7.16

[Urokhtor]

Elasticsearch version (bin/elasticsearch --version): 7.16 (Elastic Cloud)

Description of the problem including expected versus actual behavior:

After Elasticsearch was upgraded in Elastic Cloud to 7.16, our Functionbeat stopped working. We see following error in our logs:

2021-12-08T06:42:16.576Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(https://:9243)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 403 Forbidden: 
{
    "error": {
        "root_cause": [
            {
                "type": "security_exception",
                "reason": "action [indices:admin/template/get] is unauthorized for user [functionbeat] with roles [functionbeat], this action is granted by the cluster privileges [manage_index_templates,manage,all]"
            }
        ],
        "type": "security_exception",
        "reason": "action [indices:admin/template/get] is unauthorized for user [functionbeat] with roles [functionbeat], this action is granted by the cluster privileges [manage_index_templates,manage,all]"
    },
    "status": 403
}

This issue can circumvented by adding the manage_index_templates to the role. However, the question is why should Functionbeat need that privilege? Isn't this overly permissive? Why should Functionbeat be able to manage index templates? It's also not mentioned by Functionbeat 7.16 documentation, so I'm not sure whether this functionality is intended.

Steps to reproduce:

Set up Elasticsearch 7.16 and try to index something with Functionbeat without manage_index_template privilege. Version does not seem to matter.

[kaangoksal]

Same issue with filebeats

2021-12-11T15:58:33.739Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(https://xxx.io)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/get] is unauthorized for API key id [xxx] of user [xxx], this action is granted by the cluster privileges [manage_index_templates,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/template/get] is unauthorized for API key id [xxx] of user [xxx], this action is granted by the cluster privileges [manage_index_templates,manage,all]"},"status":403}
2021-12-11T15:58:33.739Z	INFO	[publisher_pipeline_output]	pipeline/output.go:145	Attempting to reconnect to backoff(elasticsearch(https://xxx.io)) with 2 reconnect attempt(s)

given the latest log4j RCE having overly permissive security configurations are getting more dangerous than ever.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[jtibshirani]

Thanks for your feedback. I've tagged our security team for their context. However, I think this change may be specific to Beats and not Elasticsearch. We might need to transfer the issue to the Beats repo (https://github.com/elastic/beats) to hear from the experts there.

[kaangoksal]

I can provide more details if needed. Have a setup that i can demonstrate the issue

[tvernum]

We've tracked this down to being triggered by elastic/elasticsearch#78832 and the fact that libbeat uses that Cat API in a way that wasn't anticipated.
We're looking at options for resolving it.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[falkowich]

Hi,

This can really wreak havoc when updating to es 7.16+ with older clients (like what is recommended in the upgrade guide).
If we not are adding "manage_index_templates" all beats (except winlogbeat?) starts to error log on the agenthosts and if you have a few hundred servers this can be really bad.

I think that this has to be added to the upgrade documentation.
We got the problem in "test", but I guess that there are users that don't try this out in test first..

--
Regards Falk

[kaangoksal]

If someone encounters this problem, this privelege structure works currently.

{
    "filebeat_writer": { 
      "cluster": ["monitor", "read_ilm", "read_pipeline", "manage_index_templates"],
      "index": [
        {
          "names": ["filebeat-*"],
          "privileges": ["view_index_metadata", "create_doc", "create_index", "manage"]
        }
      ]
    }
  }

The one in the documentation didn't work for me

https://www.elastic.co/guide/en/beats/filebeat/current/beats-api-keys.html

I'm going to give 8.xx a try to see whether this is fixed, there are couple of PRs linked to this issue.

[Nirusu]

From my experience, 8.0.0 seems to work fine without manage_index_templates after the index has been initialized previously. So I guess this has been fixed.

[jlind23]

Thanks @Nirusu for reporting it, closing it for now then.

[kaangoksal]

after a year, i still see the same problem

 {"log.level":"error","@timestamp":"2023-06-19T04:31:58.996Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://elastic.<removed>.com:9200)): Connection marked as failed because the onConnect callback failed: error loading template: failed to load template: couldn't load template: 403 Forbidden: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:admin/index_template/put] is unauthorized for user [user_filebeat] with effective roles [filebeat_writer], this action is granted by the cluster privileges [manage_index_templates,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:admin/index_template/put] is unauthorized for user [user_filebeat] with effective roles [filebeat_writer], this action is granted by the cluster privileges [manage_index_templates,manage,all]\"},\"status\":403}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:admin/index_template/put] is unauthorized for user [user_filebeat] with effective roles [filebeat_writer], this action is granted by the cluster privileges [manage_index_templates,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:admin/index_template/put] is unauthorized for user [user_filebeat] with effective roles [filebeat_writer], this action is granted by the cluster privileges [manage_index_templates,manage,all]\"},\"status\":403}","service.name":"filebeat","ecs.version":"1.6.0"}

elasticsearch 8.8.1
docker.elastic.co/beats/filebeat:8.8.1