add privileges for kIbana_system user to serve cloud security posture…

[CohenIdo]

Add privileges for kibana_system user to allow support in Cloud Security Posture's feature.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[joshdover]

Do we need delete? I don't see it as a requirement for the transform APIs nor do I expect Kibana to be deleting documents from these indices

[CohenIdo]

in the doc, it mentioned the required privileges are:
source indices: read, view_index_metadata
destination index: read, create_index, index
while when the Transform is initiated with those privileges we are getting the next error message:

task encountered irrecoverable failure: org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/delete/byquery] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [cloud_security_posture-findings_latest], this action is granted by the index privileges [delete,write,all]

As a solution, I added the following privileges: [delete,write,all]

[joshdover]

@elastic/ml-data (I hope this is the right team) are our docs out of date here? I'm surprised that installing a transform is attempting a delete by query on the destination index and that the docs don't indicate this. I'm also surprised this issue doesn't affect our other transforms installed by Fleet?

task encountered irrecoverable failure: org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/delete/byquery] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [cloud_security_posture-findings_latest], this action is granted by the index privileges [delete,write,all]

That error seems to indicate that they need one of these [delete,write,all] privileges, but not all of them. Can we test what the minimum required here is?

[Winterflower]

Hi @joshdover - I think @elastic/ml-core would help here

[benwtrent]

@joshdover we need delete by query. This is for the new data retention policies that transforms can create. Our docs our out of date if delete is not included.

[benwtrent]

I will create an ES issue to capture this inaccuracy and adding a check early on to verify security (like we do for write/read).

[benwtrent]

#85409 for tracking our missing validation and documentation.

[CohenIdo]

Thank you all, I added the delete privilege.

[joshdover]

Do we still need these?

[ywangd]

Ping @elastic/kibana-security for review

[kfirpeled]

namings lgtm

[jportner]

Kibana Security Review --

Thanks for tagging us, LGTM for posterity

[joshdover]

LGTM from a Fleet perspective, still need approval from @elastic/es-security

[sophiec20]

For full disclosure - We are just starting to work on 2 more Security packages (host risk score and beaconing) which contain multiple transforms, and we anticipate more in the near future.

Are we sure that extending the privileges of kibana_system is how we want to achieve this? In this case, we are using kibana_system as a type of service account. This is just about acceptable when transforming internal data (as in the security posture use case) but if the use case requires source data to be a data stream and as the number of transforms grows, then we should consider a longer term solution. Due to the split nature of the elasticsearch and kibana security model there are no easy options here, but I wanted to raise awareness that this is not a one-off request on kibana_system.

[albertzaharovits]

LGTM
My personal preference, not a requirement, is that system generated data go into index names that start with ..

[joshdover]

Are we sure that extending the privileges of kibana_system is how we want to achieve this? In this case, we are using kibana_system as a type of service account. This is just about acceptable when transforming internal data (as in the security posture use case) but if the use case requires source data to be a data stream and as the number of transforms grows, then we should consider a longer term solution. Due to the split nature of the elasticsearch and kibana security model there are no easy options here, but I wanted to raise awareness that this is not a one-off request on kibana_system.

@sophiec20 Agreed, this is not a long-term solution we can keep sustaining. Please see elastic/package-spec#293 for discussion of other options. AFAIK no option has been proposed that wouldn't involve having Elasticsearch implement a package install API. If transforms shipped in packages is going to be a growing feature we need to start having more discussions with Elasticsearch about implementing this feature.

cc @ruflin