[Discuss] Installing package assets that require elevated privileges

[joshdover]

Background

In elastic/kibana#111755 we started using the internal kibana_system user to install integration package assets into Elasticsearch in order to support automatic upgrades of packages without end user intervention. We then leveraged this capability to remove the requirement for end users to have the superuser role to install packages in elastic/kibana#108252 in 8.1.

One downside of this approach is that it requires that kibana_system have the required privileges to install any package assets. This poses a problem for some asset types that require more expansive privileges in Elasticsearch to create:

• transforms requires these privileges:
  • transform_admin built-in role or manage_transform cluster privileges,
  • read and view_index_metadata index privileges on source indices, and
  • create_index, index, manage, and read index privileges on destination indices
  • See [Fleet] Support installing transforms without granting kibana_system index privileges kibana#137278 for more in-depth discussion on transforms
• ILM policies requires these privileges:
  • manage_ilm cluster privilege to use this API. You must also have the manage index privilege on all indices being managed by policy. ILM performs operations as the user who last updated the policy. ILM only has the roles assigned to the user at the time of the last policy update.
  • supported in the spec, but currently only used by APM
  • Related ES issue: Decouple ILM action execution from the user setting the policy elasticsearch#72856
• enrich policies requires these privileges:
  • read index privileges for any indices used
  • The enrich_user built-in role
  • not yet supported, but proposed in [Change Proposal] Adding support for Enrichment Policy in Packages #292

To workaround this, we've been making direct edits to the kibana_system role definition in Elasticsearch when we need to update a transform or ILM policy. Thus far, this has been acceptable because these asset types are only being used by internal packages that ship with the Stack. However, long-term this is not sustainable and prevents external integration developers from using these features.

Discussion

• For other packages that would like to use these features, how can we best surface these privilege requirements to the end user? Should Fleet UI be able to automatically detect the privilege requirements to install a given package or should these be defined in the package manifest?
• If we require elevated privileges to install packages containing these assets, will there be any issues with using a mix of users (kibana_system and end user) to install different asset types?
• How could we allow packages to be installed without requiring direct access to these features? Could Elasticsearch support an install_package privilege that would grant limited access to these features? For example, could this privilege allow a user to install and remove these asset types from verified packages without granting edit access or the ability to install arbitrary assets?

[ruflin]

Long term I hope Elasticsearch should install these assets directly so Fleet does not need the permissions. Short and mid term we must ensure to keep the kibana_system permissions to a minimum.

A package itself should not be required to know about the permission model in the Stack. Fleet instead has detailed knoweledge what permissions what assets needs. If Fleet can't install the assets with the permissions of the user, could we open a message explaining what permissions are missing and refuse to install the package? Then the user can go to his admin to ask for the additional permissions or request the admin to install the package. The other option is to install only partial packages but I think this gets us into more trouble than solutions.

[joshdover]

If Fleet can't install the assets with the permissions of the user, could we open a message explaining what permissions are missing and refuse to install the package?

This would work fine for packages that the system doesn't need to upgrade itself, and is certainly a path forward. However, the packages that are currently using these features are the same ones that do need to be auto-upgraded on Stack upgrades: APM and Endpoint

[joshdover]

Another instance of this problem, APM has decided to grant kibana_system access to delete all APM indices to allow their ILM policies to work in 8.2+: elastic/elasticsearch#85085

[joshdover]

Another ILM case: elastic/elasticsearch#85844

[joshdover]

I'm trying to determine how we can make progress on this, especially the transform use case which is a growing need for some of our 1st party Security integrations. These integrations have an associated Kibana UI so they need to be updated in tandem, requiring kibana_system to have read and write access to the transform's src and dest indices.

The only short-term solution that I've been able to dream up is adding a new type of transform privilege to kibana_system which would allow us to create these transforms without also granting explicit read and write access to indices. If this could be done in a way that isn't exposed to users on the Roles API and is restricted to the data stream naming scheme, it might be a safer and more convenient short-term option than continuously adding more read/write privileges to kibana_system.

cc @elastic/ml-core @elastic/es-security

[SourinPaul]

Thanks for continuing to think through potential solutions @joshdover.

As you eluded, such privilege escalation blocks us to install certain transform-based use case packages (e.g. Identifying Beaconing Detection) that need to read events from logs-* src indices, which the kibana_system may not have.

cc: @szeitlin 👀

[joshdover]

As you eluded, such privilege escalation blocks us to install certain transform-based use case packages (e.g. Identifying Beaconing Detection) that need to read events from logs-* src indices, which the kibana_system may not have.

@SourinPaul What we could do in the short term is require a user with enough privileges to initiate the package install and upgrades with a button click or API call. You may need your UI to be able to handle the transform package being outdated (not necessarily upgraded with Kibana if the user hasn't clicked the upgrade for your package yet). If this sounds attractive to your team, we should open a Kibana issue to track making this possible in the Fleet install code.

My understanding is that @sophiec20 has some folks looking into this problem for Transforms for a long-term solution.

[szeitlin]

Confirmed with @ajosh0504 that this issue is a blocker for us, in case that helps with prioritizing.

[SourinPaul]

What we could do in the short term is require a user with enough privileges to initiate the package install and upgrades with a button click or API call. You may need your UI to be able to handle the transform package being outdated (not necessarily upgraded with Kibana if the user hasn't clicked the upgrade for your package yet). If this sounds attractive to your team, we should open a Kibana issue to track making this possible in the Fleet install code.

@joshdover Certain Entity Analytics packages (e.g., Identifying Beaconing Detection) will NOT be installed OOTB in the solution. Users will explicitly install/ enable these in their environment, similar to our 3rd party security integration workflow.

Hence, your short-term proposal to require user action to install (and upgrade) the package is very reasonable.

Such interactions, e.g. Install Package or Upgrade Package may happen within the Fleet workflow. or be routed (via API) from the security solution workflows. We do something similar today within the Rules editor for updating Rule content. I'm hopeful we (cc: @kseniia-kolpakova @paulewing ) can identify where/ how we flag ⚠️ the underlying Analytics that needs to be updated. To reduce complexity, we can avoid out-of-release updates for now.

Our end goal will be to upgrade these packages out-of-band and automatically (without any user clicks).

Please lmk if I misunderstood any aspect of your proposal or if anything needs further elaboration.

[droberts195]

especially the transform use case

The only short-term solution that I've been able to dream up is adding a new type of transform privilege to kibana_system which would allow us to create these transforms without also granting explicit read and write access to indices.

In elastic/elasticsearch#86757 we added the ability to specify secondary auth headers when creating a transform. What this means is that the primary auth headers are used to check if the transform can be created, but the credentials stored with the transform that it will use to access the source and destination indices when it's running are the completely different set of secondary credentials. This functionality could be used as follows:

• Primary auth headers are those of kibana_system, which would need to be granted the manage_transform privilege but not read/write permissions on the source and dest indices of the transform
• Secondary auth headers are those of some other user who does not need the manage_transform privilege but who has permission to read the source indices and write to the dest index of the transform

There's no getting away from the fact that somebody needs to have permission to access the indices the transform is hitting, otherwise it could be used as a back door to get access to protected data.

One more possibility would be to use an API key for the secondary credentials for creation of the transform. An API key containing permissions for the transform source and destination indices could be created the first time a package is installed, then provided as the secondary credentials for that transform when kibana_system creates it. Then that same API key could also be provided each time the transform is upgraded. This of course creates a different problem, which is how to securely store the API key between the initial install and the automated upgrades. So it might not be a helpful suggestion at all but I thought I'd mention it.

[ajosh0504]

Reading all the above comments, the following suggestion from @joshdover sounds the most reasonable for now, especially if we already have ways to handle outdated packages in the Security Solution today.

What we could do in the short term is require a user with enough privileges to initiate the package install and upgrades with a button click or API call. You may need your UI to be able to handle the transform package being outdated (not necessarily upgraded with Kibana if the user hasn't clicked the upgrade for your package yet). If this sounds attractive to your team, we should open a Kibana issue to track making this possible in the Fleet install code.

[szeitlin]

@droberts195 @joshdover @Sourin You all bring up some good points about the detailed implementation of RBAC for these layers. It would help me if we had some diagrams of how this currently works vs. what you're proposing. Can someone make some quick flowcharts so we can compare the options and plan to discuss?

[joshdover]

@droberts195 elastic/elasticsearch#86757 is great news, I hadn't been made aware of that change.

I think this should allow us to begin leveraging es-secondary-authorization to allow the general install and upgrade of transforms via user interaction (UI or API). This should solve the majority of use cases at this time. These packages would not be able to be installed via kibana.yml "preconfiguration" in this phase.

If we ever choose to support automatic package upgrades, we'd need to start capturing credentials or creating API keys for users behind the scenes and persisting those in an encrypted saved object, similar to how alerting does.

The only remaining case would then be packages that also need to be installed on Kibana startup. Today, no packages are installed with the default configuration, but users can use the kibana.yml configuration to install packages on startup and we also use this in ESS to bootstrap the APM package. We'd need to add support here to specify an API key from the secret store in kibana.yml and then change our ESS orchestration code to create this API key on cluster bootstrap. At this point, we could remove the transform-related index privileges for kibana_system completely. We might need to wait until a major release to do this though, otherwise we could break the auto-upgrades for the Endpoint package in the case we haven't captured an API key to upgrade it yet.

If that sounds right to you all, I will open a Kibana issues for these phases.

@szeitlin let me know if the above clarifies things enough. Otherwise I'm happy to throw together some diagrams.

[szeitlin]

@joshdover It seems to me that we're talking about a couple of different use cases, and it would be helpful to break them apart.

1. installing new packages for the first time, e.g. on initial cluster setup
2. updating existing packages, automatically or otherwise
3. installing new packages later

I'm not clear on whether 2 and 3 need to be, or are currently, different processes.

I'm also not clear on whether we want any packages to be installed by default as part of cluster setup.

So yes, diagrams would be helpful if it clarifies our thinking on what problems we're trying to solve, and what the priority order of those are.

For example, I can easily imagine scenarios where the cluster setup is done by one person with more privileges, but installing or updating packages might be done by someone else (or automated) using a different (more restricted) set of privileges.

Exactly how this gets implemented- which API keys would be needed, and where, should probably derive from how we're defining the problems we're trying to solve. I feel like we might be rushing to solve problems without really breaking them down first, but I'm also joining this project after it's been going for a while, so if there are already documents (and ideally, diagrams or mockups) explaining how we want this all to work, please point me at them.

[joshdover]

EDIT: this thread was getting long. I've moved the transform-specific discussion to elastic/kibana#137278

[droberts195]

While @qn895 has been working through the changes to transform installation in Fleet it occurred to me that it's not just transforms that kibana_system might not have permission to install or remove. Looking at the privileges that have been granted to kibana_system so far they seem asymmetric, so certain entities can be created but not deleted. Does this already cause problems, like not being able to uninstall and reinstall a package to reset a feature to its empty state?

It might be best to generalize the idea that each entity that a package installs gets installed by one of the following:

1. kibana_system
2. The logged-in user
3. kibana_system, but with some subset of the logged-in user's credentials used to create an API key that is passed as secondary credentials

Where the logged-in user's credentials are going to be used it would be good to confirm that they have all the required privileges (using the _has_privileges API) before installing anything.

So then each entity to be installed would have an extra setting in its YAML to say which of the 3 options to use. This could default to option 1, kibana_system, if not specified. That would preserve the behaviour for existing packages.

[joshdover]

Details are still being worked out, but it's likely that the new DLM (data lifecycle management) feature for data retention and downsampling will come with a new index-level privilege which would help us avoid this problem for integrations. Kibana would be granted this dlm privilege for logs-*,metrics-*,traces-* and be able to update data retention settings on package upgrades without the end user needing the privilege.