[FEATURE] Security support for extensions

[saratvemulapalli]

Coming from:
[Meta] Making OpenSearch Extensible opensearch-project/OpenSearch#2447
[Meta] Plugin Sandboxing (a.k.a extensions) opensearch-project/OpenSearch#1422

Problem

As we are building extensions, security is an integral part of everything we deliver. We see a bunch of problems for plugin security and would like to solve them from Day 1 for extensions.

• Plugins implement security constructs by themselves, evaluating permissions and gating resources manually.
  Anomaly Detection: https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/util/ParseUtils.java#L465
  Alerting: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/transport/SecureTransportAction.kt#L78
• All plugins have super admin permissions and can play with any information they'd like on the cluster.
  Anomaly Detection: https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/transport/IndexAnomalyDetectorTransportAction.java#L95
  Alerting: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt#L68
• Intercommunication with Plugins multiplied with permissions, is manually managed and caused bunch of security loop holes.
  Alerting-Anomaly Detection integration: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/InputService.kt#L149
• Plugins define the permissions they need with Java Security Manager but there is no gating mechanism to deny/validate permissions.
  Anomaly Detection: https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/plugin-metadata/plugin-security.policy
  Alerting: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/plugin-metadata/plugin-security.policy
• Security for plugins is driven by thread context assuming they run in the same JVM. Extensions run within, outside of JVM and existing security interfaces do not work.
• Extensions are built on SDK which is not coupled with JSM (as its deprecated) and we don't have a way to restrict extensions with host level resources (like filesystem, network etc)
• Security plugin has support to authenticate/authorize on transport APIs but do not act on any Rest APIs (which is infact the reason why all plugins implemented security at transport layer). Rest APIs are integral part of OpenSearch eco-system and cannot open all APIs to all extensions.
• Extensions access control for accessing resources within OpenSearch is not gated, which brings risk for OpenSearch cluster running 3rd party extensions.
• Plugins are validated during install time as they are only bootstrapped once during a lifetime of a node. They are validated by bin/opensearch-plugin installation tool. Extensions can be installed/modified/removed on the fly and needs identity validation before it joins the cluster.

Requirements

• [Extensions] Extensions should be able to call a common API which evaluates permissions and gates resources.
• [Extensions] Extensions by default should not have any permissions, and are explicitly granted based on defined policies.
• [Security into Core and Extensions] All API calls between extensions and OpenSearch are access controlled based on extension permission policies.
• [Extensions and SDK] Extensions SDK should be able to restrict host level access(file system, network etc) based on security policy.
• [Security into Core] Add support to gate Rest APIs for OpenSearch
• [Extensions and SDK] Add support to validate the identity of an extension during runtime. This helps in identifying the extension and claiming if it really is. This is an important requirement for extensions since extensions can be installed/modified/removed on the fly and needs identity validation before it joins the cluster.
• [Extensions and SDK] Compliance: Add support for extensions to do audit logging.

Refactored these requirements from June 2022 to explain the problems above.

To build a MVP for extensions we are working on building Anomaly Detection plugin and help run it as an extension.
Our end goal is to have:

• Load third party extensions securely
• Protect the cluster resources when extensions are installed
• Access control for resources created by extensions (Including APIs, application resources etc)
• Access control OpenSearch resources from extensions.

We can chat over this and get more feedback, but these are the resources we could think of to enable secure extensions.

For a start as we are looking at AD: opensearch-project/OpenSearch#3635
We need help to figure out how AD extension can work with security (adding extension support for common-utils):

• AD uses User information for resource access control
• AD uses InjectUser to inject user permissions for background jobs.

[peternied]

This security plugin is not in a good position to support a feature that is built into OpenSearch's codebase, looking at how we are considering building extensions security related features should be built into the core of OpenSearch instead of through its plugin systems. I am going to close this issue.

However, having a hand in developing security for extensions is a discussion we would like to influence - I will create an issue in core and we can have a basis of discussion there.

For the specific topics mentioned:

AD uses User information for resource access control

There is an existing user object in the common-utils codebase, this object can't be injected/extracted as is currently done since it requires a shared thread context model. There is work in opensearch-project/opensearch-sdk-java#14 that I believe is the foundation for resolving this problem. Is there a place where API contracts of extensions are under discussion or should I start proposing the api request/responses?

AD uses InjectUser to inject user permissions for background jobs.

Persisting identity is a very difficult problem to solve. I think we need a new series of APIs that allow for out of band/non-interactive request to create actions on behalf of a user. This will require first coming to terms with the identity management system in OpenSearch, but also the access mechanisms (eg, do we follow a pattern from AWSs assume role APIs, or is there another convention we should be adopting

[peternied]

All in all - I think we have an opportunity to design a better security posture for OpenSearch and its extensibility models. Asks like this highlight the need for more thought and alignment generation in this space, thanks for calling out these concerns.

[peternied]

Note; Leaving this open until those called out areas are documented in the SDK

[jimishs]

Hi Sarat, Thanks for starting this thread. Could you pls elaborate on specific needs across the following themes that you highlighted (comments inline)

• Load third party extensions securely
  [Jimish] What does it mean to load extensions securely? Are you suggesting that extensions run inside a sandbox that is provided by a general security framework? Something like a separate container perhaps?

• Protect the cluster resources when extensions are installed
  [Jimish] Can you elaborate. I believe in earlier discussions you mentioned host level security. Are we thinking of resource limiting controls for extensions (eg: max CPU, memory etc)? What are other controls you are looking for?

• Access control for resources created by extensions (Including APIs, application resources etc)
  [Jimish] Can you expand on what resources will be created by extensions? Is the reference to APIs about security plugin APIs to access data or something else? What application resources would you like to have access control configuration for?

• Access control OpenSearch resources from extensions
  [Jimish] What resources are you referring to? Also what granularity of control do you need for such resources?

In addition im also curious to get your thoughts on whether extensions will be able to create their own metadata/storage files, or is that something that could be part of a security SDK?

Thanks

[saratvemulapalli]

Thanks @jimishs for these questions. This was written 6months ago, I have updated the issue with more granular details of the problems.

[Jimish] What does it mean to load extensions securely? Are you suggesting that extensions run inside a sandbox that is provided by a general security framework? Something like a separate container perhaps?

"Plugins are validated during install time as they are only bootstrapped once during a lifetime of a node. They are validated by bin/opensearch-plugin installation tool. But, Extensions can be installed/modified/removed on the fly and needs identity validation before it joins the cluster."
We need to add a mechanism to support verification of identity of extensions which they really claim to be.

[Jimish] Can you elaborate. I believe in earlier discussions you mentioned host level security. Are we thinking of resource limiting controls for extensions (eg: max CPU, memory etc)? What are other controls you are looking for?

"Extensions SDK should be able to restrict host level access(file system, network, execution etc) based on security policy."
That is right, plugins do have this via JSM using security policy, and we need similar support for extensions.

[Jimish] Can you expand on what resources will be created by extensions? Is the reference to APIs about security plugin APIs to access data or something else? What application resources would you like to have access control configuration for?

Sure, Anomaly Detection extension will create detectors, Alerting will create monitors. These resources have to be gated by user identity.

[Jimish] What resources are you referring to? Also what granularity of control do you need for such resources?

"[Extensions] Extensions by default should not have any permissions, and are explicitly granted based on defined policies.
[Security into Core and Extensions] All API calls between extensions and OpenSearch are access controlled based on extension permission policies."
All APIs(Rest and transport) extensions make (via SDK) to OpenSearch should be evaluated based on extension permissions.
Not all APIs will be accessible by all extensions.

I hope that helps. Let me know what other questions do you have.

On a side note: @peternied @dbwiddis @dblock @nknize I've updated the issue with sets of problems/requirements, would love your feedback and thoughts.

[peternied]

Thanks for the updates @saratvemulapalli some areas I'd like more details on:

• [Security into Core and Extensions] All API calls between extensions and OpenSearch are access controlled based on extension permission policies.
• [Security into Core] Add support to gate Rest APIs for OpenSearch

Can you give me an example for how the first requirement is different from the second?

---

• [Extensions and SDK] Extensions SDK should be able to restrict host level access(file system, network etc) based on security policy.

Is this needed for remote extensions? When you say security policy, does this mean the intention is to keep reusing the JSM based policy files for extensions sharing a host?

FYI @cwperks

[dbwiddis]

Coming at this from a slightly different perspective, so this comment is not particularly a reply to anything above but more my own understanding of how things are planned to work.

Current State:

• Security implemented via org.opensearch.security.user.User injected/shared around OpenSearch and plugins. We want to get rid of this.
• User has the following features/capabilities which are leveraged by plugins:
  • Name (String)
  • Roles (Set of Strings)
  • Security Roles (Set of Strings)
  • Attributes (Map of String to String)
  • Requested Tenant (String)

Future State with extensions:

• Extension will not have access to the above User object.
• Extension has no need to understand any of the User details so there is no need to send all this information to the Extension.
• What Extension does need is two key components:
  • Identity. A unique representation of a user (Principal).
    • OpenSearch will maintain a more rich identity.
    • OpenSearch will only pass the extension a unique key for this identity.
    • It is not necessary, but helpful, for the extension to have some reasonable human-readable name associated with the principal's key.
    • I think a JWT Identity Token suits the above requirements quite well.
  • Access. A set of permissions that the user has.
  • The Extension doesn't need to know the details of all the access.
  • The extension can just make requests and if they are not allowed (or return partial/redacted results) it can receive the appropriate status/error messages.
  • It is not necessary, but helpful, to have lightweight "access check" functionality (via REST) before sending large amounts of data in a request (fail fast).
  • I think a JWT Access Token suits the above requirements quite well.
• The Extension will be interacting with OpenSearch via REST requests
  • Ideally will use the opensearch-java client. Temporarily using RestHighLevelClient. These both implement security/user/authentication via Apache Http 5 authorization schemes. If we stick with standards (e.g., JWT) this should be simple to implement on the extension side.
    • When an extension gets forwarded a REST request from OpenSearch it will contain the necessary identity and access tokens to integrate via this client.
    • OpenSearch needs to parse these authorization schemes on receipt of REST requests and control access appropriately

Overall summary:

• On Extension side:
  • Need to receive user principal identity and access token and register in the Rest Client being used for requests back to OpenSearch
• On OpenSearch side:
  • Need to generate these principal identity and access tokens with each request (scoped for the time of the request) and forward to an extension
  • Need to receive these tokens back with REST requests and apply associated security "single point of access" checks here
  • Want to add a few basic requests to "preview" permissions to handle some use cases where this is useful

[saratvemulapalli]

Thanks for the updates @saratvemulapalli some areas I'd like more details on:
Can you give me an example for how the first requirement is different from the second?

@peternied Thats a great question, my head wanted to say Transport APIs and Rest APIs. I'll combine both of them together to make it concise.

Is this needed for remote extensions? When you say security policy, does this mean the intention is to keep reusing the JSM based policy files for extensions sharing a host?

Yes, its for all extensions running with the SDK. Not really leaning onto one solution, I've used security policy as an example which everybody gets it, but we could piggy back on JSM but the risks are obvious as its deprecated.

[peternied]

I'm not seeing how restrict host level access falls into the Extensions SDK, could you expand on this and frame it this with a user problem?