Cross replicate internal users and roles database

[petro2050]

Is your feature request related to a problem? Please describe.
I'm doing a multi-region AWS setup where each region has an OpenSearch cluster. I enabled cross-region replication so only one region acts as the leader. The problem is I don't see a way to replicate internal users, which forces me to define them manually through the security plugin. I'd also like to periodically rotate user passwords (for example with a lambda-based secret rotation). This means the lambda has to issue a PUT request to each and every cluster. This work seems unnecessarily complex and error-prone.

Describe the solution you'd like
We should be able to cross-replicate the internal users and roles database across clusters.

[anasalkouz]

@opensearch-project/admin could you move this to security plugin repo?

[peternied]

Love to have the thoughts of the @opensearch-project/replication folks, what do y'all think of this scenario and feature?

[ankitkala]

Cross cluster replication in the current state can replicate the indices as well as their metadata. But we explicitly don't allow system/hidden indices(security index in this case) because supporting these would required deep integration with the plugins.

What I mean is that security plugin on leader cluster would operate as usual with both reads and writes happening. But the index on follower would have to be read only, security plugin can't write anything but would be expected to react to changes happening in the index (for e.g. permission changes propagated by leader).
Essentially security plugin(and any other plugin with system indices) would have to support these 2 modes of operation: leader mode(read & write) and follower mode(read only). I'm assuming it'll require major changes in the security plugin architecture. This is something we'd want to achieve someday but is not in our short term roadmap.

Another thing that makes it more tricky is the fact that CCR is index level replication so the propagating security permissions is not straightforward as you'd want to replicate permissions specific to the indices that are getting replicated. Cluster level permissions might or might not need to be replicated depending on case to case.
This problem will go away with Full Cluster Replication where we want to replicate all the data(and not select indices). That would be the right time to revisit how to propagate the security configuration. But again, its not planned for short/mid term.

[wbeckler]

Would decoupling the data source cluster from the dashboard cluster be another solution to this problem? In that scenario, user permissions live in a single region, but other regions can talk to that region. This would be a follow on to the multi data source functionality: opensearch-project/OpenSearch-Dashboards#1388

…

On Wed, Jun 14, 2023, 1:24 AM Ankit Kala ***@***.***> wrote: Cross cluster replication in the current state can replicate the indices as well as their metadata. But we explicitly don't allow system/hidden indices(security index in this case) because supporting these would required deep integration with the plugins. What I mean is that security plugin on leader cluster would operate as usual with both reads and writes happening. But the index on follower would have to be read only, security plugin can't write anything but would be expected to react to changes happening in the index (for e.g. permission changes propagated by leader). Essentially security plugin(and any other plugin with system indices) would have to support these 2 modes of operation: leader mode(read & write) and follower mode(read only). I'm assuming it'll require major changes in the security plugin architecture. This is something we'd want to achieve someday but is not in our short term roadmap. Another thing that makes it more tricky is the fact that CCR is index level replication so the propagating security permissions is not straightforward as you'd want to replicate permissions specific to the indices that are getting replicated. Cluster level permissions might or might not need to be replicated depending on case to case. This problem will go away with Full Cluster Replication where we want to replicate all the data(and not select indices). That would be the right time to revisit how to propagate the security configuration. But again, its not planned for short/mid term. — Reply to this email directly, view it on GitHub <#1076>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5PRLSUSPTQQQRBXPWKM3LXLFDH7ANCNFSM6AAAAAAZFNLTNQ> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[stephen-crawford]

Hi @ankitkala, is there anything that prevents the use of an external IdP which would span the regions? It seems like replicating the entire index would be resource intensive while using the Security Admin Tool could lead to concurrency issues where deleting users in one region may not happen in the replicated region in an appropriate matter. Are there any alternatives you have considered?

[ankitkala]

Would decoupling the data source cluster from the dashboard cluster be another solution to this problem? In that scenario, user permissions live in a single region, but other regions can talk to that region. This would be a follow on to the multi data source functionality: opensearch-project/OpenSearch-Dashboards#1388
…
On Wed, Jun 14, 2023, 1:24 AM Ankit Kala @.> wrote: Cross cluster replication in the current state can replicate the indices as well as their metadata. But we explicitly don't allow system/hidden indices(security index in this case) because supporting these would required deep integration with the plugins. What I mean is that security plugin on leader cluster would operate as usual with both reads and writes happening. But the index on follower would have to be read only, security plugin can't write anything but would be expected to react to changes happening in the index (for e.g. permission changes propagated by leader). Essentially security plugin(and any other plugin with system indices) would have to support these 2 modes of operation: leader mode(read & write) and follower mode(read only). I'm assuming it'll require major changes in the security plugin architecture. This is something we'd want to achieve someday but is not in our short term roadmap. Another thing that makes it more tricky is the fact that CCR is index level replication so the propagating security permissions is not straightforward as you'd want to replicate permissions specific to the indices that are getting replicated. Cluster level permissions might or might not need to be replicated depending on case to case. This problem will go away with Full Cluster Replication where we want to replicate all the data(and not select indices). That would be the right time to revisit how to propagate the security configuration. But again, its not planned for short/mid term. — Reply to this email directly, view it on GitHub <#2856 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA5PRLSUSPTQQQRBXPWKM3LXLFDH7ANCNFSM6AAAAAAZFNLTNQ . You are receiving this because you are on a team that was mentioned.Message ID: @.>

I'm assuming that, with the proposed change, each cluster will have to maintain its own set of permission rules. Else, we'll need to figure out a way to replicate the security configurations across the clusters.

[ankitkala]

Hi @ankitkala, is there anything that prevents the use of an external IdP which would span the regions? It seems like replicating the entire index would be resource intensive while using the Security Admin Tool could lead to concurrency issues where deleting users in one region may not happen in the replicated region in an appropriate matter. Are there any alternatives you have considered?

We're not actively exploring the security index replication. This is something we were thinking for Full Cluster Replication but its not being actively worked upon right now.

Using an external IDP can solve part of the problem but then you're also forcing the existing customers to move away from their existing configuration which might not be acceptable to everyone.

[peternied]

[Security Triage] This feels out of scope for the security plugin, while there might be features / asks to support cross cluster scenarios, the security plugin is focused on administration of a single OpenSearch instance, transferring this to cross cluster repo where there might be better traction.

[zalseryani]

Any update on this ?

[ankitkala]

This is not planned as of now as it requires significant changes.