Security Plugin cannot startup due to AccessControlException: access denied

[peternied]

The security plugin cannot be loaded because of an issue with the bouncy castle dependancy coming from OpenSearch core.

 java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
�  Likely root cause: java.lang.InternalError: cannot create instance of org.bouncycastle.jcajce.provider.digest.GOST3411$Mappings : java.security.AccessControlException: access denied ("java.security.SecurityPermission" "putProviderProperty.BC")
�  	at org.bouncycastle.jce.provider.BouncyCastleProvider.loadServiceClass(Unknown Source)
�  	at org.bouncycastle.jce.provider.BouncyCastleProvider.loadAlgorithms(Unknown Source)
�  	at org.bouncycastle.jce.provider.BouncyCastleProvider.setup(Unknown Source)
�  	at org.bouncycastle.jce.provider.BouncyCastleProvider.access$000(Unknown Source)
�  	at org.bouncycastle.jce.provider.BouncyCastleProvider$1.run(Unknown Source)
�  	at java.base/java.security.AccessController.doPrivileged(Native Method)
�  	at org.bouncycastle.jce.provider.BouncyCastleProvider.<init>(Unknown Source)
�  	at org.opensearch.security.OpenSearchSecurityPlugin$2.run(OpenSearchSecurityPlugin.java:323)
...

Additional Context

• Plugin install failure https://github.com/opensearch-project/security/actions/runs/6086574945/job/16513238110?pr=3307
• BWC test startup failure https://github.com/opensearch-project/security/actions/runs/6086574929/job/16513238774?pr=3307
• SecurityManager permissions issue with other plugins [BUG] Somehow security plugin rewrites permissions for other plugins which use Bouncy Castle #3213

[peternied]

Root cause change in main [1] & backport to 2.x [2]

• [1] Add encryption support for repository  OpenSearch#9289
• [2] [Backport 2.x] Add encryption support for repository (#9289) OpenSearch#9736

[peternied]

@cwperks Can you help drive this from the security plugin?

FYI - @reta @willyborankin

[reta]

@cwperks let me know if you need any help

[cwperks]

@reta all eyes are appreciated on this issue. This is release blocking for 2.10.0.

[reta]

Sure, I am taking it, @cwperks!

[cwperks]

@willyborankin opened a related issue a few days ago: #3213

It looks like the plugin-security.policy file in this repo is being ignored.

[peternied]

@cwperks @reta I think we need to remove the opensearch-encryption-sdk from OpenSearch to get unblocked - I am open to other suggestions, but we need to be wary of the impact to the security posture of OpenSearch.

[reta]

@peternied on the same page, I think I found the solution to unblock us, hold on please

[willyborankin]

@peternied im for reverting opensearch-encryption-sdk PR. Im for crypto-sdk but it need t be implemented differently.

[cwperks]

I'm on the same page as well. I'm trying a few different things in the security plugin and in core, but reverting would get the release unblocked while a strategic fix is worked on