Support defining .opensearch-knn-models as a secure system index

[jmazanec15]

Description

For 2.6, we onboarded k-NN into the security plugin, however, we had to revert the change due to failed permissions writing to system index in our code.

The problem was that we marked the index as a system index in the default security demo install script (ref).

In the security integration tests ran as a part of the 2.6 release testing, we were getting a failure that looked like:

[2023-02-23T22:29:33,608][ERROR][o.o.k.t.TrainingJobRunner] [dev-dsk-jmazane-2a-180e0128.us-west-2.amazon.com] Unable to initialize model serialization: no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]

The issue is that when we try to write to the system index in ModelDao.

The problem is that the thread context when executing those calls has the user that made the requests credentials in the headers. In the security plugin, only superadmins can use make requests to modify the index (refer to this comment in AD).

One method for solving this is to stash the thread context before making the call. This will remove the headers. Security will interpret this as superadmin making request and allowing it to happen. See example in AnomalyDetection.

Related security plugin code

(thanks for providing @cwperks)

1. https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L161-L163
2. https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L258-L266
3. https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/privileges/SecurityIndexAccessEvaluator.java#L124

Related Issues/Prs

1. Marking .opensearch-knn-models as a system index causes failures in k-NN plugin security#2478
2. Onboard KNN Plugin features into security security#2265
3. Integrate k-NN functionality with security plugin security#2274