Converts .opendistro-job-scheduler-lock index into a system index

[joshpalis]

Description

Actioning two issues to convert the .opendistro-job-scheduler-lock index into a system index.
This PR makes the following modifications :

• makes JobSchedulerPlugin implement the SystemIndexPlugin and overrides the getSystemIndexDescriptors extension point
• wraps each instance in which the plugin interacts with the system index within the LockService to stash the threadcontext prior to executing a CRUD operation. From a security perspective this allows the plugin to go into a local trusted mode in which they are allowed to interact with the system index

Next steps :

• Modify the installation scripts (sh, bat) for unix/windows systems to add the system index name to the list of system indices that the security plugin maintains
• Companion PR : Adds .opendistro-job-scheduler-lock to list of protected indices security#3237

Issues Resolved

#466 - System Index Usage Verification
#305

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #474 (fac7d7d) into main (2db686b) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main     #474      +/-   ##
============================================
- Coverage     29.19%   29.17%   -0.03%     
  Complexity       98       98              
============================================
  Files            22       22              
  Lines          1185     1186       +1     
  Branches        109      109              
============================================
  Hits            346      346              
- Misses          818      819       +1     
  Partials         21       21              

Files Changed	Coverage Δ
...rg/opensearch/jobscheduler/JobSchedulerPlugin.java	31.14% <0.00%> (-0.52%)	⬇️

[joshpalis]

Interestingly the SampleJobRunnerIT tests are failing consistently, but pass locally. I will continue to look into this

[vibrantvarun]

Tests are failing

[cwperks]

It may be the case that JobScheduler does not need to wrap calls to interact with the lock system index inside of a threadContext.stashContext { ... } block, but I don't think it would hurt either.

Typically stashContext is used within plugins when handling authenticated requests. (i.e. API Endpoint Handling) which JS does not do to my knowledge.

A good example is Anomaly Detection create detector.

1. User calls on endpoint to create detector and passes credentials with the request
2. The security plugin wraps the REST handler, verifies the credentials and authenticates the request. Authentication here means that the security plugin injects the User into the threadcontext
3. When AD wants to write a document to the detector system index it will stash the threadcontext (effectively switching from the context of the authenticated user into a new trusted context where it can do anything) and index the document into its system index containing the detector definitions.

If JS is always outside of an authenticated user context than stashing the threadcontext could be considered redundant, however, there are other avenues for plugins to inject roles/user into the threadcontext so stashing the context ensures that the block executing code has a clean slate.

[joshpalis]

It may be the case that JobScheduler does not need to wrap calls to interact with the lock system index inside of a threadContext.stashContext { ... } block, but I don't think it would hurt either.

I've observed that the integration tests would fail consistently after this change, at first I considered it might be a transitive error but after a few test runs it hasn't been resolved. I'll opt to revert the changes to stash the thread context