[BUG] BWC tests are failing on neural search when load model api is called

[vibrantvarun]

What is the bug?
The bwc tests on neural search are failing due to below error

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't update non dynamic settings [[index.number_of_shards]] for open indices [[.plugins-ml-task/B7ApSs9vTtKUurU54IuV6A]]"}],"type":"illegal_argument_exception","reason":"Can't update non dynamic settings [[index.number_of_shards]] for open indices [[.plugins-ml-task/B7ApSs9vTtKUurU54IuV6A]]"},"status":400}
        at __randomizedtesting.SeedInfo.seed([A532B5BBB1EF0C09:B5AEE3F5FC9D5C50]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.makeRequest(BaseNeuralSearchIT.java:735)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.makeRequest(BaseNeuralSearchIT.java:708)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.loadModel(BaseNeuralSearchIT.java:176)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.testTextEmbeddingProcessor_E2EFlow(SemanticSearchIT.java:42)

How can one reproduce the bug?
Steps to reproduce the behavior:

1. checkout neural search repo and branch 2.x
2. ./gradlew bwcTestSuite and then consecutively run for other versions by passing parameter for -Dbwc.version (for 2.9, 2.10, 2.11)
3. Test will fail with above error
4. See error

What is the expected behavior?
Bwc tests should pass successfully

What is your host/environment?

• OS: linux, windows

Do you have any additional context?
This fix might be missing in the bwc versions #2137

[vibrantvarun]

I ran below command on branch 2.x on my local machine

./gradlew :qa:restart-upgrade:testAgainstNewCluster -Dbwc.version=2.9.0 

and facing above error

[b4sjoo]

This is because in this #1315 we have bumped the .plugins-ml-task version issue in 2.9, and in #2137 we have introduced a static setting in the index setting, so it will break bwc. I’ve already submitted a fix PR #2156

[b4sjoo]

Can confirm this issue is resolved.