Fixing flaky test for knn codecs

[martin-gaievski]

Signed-off-by: Martin Gaievski [email protected]

Description

Trying new PR as a duplicate of #610, seems in that one github is pulling incorrect commit and rolling upgrade bwc are failing. I tested bwc in dev environment they work fine. Creating this as a draft for now to see if checks are ok

Fixed flaky unit tests, narrow assertion logic and adjust for test runs with multi node mock cluster

Repro for certain seeds:

./gradlew ':test' --tests "org.opensearch.knn.index.codec.KNN940Codec.KNN940CodecTests.testKnnVectorIndex" -Dtests.seed=B5E0DC7BF07CBCC1 -Dtests.security.manager=false -Dtests.locale=el-CY -Dtests.timezone=America/Mendoza -Druntime.java=17

org.opensearch.knn.index.codec.KNN940Codec.KNN940CodecTests > testKnnVectorIndex FAILED
    org.mockito.exceptions.verification.TooManyActualInvocations:
    kNN940PerFieldKnnVectorsFormat.getKnnVectorsFormatForField(
        <any string>
    );

After change test runs are succesful:

./gradlew ':test' -Dtests.seed=B5E0DC7BF07CBCC1

BUILD SUCCESSFUL in 1m 35s
15 actionable tasks: 6 executed, 9 up-to-date

Issues Resolved

[List any issues this PR will resolve]

Check List

• All tests pass
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.