Fix for Flaky test for issue 384

[vibrantvarun]

Description

Currently, we are facing falkiness in the integ test behaviour of neural search because of following reasons.

1. We try to findAllDeployedModels at the end of the single test and try to delete it. However, if at the same time any other test from some other test class starts deploying model than we see below error due to multithreading environment

�  java.lang.Exception: Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete

2. Moreover, we use the same index name in few classes such that it causes problems in multithreading environment.
3. Also, at some places we are not deleting the index after the test execution.

This PR address all the above issues.

Issues Resolved

384

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f9937a4) 84.39% compared to head (71d4e8d) 84.39%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main     #559   +/-   ##
=========================================
  Coverage     84.39%   84.39%           
  Complexity      535      535           
=========================================
  Files            40       40           
  Lines          1570     1570           
  Branches        245      245           
=========================================
  Hits           1325     1325           
  Misses          133      133           
  Partials        112      112           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[martin-gaievski]

One BWC check is failing, please check if that's not for the same reason as we're trying to fix here

[martin-gaievski]

I think we're missing one piece in this - we're not using that model id that we read in setUp section. I see that in tests like this one we're reading model id: https://github.com/vibrantvarun/neural-search/blob/Flaky_test_384/src/test/java/org/opensearch/neuralsearch/processor/NormalizationProcessorIT.java#L94. We need to use that id we get in setUp method instead

[vibrantvarun]

No it is not. It is due to sometimes when we push multiple commits back to back then due to resources sometimes tests fails.

[navneet1v]

@vibrantvarun what is pending on this PR?

[vibrantvarun]

@navneet1v @heemin32 advised to add finally block in adding wipeoftestResources. Therefore I added it and now raised a revision.

[martin-gaievski]

Nice rework for tests Varun

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-559-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d298e2ba7f436f1cea2392de34d54cfc093fe435
# Push it to GitHub
git push --set-upstream origin backport/backport-559-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-559-to-2.x.