[Flaky test] org.opensearch.knn.indices.ModelDaoTests.testDelete

[jmazanec15]

Details

Seeing flaky behavior around the test ModelDaoTests.testDelete.

Instances Ive seen (please comment more below)

1. https://github.com/opensearch-project/k-NN/actions/runs/7643391383/job/20825300981?pr=1393
2. https://github.com/opensearch-project/k-NN/actions/runs/7466812345/job/20319024945

Stack trace if above get delete:

org.opensearch.knn.indices.ModelDaoTests > testDelete FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=269, name=opensearch[node_s_0][generic][T#1], state=RUNNABLE, group=TGRP-ModelDaoTests]
        at __randomizedtesting.SeedInfo.seed([DACDCF41AB8C9CD:40F363E0EA6F6050]:0)

        Caused by:
        java.lang.NullPointerException
            at __randomizedtesting.SeedInfo.seed([DACDCF41AB8C9CD]:0)
            at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.get(ModelDao.java:394)
            at org.opensearch.knn.training.TrainingJobClusterStateListener.updateModelsNewCluster(TrainingJobClusterStateListener.java:112)
            at org.opensearch.knn.training.TrainingJobClusterStateListener.lambda$clusterChanged$0(TrainingJobClusterStateListener.java:90)
            at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:829)
REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.indices.ModelDaoTests.testDelete" -Dtests.seed=DACDCF41AB8C9CD -Dtests.security.manager=false -Dtests.locale=nb -Dtests.timezone=America/Cancun -Druntime.java=11