[BUG] Integ test for 1.3.2 failed on k-NN on both arm64 & x64

[zelinh]

What is the bug?
Integ test for 1.3.2 failed on k-NN on both arm64 & x64
Here is a brief context:

2022-05-03 18:25:43 INFO     Running integration tests for k-NN
2022-05-03 18:25:43 INFO     ===============================================
2022-05-03 18:25:43 INFO     Executing "/var/jenkins/workspace/integ-test/scripts/default/integtest.sh -b localhost -p 9200 -s true -v 1.3.2" in /tmp/tmp0dsv77sj/k-NN
2022-05-03 18:26:40 INFO     Recording component test results for k-NN at /var/jenkins/workspace/integ-test/test-results/1/integ-test/k-NN/with-security/test-results
2022-05-03 18:26:40 INFO     Integration test run failed for component k-NN
2022-05-03 18:26:40 INFO     Note: /tmp/tmp0dsv77sj/k-NN/src/main/java/org/opensearch/knn/index/KNNWeight.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.knn.index.FaissIT.testEndToEnd_fromModel" -Dtests.seed=2187012C1883C8A0 -Dtests.security.manager=false -Dtests.locale=sr-BA -Dtests.timezone=W-SU -Druntime.java=11
REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.knn.index.FaissIT.testDocDeletion" -Dtests.seed=2187012C1883C8A0 -Dtests.security.manager=false -Dtests.locale=sr-BA -Dtests.timezone=W-SU -Druntime.java=11
REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.knn.index.FaissIT.testEndToEnd_fromMethod" -Dtests.seed=2187012C1883C8A0 -Dtests.security.manager=false -Dtests.locale=sr-BA -Dtests.timezone=W-SU -Druntime.java=11
REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.knn.index.FaissIT.testDocUpdate" -Dtests.seed=2187012C1883C8A0 -Dtests.security.manager=false -Dtests.locale=sr-BA -Dtests.timezone=W-SU -Druntime.java=11

Suite: Test class org.opensearch.knn.index.FaissIT
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.knn.index.FaissIT.testEndToEnd_fromModel" -Dtests.seed=2187012C1883C8A0 -Dtests.security.manager=false -Dtests.locale=sr-BA -Dtests.timezone=W-SU -Druntime.java=11
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.knn.index.FaissIT.testEndToEnd_fromModel" -Dtests.seed=2187012C1883C8A0 -Dtests.security.manager=false -Dtests.locale=sr-BA -Dtests.timezone=W-SU -Druntime.java=11
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.knn.index.FaissIT.testEndToEnd_fromModel" -Dtests.seed=2187012C1883C8A0 -Dtests.security.manager=false -Dtests.locale=sr-BA -Dtests.timezone=W-SU -Druntime.java=11
  2> java.net.ConnectException: Connection refused
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:907)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:301)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:289)
        at org.opensearch.knn.KNNRestTestCase.getModel(KNNRestTestCase.java:932)
        at org.opensearch.knn.KNNRestTestCase.assertTrainingSucceeds(KNNRestTestCase.java:945)
        at org.opensearch.knn.index.FaissIT.testEndToEnd_fromModel(FaissIT.java:235)

        Caused by:
        java.net.ConnectException: Connection refused
            at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
            at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
            at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
            at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
            at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
            at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
            at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
            at java.base/java.lang.Thread.run(Thread.java:829)

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[jmazanec15]

@zelinh Where is the full stack trace? It appears the trace above just contains exception for runner.

[jmazanec15]

Can you provide environment in which cluster was running @zelinh?

[zelinh]

Docker container: opensearchstaging/ci-runner:centos7-x64-arm64-jdkmulti-node10.24.1-cypress6.9.1-20211028

[jmazanec15]

Repro locally:

java.lang.UnsatisfiedLinkError: no opensearchknn_nmslib in java.library.path: [., /home/test/opensearch-1.3.2/plugins/opensearch-knn/lib, /usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2680)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:807)
        at java.base/java.lang.System.loadLibrary(System.java:1907)
        at org.opensearch.knn.jni.NmslibService.lambda$static$0(NmslibService.java:34)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
        at org.opensearch.knn.jni.NmslibService.<clinit>(NmslibService.java:33)
        at org.opensearch.knn.jni.JNIService.createIndex(JNIService.java:36)
        at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.lambda$createKNNIndexFromScratch$1(KNN80DocValuesConsumer.java:198)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
        at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.createKNNIndexFromScratch(KNN80DocValuesConsumer.java:196)
        at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.addKNNBinaryField(KNN80DocValuesConsumer.java:122)
        at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.addBinaryField(KNN80DocValuesConsumer.java:70)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addBinaryField(PerFieldDocValuesFormat.java:114)
        at org.apache.lucene.index.BinaryDocValuesWriter.flush(BinaryDocValuesWriter.java:123)
        at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:348)
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:228)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476)
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:605)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258)
        at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
        at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:173)
        at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:81)
        at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:53)
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:400)
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:380)
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
        at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1897)
        at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1874)
        at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1201)
        at org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.<init>(TransportWriteAction.java:420)
        at org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:305)
        at org.opensearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:176)
        at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78)
        at org.opensearch.action.ActionListener.completeWith(ActionListener.java:351)
        at org.opensearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:264)
        at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:227)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
        at org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:276)
        at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:165)
        at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:98)
        at org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:220)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:832)