Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] SegmentReplicationWithRemoteStorePressureIT.testAddReplicaWhileWritesBlocked flaky test failure #8887

Closed
dreamer-89 opened this issue Jul 26, 2023 · 3 comments · Fixed by #9501
Closed

[BUG] SegmentReplicationWithRemoteStorePressureIT.testAddReplicaWhileWritesBlocked flaky test failure #8887

dreamer-89 opened this issue Jul 26, 2023 · 3 comments · Fixed by #9501
Assignees
@mch2
Labels
bug Something isn't working distributed framework flaky-test Random test failure that succeeds on second run :test Adding or fixing a test >test-failure Test failure from CI, local build, etc.

Comments

@dreamer-89
Copy link
Member

org.opensearch.remotestore.SegmentReplicationWithRemoteStorePressureIT.testAddReplicaWhileWritesBlocked flaky test failure.

Gradle check: https://build.ci.opensearch.org/job/gradle-check/20946
@dreamer-89 dreamer-89 added bug Something isn't working untriaged :test Adding or fixing a test >test-failure Test failure from CI, local build, etc. flaky-test Random test failure that succeeds on second run labels Jul 26, 2023
@dreamer-89 dreamer-89 mentioned this issue Jul 26, 2023
Merged
@Rishikesh1159 Rishikesh1159 mentioned this issue Aug 1, 2023
Merged
6 tasks
@ashking94 ashking94 mentioned this issue Aug 5, 2023
Merged
6 tasks
@mch2 mch2 mentioned this issue Aug 21, 2023
Closed
16 tasks
@tlfeng
Copy link
Collaborator

tlfeng commented Aug 22, 2023 •
edited
Loading

Console output in the build No.20946

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.SegmentReplicationWithRemoteStorePressureIT.testAddReplicaWhileWritesBlocked" -Dtests.seed=815CC5351C020C0A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-CR -Dtests.timezone=Eire -Druntime.java=17

org.opensearch.remotestore.SegmentReplicationWithRemoteStorePressureIT > testAddReplicaWhileWritesBlocked FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([815CC5351C020C0A:F83052A582BD6B64]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:1011)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:942)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:931)
        at org.opensearch.index.SegmentReplicationPressureIT.testAddReplicaWhileWritesBlocked(SegmentReplicationPressureIT.java:176)
...

The assertion in this line failed:

OpenSearch/server/src/internalClusterTest/java/org/opensearch/index/SegmentReplicationPressureIT.java

Line 176 in e354201

ensureGreen(INDEX_NAME);

Test failure occurrence copied from the Meta issue:

5 org.opensearch.remotestore.SegmentReplicationWithRemoteStorePressureIT.testAddReplicaWhileWritesBlocked (20771,20796,20806,20899,20900)
4 org.opensearch.remotestore.SegmentReplicationWithRemoteStorePressureIT.testAddReplicaWhileWritesBlocked (20900,20913,20920,21006)

Detailed logs:

  1> [2023-07-26T02:25:18,316][INFO ][o.o.r.SegmentReplicationWithRemoteStorePressureIT] [testAddReplicaWhileWritesBlocked] ensureGreen timed out, cluster state:
  1> cluster uuid: D2tAtY63RMeJRBT3zfMltQ [committed: true]
  1> version: 15
  1> state uuid: ML9jsIUxStOymiXPjtlmDw
  1> from_diff: false
  1> meta data version: 10
  1>    coordination_metadata:
  1>       term: 1
  1>       last_committed_config: VotingConfiguration{PYRdPC2HSKCmNXgGUYzZKA,Cme57aq_S5eua8OtUF--Tw,9AwKz3dATLKV4I3D-89rig}
  1>       last_accepted_config: VotingConfiguration{PYRdPC2HSKCmNXgGUYzZKA,Cme57aq_S5eua8OtUF--Tw,9AwKz3dATLKV4I3D-89rig}
  1>       voting tombstones: []
  1>    [test-idx-1/cRKh4FmHTSa6x6cWOVmNhA]: v[9], mv[2], sv[2], av[1]
  1>       0: p_term [1], isa_ids [wHcCJoXLTZenqDC82I8niw, 42ohKRQJR-KKUzzjL1g2DA]
  1> metadata customs:
  1>    repositories: {"test-remote-store-repo":{"type":"fs","settings":{"location":"/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.remotestore.SegmentReplicationWithRemoteStorePressureIT_815CC5351C020C0A-002/tempDir-002/repos/gosXqsdglX"},"generation":-2,"pending_generation":-1}}   index-graveyard: IndexGraveyard[[]]
  1> nodes: 
  1>    {node_t2}{PYRdPC2HSKCmNXgGUYzZKA}{HAyK_7h3QOC6MfEIyyiu4w}{127.0.0.1}{127.0.0.1:40683}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t0}{Cme57aq_S5eua8OtUF--Tw}{XqxawrH9Tl-3NdmPQdRmBQ}{127.0.0.1}{127.0.0.1:45019}{m}{shard_indexing_pressure_enabled=true}, local, cluster-manager
  1>    {node_t1}{9AwKz3dATLKV4I3D-89rig}{L0pL_bpSRGmHd0fAkTtLzA}{127.0.0.1}{127.0.0.1:40945}{dimr}{shard_indexing_pressure_enabled=true}
  1>    {node_t3}{O0NYxJ6xRBy5BnFHH44OpQ}{7p66-gi7T1a7o4jz530oiQ}{127.0.0.1}{127.0.0.1:41945}{dimr}{shard_indexing_pressure_enabled=true}
  1> routing_table (version 8):
  1> -- index [[test-idx-1/cRKh4FmHTSa6x6cWOVmNhA]]
  1> ----shard_id [test-idx-1][0]
  1> --------[test-idx-1][0], node[PYRdPC2HSKCmNXgGUYzZKA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=fNS7lxOkR4SHAJ7wLUfXUg], unassigned_info[[reason=ALLOCATION_FAILED], at[2023-07-26T01:24:45.611Z], failed_attempts[1], delayed=false, details[failed shard on node [PYRdPC2HSKCmNXgGUYzZKA]: replica too far behind primary, marking as stale], allocation_status[no_attempt]]
  1> --------[test-idx-1][0], node[9AwKz3dATLKV4I3D-89rig], [P], s[STARTED], a[id=42ohKRQJR-KKUzzjL1g2DA]
  1> --------[test-idx-1][0], node[O0NYxJ6xRBy5BnFHH44OpQ], [R], s[STARTED], a[id=wHcCJoXLTZenqDC82I8niw]

  1> routing_nodes:
  1> -----node_id[PYRdPC2HSKCmNXgGUYzZKA][V]
  1> --------[test-idx-1][0], node[PYRdPC2HSKCmNXgGUYzZKA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=fNS7lxOkR4SHAJ7wLUfXUg], unassigned_info[[reason=ALLOCATION_FAILED], at[2023-07-26T01:24:45.611Z], failed_attempts[1], delayed=false, details[failed shard on node [PYRdPC2HSKCmNXgGUYzZKA]: replica too far behind primary, marking as stale], allocation_status[no_attempt]]
  1> -----node_id[9AwKz3dATLKV4I3D-89rig][V]
  1> --------[test-idx-1][0], node[9AwKz3dATLKV4I3D-89rig], [P], s[STARTED], a[id=42ohKRQJR-KKUzzjL1g2DA]
  1> -----node_id[O0NYxJ6xRBy5BnFHH44OpQ][V]
  1> --------[test-idx-1][0], node[O0NYxJ6xRBy5BnFHH44OpQ], [R], s[STARTED], a[id=wHcCJoXLTZenqDC82I8niw]
  1> ---- unassigned

@mch2 mch2 mentioned this issue Aug 23, 2023
Merged
6 tasks
@mch2 mch2 self-assigned this Aug 23, 2023
@github-project-automation github-project-automation bot moved this from Todo to Done in Segment Replication Aug 23, 2023
@VachaShah VachaShah mentioned this issue Sep 20, 2023
Merged
6 tasks
@reta reta reopened this Jun 10, 2024
@github-project-automation github-project-automation bot moved this from Done to In Progress in Segment Replication Jun 10, 2024
@reta
Copy link
Collaborator

reta commented Jun 10, 2024

The issue is back: https://build.ci.opensearch.org/job/gradle-check/40573/testReport/junit/org.opensearch.remotestore/SegmentReplicationWithRemoteStorePressureIT/testAddReplicaWhileWritesBlocked/

java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
	at __randomizedtesting.SeedInfo.seed([3BEC575734C5562E:4280C0C7AA7A3140]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1128)
	at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:118)
	at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:113)
	at org.opensearch.index.SegmentReplicationPressureIT.testAddReplicaWhileWritesBlocked(SegmentReplicationPressureIT.java:180)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more
	Suppressed: java.lang.AssertionError: Expected search hits on node: node_t2 to be at least 4 but was: 3
		at org.junit.Assert.fail(Assert.java:89)
		at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:123)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 42 more

Standard Output

[2024-06-10T15:51:11,375][INFO ][o.o.t.OpenSearchTestClusterRule] [testWritesRejected] [SegmentReplicationWithRemoteStorePressureIT#testWritesRejected]: setting up test
[2024-06-10T15:51:11,377][INFO ][o.o.t.InternalTestCluster] [testWritesRejected] Setup InternalTestCluster [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster] with seed [EEC00374229F2435] using [0] dedicated cluster-managers, [0] (data) nodes and [0] coord only nodes (min_cluster_manager_nodes are [auto-managed])
[2024-06-10T15:51:11,377][INFO ][o.o.t.OpenSearchTestClusterRule] [testWritesRejected] [SegmentReplicationWithRemoteStorePressureIT#testWritesRejected]: all set up test
[2024-06-10T15:51:11,379][INFO ][o.o.r.SegmentReplicationWithRemoteStorePressureIT] [testWritesRejected] before test
[2024-06-10T15:51:11,382][INFO ][o.o.n.Node               ] [testWritesRejected] version[3.0.0-SNAPSHOT], pid[273801], build[unknown/ead733cf85d7433dd9b075048f43dbae273e5f76/2024-06-10T21:23:34.135290907Z], OS[Linux/5.15.0-1056-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-06-10T15:51:11,382][INFO ][o.o.n.Node               ] [testWritesRejected] JVM home [/home/ubuntu/.gradle/caches/transforms-4/694b5d2aff184189bcea38c919099622/transformed/linux-21.0.3-x64.tar.gz]
[2024-06-10T15:51:11,382][DEPRECATION][o.o.d.n.Node             ] [testWritesRejected] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
[2024-06-10T15:51:11,383][INFO ][o.o.n.Node               ] [testWritesRejected] JVM arguments [-Dgradle.dist.lib=/home/ubuntu/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib, -Dgradle.user.home=/home/ubuntu/.gradle, -Dgradle.worker.jar=/home/ubuntu/.gradle/caches/8.7/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.coverage=true, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=3BEC575734C5562E, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:ReservedCodeCacheSize=64m, -esa, --add-modules=jdk.incubator.vector, -XX:HeapDumpPath=/var/jenkins/workspace/gradle-check/search/server/build/heapdump, -javaagent:/var/jenkins/workspace/gradle-check/search/server/build/tmp/.cache/expanded/zip_b7e3e3efc9fa7b01ca2c5374419709e6/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea]
[2024-06-10T15:51:11,383][WARN ][o.o.n.Node               ] [testWritesRejected] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-06-10T15:51:11,383][INFO ][o.o.p.PluginsService     ] [testWritesRejected] no modules loaded
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-06-10T15:51:11,384][INFO ][o.o.e.ExtensionsManager  ] [testWritesRejected] ExtensionsManager initialized
[2024-06-10T15:51:11,396][INFO ][o.o.e.NodeEnvironment    ] [testWritesRejected] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [251.8gb], net total_space [290.5gb], types [ext4]
[2024-06-10T15:51:11,397][INFO ][o.o.e.NodeEnvironment    ] [testWritesRejected] heap size [3gb], compressed ordinary object pointers [true]
[2024-06-10T15:51:11,403][INFO ][o.o.n.Node               ] [testWritesRejected] node name [node_t0], node ID [PDqg5oqMTg2-G-BUr4dv1A], cluster name [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster], roles [cluster_manager]
[2024-06-10T15:51:11,419][INFO ][o.o.d.DiscoveryModule    ] [testWritesRejected] using discovery type [zen] and seed hosts providers [settings, file]
[2024-06-10T15:51:11,428][WARN ][o.o.g.DanglingIndicesState] [testWritesRejected] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-06-10T15:51:11,438][INFO ][o.o.n.Node               ] [testWritesRejected] initialized
[2024-06-10T15:51:11,440][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#1]]] starting ...
[2024-06-10T15:51:11,443][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#1]]] publish_address {127.0.0.1:41823}, bound_addresses {[::1]:39455}, {127.0.0.1:41823}
[2024-06-10T15:51:11,444][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#1]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:51:11,451][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#1]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:51:11,560][INFO ][o.o.c.c.Coordinator      ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#1]]] setting initial configuration to VotingConfiguration{PDqg5oqMTg2-G-BUr4dv1A}
[2024-06-10T15:51:11,656][INFO ][o.o.c.s.MasterService    ] [node_t0] elected-as-cluster-manager ([1] nodes joined)[{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}]}
[2024-06-10T15:51:11,729][INFO ][o.o.c.c.CoordinationState] [node_t0] cluster UUID set to [Il_mLeBUSBGNXT-yvTA_KA]
[2024-06-10T15:51:11,765][INFO ][o.o.c.s.ClusterApplierService] [node_t0] cluster-manager node changed {previous [], current [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2024-06-10T15:51:11,766][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#1]]] started
[2024-06-10T15:51:11,766][INFO ][o.o.d.PeerFinder         ] [node_t0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:51:11,766][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:11,766][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:11,770][INFO ][o.o.g.GatewayService     ] [node_t0] recovered [0] indices into cluster_state
[2024-06-10T15:51:11,776][INFO ][o.o.n.Node               ] [testWritesRejected] version[3.0.0-SNAPSHOT], pid[273801], build[unknown/ead733cf85d7433dd9b075048f43dbae273e5f76/2024-06-10T21:23:34.135290907Z], OS[Linux/5.15.0-1056-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-06-10T15:51:11,776][INFO ][o.o.n.Node               ] [testWritesRejected] JVM home [/home/ubuntu/.gradle/caches/transforms-4/694b5d2aff184189bcea38c919099622/transformed/linux-21.0.3-x64.tar.gz]
[2024-06-10T15:51:11,776][INFO ][o.o.n.Node               ] [testWritesRejected] JVM arguments [-Dgradle.dist.lib=/home/ubuntu/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib, -Dgradle.user.home=/home/ubuntu/.gradle, -Dgradle.worker.jar=/home/ubuntu/.gradle/caches/8.7/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.coverage=true, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=3BEC575734C5562E, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:ReservedCodeCacheSize=64m, -esa, --add-modules=jdk.incubator.vector, -XX:HeapDumpPath=/var/jenkins/workspace/gradle-check/search/server/build/heapdump, -javaagent:/var/jenkins/workspace/gradle-check/search/server/build/tmp/.cache/expanded/zip_b7e3e3efc9fa7b01ca2c5374419709e6/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea]
[2024-06-10T15:51:11,776][WARN ][o.o.n.Node               ] [testWritesRejected] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] no modules loaded
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-06-10T15:51:11,777][INFO ][o.o.e.ExtensionsManager  ] [testWritesRejected] ExtensionsManager initialized
[2024-06-10T15:51:11,780][INFO ][o.o.e.NodeEnvironment    ] [testWritesRejected] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [251.8gb], net total_space [290.5gb], types [ext4]
[2024-06-10T15:51:11,780][INFO ][o.o.e.NodeEnvironment    ] [testWritesRejected] heap size [3gb], compressed ordinary object pointers [true]
[2024-06-10T15:51:11,783][INFO ][o.o.n.Node               ] [testWritesRejected] node name [node_t1], node ID [mldugdb4QeG0mt7x2c8ZWQ], cluster name [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-06-10T15:51:11,802][INFO ][o.o.d.DiscoveryModule    ] [testWritesRejected] using discovery type [zen] and seed hosts providers [settings, file]
[2024-06-10T15:51:11,810][WARN ][o.o.g.DanglingIndicesState] [testWritesRejected] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-06-10T15:51:11,819][INFO ][o.o.n.Node               ] [testWritesRejected] initialized
[2024-06-10T15:51:11,820][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#2]]] starting ...
[2024-06-10T15:51:11,823][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#2]]] publish_address {127.0.0.1:40239}, bound_addresses {[::1]:39255}, {127.0.0.1:40239}
[2024-06-10T15:51:11,824][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#2]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:51:11,830][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#2]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:51:11,830][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#2]]] Remote clusters initialized successfully.
[2024-06-10T15:51:11,929][INFO ][o.o.c.s.MasterService    ] [node_t0] node-join[{node_t1}{mldugdb4QeG0mt7x2c8ZWQ}{eJchZ0fiQH6jt6eZM1088g}{127.0.0.1}{127.0.0.1:40239}{dimr}{shard_indexing_pressure_enabled=true} join existing leader], term: 1, version: 3, delta: added {{node_t1}{mldugdb4QeG0mt7x2c8ZWQ}{eJchZ0fiQH6jt6eZM1088g}{127.0.0.1}{127.0.0.1:40239}{dimr}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:51:11,996][INFO ][o.o.c.s.ClusterApplierService] [node_t1] cluster-manager node changed {previous [], current [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:51:11,997][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#2]]] started
[2024-06-10T15:51:11,997][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t1}{mldugdb4QeG0mt7x2c8ZWQ}{eJchZ0fiQH6jt6eZM1088g}{127.0.0.1}{127.0.0.1:40239}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 3, reason: Publication{term=1, version=3}
[2024-06-10T15:51:11,998][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:11,998][INFO ][o.o.d.PeerFinder         ] [node_t1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t1}{mldugdb4QeG0mt7x2c8ZWQ}{eJchZ0fiQH6jt6eZM1088g}{127.0.0.1}{127.0.0.1:40239}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:51:12,000][INFO ][o.o.p.PluginsService     ] [node_t0] PluginService:onIndexModule index:[test-idx-1/Cy8TY-5nQ3GLO4Po1pCW4w]
[2024-06-10T15:51:12,001][INFO ][o.o.c.m.MetadataCreateIndexService] [node_t0] [test-idx-1] creating index, cause [api], templates [], shards [1]/[1]
[2024-06-10T15:51:12,002][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:12,052][INFO ][o.o.p.PluginsService     ] [node_t1] PluginService:onIndexModule index:[test-idx-1/Cy8TY-5nQ3GLO4Po1pCW4w]
[2024-06-10T15:51:12,134][INFO ][o.o.i.t.RemoteFsTranslog ] [node_t1] [test-idx-1][0] Downloaded data from remote translog till maxSeqNo = -1
[2024-06-10T15:51:12,156][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:51:12,156][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:51:12,156][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:51:12,156][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:51:12,262][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=STARTED engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:51:12,262][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=STARTED engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:51:12,262][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Scheduled retry with didRefresh=true
[2024-06-10T15:51:12,325][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:12,328][INFO ][o.o.n.Node               ] [testWritesRejected] version[3.0.0-SNAPSHOT], pid[273801], build[unknown/ead733cf85d7433dd9b075048f43dbae273e5f76/2024-06-10T21:23:34.135290907Z], OS[Linux/5.15.0-1056-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-06-10T15:51:12,328][INFO ][o.o.n.Node               ] [testWritesRejected] JVM home [/home/ubuntu/.gradle/caches/transforms-4/694b5d2aff184189bcea38c919099622/transformed/linux-21.0.3-x64.tar.gz]
[2024-06-10T15:51:12,328][INFO ][o.o.n.Node               ] [testWritesRejected] JVM arguments [-Dgradle.dist.lib=/home/ubuntu/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib, -Dgradle.user.home=/home/ubuntu/.gradle, -Dgradle.worker.jar=/home/ubuntu/.gradle/caches/8.7/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.coverage=true, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=3BEC575734C5562E, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:ReservedCodeCacheSize=64m, -esa, --add-modules=jdk.incubator.vector, -XX:HeapDumpPath=/var/jenkins/workspace/gradle-check/search/server/build/heapdump, -javaagent:/var/jenkins/workspace/gradle-check/search/server/build/tmp/.cache/expanded/zip_b7e3e3efc9fa7b01ca2c5374419709e6/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea]
[2024-06-10T15:51:12,328][WARN ][o.o.n.Node               ] [testWritesRejected] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] no modules loaded
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.p.PluginsService     ] [testWritesRejected] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-06-10T15:51:12,329][INFO ][o.o.e.ExtensionsManager  ] [testWritesRejected] ExtensionsManager initialized
[2024-06-10T15:51:12,333][INFO ][o.o.e.NodeEnvironment    ] [testWritesRejected] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [251.8gb], net total_space [290.5gb], types [ext4]
[2024-06-10T15:51:12,333][INFO ][o.o.e.NodeEnvironment    ] [testWritesRejected] heap size [3gb], compressed ordinary object pointers [true]
[2024-06-10T15:51:12,341][INFO ][o.o.n.Node               ] [testWritesRejected] node name [node_t2], node ID [UYf3hWovSo6sfzx1N2L6YQ], cluster name [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-06-10T15:51:12,358][INFO ][o.o.d.DiscoveryModule    ] [testWritesRejected] using discovery type [zen] and seed hosts providers [settings, file]
[2024-06-10T15:51:12,370][WARN ][o.o.g.DanglingIndicesState] [testWritesRejected] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-06-10T15:51:12,383][INFO ][o.o.n.Node               ] [testWritesRejected] initialized
[2024-06-10T15:51:12,387][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#3]]] starting ...
[2024-06-10T15:51:12,391][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#3]]] publish_address {127.0.0.1:35127}, bound_addresses {[::1]:46013}, {127.0.0.1:35127}
[2024-06-10T15:51:12,391][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#3]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:51:12,396][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#3]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:51:12,396][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#3]]] Remote clusters initialized successfully.
[2024-06-10T15:51:12,522][INFO ][o.o.c.s.MasterService    ] [node_t0] node-join[{node_t2}{UYf3hWovSo6sfzx1N2L6YQ}{Op-w1W2dT3ilj-aJHJzBpg}{127.0.0.1}{127.0.0.1:35127}{dimr}{shard_indexing_pressure_enabled=true} join existing leader], term: 1, version: 6, delta: added {{node_t2}{UYf3hWovSo6sfzx1N2L6YQ}{Op-w1W2dT3ilj-aJHJzBpg}{127.0.0.1}{127.0.0.1:35127}{dimr}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:51:12,526][INFO ][o.o.c.s.ClusterApplierService] [node_t1] added {{node_t2}{UYf3hWovSo6sfzx1N2L6YQ}{Op-w1W2dT3ilj-aJHJzBpg}{127.0.0.1}{127.0.0.1:35127}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:51:12,598][INFO ][o.o.c.s.ClusterApplierService] [node_t2] cluster-manager node changed {previous [], current [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true},{node_t1}{mldugdb4QeG0mt7x2c8ZWQ}{eJchZ0fiQH6jt6eZM1088g}{127.0.0.1}{127.0.0.1:40239}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:51:12,598][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1242989699822312395]-HASH=[2315BA912C1]-cluster[T#3]]] started
[2024-06-10T15:51:12,599][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t2}{UYf3hWovSo6sfzx1N2L6YQ}{Op-w1W2dT3ilj-aJHJzBpg}{127.0.0.1}{127.0.0.1:35127}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 6, reason: Publication{term=1, version=6}
[2024-06-10T15:51:12,600][INFO ][o.o.d.PeerFinder         ] [node_t2] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t2}{UYf3hWovSo6sfzx1N2L6YQ}{Op-w1W2dT3ilj-aJHJzBpg}{127.0.0.1}{127.0.0.1:35127}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:51:12,824][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:12,852][INFO ][o.o.p.PluginsService     ] [node_t2] PluginService:onIndexModule index:[test-idx-1/Cy8TY-5nQ3GLO4Po1pCW4w]
[2024-06-10T15:51:13,072][INFO ][o.o.i.r.RecoverySourceHandler] [node_t1] [test-idx-1][0][recover to node_t2] finalizing recovery took [17.1ms]
[2024-06-10T15:51:13,073][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test-idx-1][0]]]).
[2024-06-10T15:51:13,178][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:13,191][INFO ][o.o.p.PluginsService     ] [node_t0] PluginService:onIndexModule index:[test-idx-1/Cy8TY-5nQ3GLO4Po1pCW4w]
[2024-06-10T15:51:13,194][INFO ][o.o.c.m.MetadataMappingService] [node_t0] [test-idx-1/Cy8TY-5nQ3GLO4Po1pCW4w] create_mapping
[2024-06-10T15:51:14,748][WARN ][o.o.i.SegmentReplicationPressureService] [node_t1] Rejecting write requests for shard, stale shards [100.0%] shards: [SegmentReplicationShardStats{allocationId=yylfK3XGTbmufHyRm7IOKQ, checkpointsBehindCount=3, bytesBehindCount=10689, currentReplicationLagMillis=1341, currentReplicationTimeMillis=1320, lastCompletedReplicationTimeMillis=0, currentReplicationState=null}]
[2024-06-10T15:51:14,751][WARN ][o.o.i.SegmentReplicationPressureService] [node_t1] Rejecting write requests for shard, stale shards [100.0%] shards: [SegmentReplicationShardStats{allocationId=yylfK3XGTbmufHyRm7IOKQ, checkpointsBehindCount=3, bytesBehindCount=10689, currentReplicationLagMillis=1345, currentReplicationTimeMillis=1324, lastCompletedReplicationTimeMillis=0, currentReplicationState=null}]
[2024-06-10T15:51:15,414][INFO ][o.o.a.a.c.r.c.TransportCleanupRepositoryAction] [node_t0] Running cleanup operations on repository [test-remote-store-repo][-1]
[2024-06-10T15:51:15,434][INFO ][o.o.a.a.c.r.c.TransportCleanupRepositoryAction] [node_t0] Done with repository cleanup on [test-remote-store-repo][-1] with result [{"deleted_bytes":0,"deleted_blobs":0}]
[2024-06-10T15:51:15,436][INFO ][o.o.r.SegmentReplicationWithRemoteStorePressureIT] [testWritesRejected] after test
[2024-06-10T15:51:15,436][INFO ][o.o.t.OpenSearchTestClusterRule] [testWritesRejected] [SegmentReplicationWithRemoteStorePressureIT#testWritesRejected]: cleaning up after test
[2024-06-10T15:51:15,445][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:15,450][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_t0] [test-idx-1/Cy8TY-5nQ3GLO4Po1pCW4w] deleting index
[2024-06-10T15:51:15,451][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:51:15,538][INFO ][o.o.i.t.t.TranslogTransferManager] [node_t1] [test-idx-1][0] Deleted all remote translog data at path=[_00100000001000][Cy8TY-5nQ3GLO4Po1pCW4w][0][translog][data]
[2024-06-10T15:51:15,546][INFO ][o.o.i.t.t.TranslogTransferManager] [node_t1] [test-idx-1][0] Deleted all remote translog data at path=[V10110000110101][Cy8TY-5nQ3GLO4Po1pCW4w][0][translog][metadata]
[2024-06-10T15:51:15,564][INFO ][o.o.n.Node               ] [testWritesRejected] stopping ...
[2024-06-10T15:51:15,564][INFO ][o.o.c.c.FollowersChecker ] [testWritesRejected] FollowerChecker{discoveryNode={node_t2}{UYf3hWovSo6sfzx1N2L6YQ}{Op-w1W2dT3ilj-aJHJzBpg}{127.0.0.1}{127.0.0.1:35127}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-06-10T15:51:15,565][INFO ][o.o.c.c.FollowersChecker ] [testWritesRejected] FollowerChecker{discoveryNode={node_t1}{mldugdb4QeG0mt7x2c8ZWQ}{eJchZ0fiQH6jt6eZM1088g}{127.0.0.1}{127.0.0.1:40239}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-06-10T15:51:15,565][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t1}{mldugdb4QeG0mt7x2c8ZWQ}{eJchZ0fiQH6jt6eZM1088g}{127.0.0.1}{127.0.0.1:40239}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-06-10T15:51:15,565][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t2}{UYf3hWovSo6sfzx1N2L6YQ}{Op-w1W2dT3ilj-aJHJzBpg}{127.0.0.1}{127.0.0.1:35127}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-06-10T15:51:15,566][INFO ][o.o.n.Node               ] [testWritesRejected] stopped
[2024-06-10T15:51:15,566][INFO ][o.o.n.Node               ] [testWritesRejected] closing ...
[2024-06-10T15:51:15,565][INFO ][o.o.c.c.Coordinator      ] [node_t1] cluster-manager node [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:41823][disconnected] disconnected
[2024-06-10T15:51:15,565][INFO ][o.o.c.c.Coordinator      ] [node_t2] cluster-manager node [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:41823][disconnected] disconnected
[2024-06-10T15:51:15,566][INFO ][o.o.c.s.ClusterApplierService] [node_t1] cluster-manager node changed {previous [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 14, reason: becoming candidate: onLeaderFailure
[2024-06-10T15:51:15,566][INFO ][o.o.c.s.ClusterApplierService] [node_t2] cluster-manager node changed {previous [{node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 14, reason: becoming candidate: onLeaderFailure
[2024-06-10T15:51:15,567][INFO ][o.o.n.Node               ] [testWritesRejected] closed
[2024-06-10T15:51:15,567][WARN ][o.o.c.NodeConnectionsService] [node_t1] failed to connect to {node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:41823] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1098) ~[main/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	... 9 more
[2024-06-10T15:51:15,567][WARN ][o.o.c.NodeConnectionsService] [node_t2] failed to connect to {node_t0}{PDqg5oqMTg2-G-BUr4dv1A}{GuK_55mXTeOFlDCUqVAubg}{127.0.0.1}{127.0.0.1:41823}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:41823] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1098) ~[main/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	... 9 more
[2024-06-10T15:51:15,568][INFO ][o.o.n.Node               ] [testWritesRejected] stopping ...
[2024-06-10T15:51:15,569][INFO ][o.o.n.Node               ] [testWritesRejected] stopped
[2024-06-10T15:51:15,569][INFO ][o.o.n.Node               ] [testWritesRejected] closing ...
[2024-06-10T15:51:15,571][INFO ][o.o.n.Node               ] [testWritesRejected] closed
[2024-06-10T15:51:15,591][INFO ][o.o.n.Node               ] [testWritesRejected] stopping ...
[2024-06-10T15:51:15,592][INFO ][o.o.n.Node               ] [testWritesRejected] stopped
[2024-06-10T15:51:15,592][INFO ][o.o.n.Node               ] [testWritesRejected] closing ...
[2024-06-10T15:51:15,593][INFO ][o.o.n.Node               ] [testWritesRejected] closed
[2024-06-10T15:51:15,594][INFO ][o.o.t.OpenSearchTestClusterRule] [testWritesRejected] [SegmentReplicationWithRemoteStorePressureIT#testWritesRejected]: cleaned up after test
[2024-06-10T15:51:15,599][INFO ][o.o.t.OpenSearchTestClusterRule] [testFailStaleReplica] [SegmentReplicationWithRemoteStorePressureIT#testFailStaleReplica]: setting up test
[2024-06-10T15:51:15,602][INFO ][o.o.t.InternalTestCluster] [testFailStaleReplica] Setup InternalTestCluster [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster] with seed [96D8EAB516C105E] using [0] dedicated cluster-managers, [0] (data) nodes and [0] coord only nodes (min_cluster_manager_nodes are [auto-managed])
[2024-06-10T15:51:15,602][INFO ][o.o.t.OpenSearchTestClusterRule] [testFailStaleReplica] [SegmentReplicationWithRemoteStorePressureIT#testFailStaleReplica]: all set up test
[2024-06-10T15:51:15,605][INFO ][o.o.r.SegmentReplicationWithRemoteStorePressureIT] [testFailStaleReplica] before test
[2024-06-10T15:51:15,607][INFO ][o.o.n.Node               ] [testFailStaleReplica] version[3.0.0-SNAPSHOT], pid[273801], build[unknown/ead733cf85d7433dd9b075048f43dbae273e5f76/2024-06-10T21:23:34.135290907Z], OS[Linux/5.15.0-1056-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-06-10T15:51:15,607][INFO ][o.o.n.Node               ] [testFailStaleReplica] JVM home [/home/ubuntu/.gradle/caches/transforms-4/694b5d2aff184189bcea38c919099622/transformed/linux-21.0.3-x64.tar.gz]
[2024-06-10T15:51:15,607][DEPRECATION][o.o.d.n.Node             ] [testFailStaleReplica] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
[2024-06-10T15:51:15,607][INFO ][o.o.n.Node               ] [testFailStaleReplica] JVM arguments [-Dgradle.dist.lib=/home/ubuntu/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib, -Dgradle.user.home=/home/ubuntu/.gradle, -Dgradle.worker.jar=/home/ubuntu/.gradle/caches/8.7/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.coverage=true, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=3BEC575734C5562E, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:ReservedCodeCacheSize=64m, -esa, --add-modules=jdk.incubator.vector, -XX:HeapDumpPath=/var/jenkins/workspace/gradle-check/search/server/build/heapdump, -javaagent:/var/jenkins/workspace/gradle-check/search/server/build/tmp/.cache/expanded/zip_b7e3e3efc9fa7b01ca2c5374419709e6/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea]
[2024-06-10T15:51:15,607][WARN ][o.o.n.Node               ] [testFailStaleReplica] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] no modules loaded
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.index.MockEngineFactoryPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.search.MockSearchService$TestPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.p.PluginsService     ] [testFailStaleReplica] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-06-10T15:51:15,608][INFO ][o.o.e.ExtensionsManager  ] [testFailStaleReplica] ExtensionsManager initialized
[2024-06-10T15:51:15,617][INFO ][o.o.e.NodeEnvironment    ] [testFailStaleReplica] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [251.8gb], net total_space [290.5gb], types [ext4]
[2024-06-10T15:51:15,617][INFO ][o.o.e.NodeEnvironment    ] [testFailStaleReplica] heap size [3gb], compressed ordinary object pointers [true]
[2024-06-10T15:51:15,626][INFO ][o.o.n.Node               ] [testFailStaleReplica] node name [node_t0], node ID [i0C53yCXTkmK6AjduPY1xA], cluster name [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster], roles [cluster_manager]
[2024-06-10T15:51:15,643][INFO ][o.o.d.DiscoveryModule    ] [testFailStaleReplica] using discovery type [zen] and seed hosts providers [settings, file]
[2024-06-10T15:51:15,654][WARN ][o.o.g.DanglingIndicesState] [testFailStaleReplica] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-06-10T15:51:15,665][INFO ][o.o.n.Node               ] [testFailStaleReplica] initialized
[2024-06-10T15:51:15,673][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster[T#1]]] starting ...
[2024-06-10T15:51:15,676][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster[T#1]]] publish_address {127.0.0.1:34585}, bound_addresses {[::1]:45839}, {127.0.0.1:34585}
[2024-06-10T15:51:15,677][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster[T#1]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:51:15,694][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster[T#1]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:51:15,809][INFO ][o.o.c.c.Coordinator      ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster[T#1]]] setting initial configuration to VotingConfiguration{i0C53yCXTkmK6AjduPY1xA}
[2024-06-10T15:51:15,925][INFO ][o.o.c.s.MasterService    ] [node_t0] elected-as-cluster-manager ([1] nodes joined)[{node_t0}{i0C53yCXTkmK6AjduPY1xA}{v2pgqQqDSP-Wgm6HSGFbVw}{127.0.0.1}{127.0.0.1:34585}{m}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{node_t0}{i0C53yCXTkmK6AjduPY1xA}{v2pgqQqDSP-Wgm6HSGFbVw}{127.0.0.1}{127.0.0.1:34585}{m}{shard_indexing_pressure_enabled=true}]}
[2024-06-10T15:51:15,975][INFO ][o.o.c.c.CoordinationState] [node_t0] cluster UUID set to [1Gi1UXojTgS2KiifubT5tw]
[2024-06-10T15:51:16,019][INFO ][o.o.c.s.ClusterApplierService] [node_t0] cluster-manager node changed {previous [], current [{node_t0}{i0C53yCXTkmK6AjduPY1xA}{v2pgqQqDSP-Wgm6HSGFbVw}{127.0.0.1}{127.0.0.1:34585}{m}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2024-06-10T15:51:16,019][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[679355985259401310]-HASH=[232577F2EEA]-cluster[T#1]]] started
[2024-06-10T15:51:16,020][INFO ][o.o.d.PeerFinder         ] [node_t0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t0}{i0C53yCXTkmK6AjduPY1xA}{v2pgqQqDSP-Wgm6HSGFbVw}{127.0.0.1}{127.0.0.1:34585}{m}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:51:16,020][WAR
...[truncated 222051 chars]...
15:53:16,991][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#1]]] starting ...
[2024-06-10T15:53:16,994][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#1]]] publish_address {127.0.0.1:45807}, bound_addresses {[::1]:33073}, {127.0.0.1:45807}
[2024-06-10T15:53:16,995][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#1]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:53:17,001][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#1]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:53:17,058][INFO ][o.o.c.c.Coordinator      ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#1]]] setting initial configuration to VotingConfiguration{WBopOJJxSy-VUkeNNw9K0w}
[2024-06-10T15:53:17,144][INFO ][o.o.c.s.MasterService    ] [node_t0] elected-as-cluster-manager ([1] nodes joined)[{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}]}
[2024-06-10T15:53:17,185][INFO ][o.o.c.c.CoordinationState] [node_t0] cluster UUID set to [-PR9IHT_R2S8iZdiyVGC1A]
[2024-06-10T15:53:17,268][INFO ][o.o.c.s.ClusterApplierService] [node_t0] cluster-manager node changed {previous [], current [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2024-06-10T15:53:17,269][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#1]]] started
[2024-06-10T15:53:17,269][INFO ][o.o.d.PeerFinder         ] [node_t0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:53:17,269][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:17,270][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:17,277][INFO ][o.o.g.GatewayService     ] [node_t0] recovered [0] indices into cluster_state
[2024-06-10T15:53:17,280][INFO ][o.o.n.Node               ] [testBulkWritesRejected] version[3.0.0-SNAPSHOT], pid[273801], build[unknown/ead733cf85d7433dd9b075048f43dbae273e5f76/2024-06-10T21:23:34.135290907Z], OS[Linux/5.15.0-1056-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-06-10T15:53:17,280][INFO ][o.o.n.Node               ] [testBulkWritesRejected] JVM home [/home/ubuntu/.gradle/caches/transforms-4/694b5d2aff184189bcea38c919099622/transformed/linux-21.0.3-x64.tar.gz]
[2024-06-10T15:53:17,281][INFO ][o.o.n.Node               ] [testBulkWritesRejected] JVM arguments [-Dgradle.dist.lib=/home/ubuntu/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib, -Dgradle.user.home=/home/ubuntu/.gradle, -Dgradle.worker.jar=/home/ubuntu/.gradle/caches/8.7/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.coverage=true, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=3BEC575734C5562E, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:ReservedCodeCacheSize=64m, -esa, --add-modules=jdk.incubator.vector, -XX:HeapDumpPath=/var/jenkins/workspace/gradle-check/search/server/build/heapdump, -javaagent:/var/jenkins/workspace/gradle-check/search/server/build/tmp/.cache/expanded/zip_b7e3e3efc9fa7b01ca2c5374419709e6/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea]
[2024-06-10T15:53:17,281][WARN ][o.o.n.Node               ] [testBulkWritesRejected] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] no modules loaded
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.node.NodeMocksPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.search.MockSearchService$TestPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-06-10T15:53:17,283][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-06-10T15:53:17,284][INFO ][o.o.e.ExtensionsManager  ] [testBulkWritesRejected] ExtensionsManager initialized
[2024-06-10T15:53:17,295][INFO ][o.o.e.NodeEnvironment    ] [testBulkWritesRejected] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [251.8gb], net total_space [290.5gb], types [ext4]
[2024-06-10T15:53:17,295][INFO ][o.o.e.NodeEnvironment    ] [testBulkWritesRejected] heap size [3gb], compressed ordinary object pointers [true]
[2024-06-10T15:53:17,326][INFO ][o.o.n.Node               ] [testBulkWritesRejected] node name [node_t1], node ID [34jq6quTRl-trBvBScha8g], cluster name [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-06-10T15:53:17,346][INFO ][o.o.d.DiscoveryModule    ] [testBulkWritesRejected] using discovery type [zen] and seed hosts providers [settings, file]
[2024-06-10T15:53:17,356][WARN ][o.o.g.DanglingIndicesState] [testBulkWritesRejected] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-06-10T15:53:17,370][INFO ][o.o.n.Node               ] [testBulkWritesRejected] initialized
[2024-06-10T15:53:17,371][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#2]]] starting ...
[2024-06-10T15:53:17,375][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#2]]] publish_address {127.0.0.1:42745}, bound_addresses {[::1]:38817}, {127.0.0.1:42745}
[2024-06-10T15:53:17,375][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#2]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:53:17,378][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#2]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:53:17,378][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#2]]] Remote clusters initialized successfully.
[2024-06-10T15:53:17,555][INFO ][o.o.c.s.MasterService    ] [node_t0] node-join[{node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true} join existing leader], term: 1, version: 3, delta: added {{node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:17,638][INFO ][o.o.c.s.ClusterApplierService] [node_t1] cluster-manager node changed {previous [], current [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:17,639][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#2]]] started
[2024-06-10T15:53:17,639][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 3, reason: Publication{term=1, version=3}
[2024-06-10T15:53:17,640][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:17,640][INFO ][o.o.d.PeerFinder         ] [node_t1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:53:17,643][INFO ][o.o.p.PluginsService     ] [node_t0] PluginService:onIndexModule index:[test-idx-1/1VhS7U_8Rzyrnnb6YmyJqQ]
[2024-06-10T15:53:17,644][INFO ][o.o.c.m.MetadataCreateIndexService] [node_t0] [test-idx-1] creating index, cause [api], templates [], shards [1]/[1]
[2024-06-10T15:53:17,644][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:17,724][INFO ][o.o.p.PluginsService     ] [node_t1] PluginService:onIndexModule index:[test-idx-1/1VhS7U_8Rzyrnnb6YmyJqQ]
[2024-06-10T15:53:17,831][INFO ][o.o.i.t.RemoteFsTranslog ] [node_t1] [test-idx-1][0] Downloaded data from remote translog till maxSeqNo = -1
[2024-06-10T15:53:17,866][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:53:17,866][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:53:17,867][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:53:17,867][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=RECOVERING engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:53:17,915][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=STARTED engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:53:17,915][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Skipped syncing segments with primaryMode=false indexShardState=STARTED engineType=InternalEngine recoverySourceType=EMPTY_STORE primary=true
[2024-06-10T15:53:17,915][INFO ][o.o.i.s.RemoteStoreRefreshListener] [node_t1] [test-idx-1][0] Scheduled retry with didRefresh=true
[2024-06-10T15:53:17,940][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:17,943][INFO ][o.o.n.Node               ] [testBulkWritesRejected] version[3.0.0-SNAPSHOT], pid[273801], build[unknown/ead733cf85d7433dd9b075048f43dbae273e5f76/2024-06-10T21:23:34.135290907Z], OS[Linux/5.15.0-1056-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-06-10T15:53:17,943][INFO ][o.o.n.Node               ] [testBulkWritesRejected] JVM home [/home/ubuntu/.gradle/caches/transforms-4/694b5d2aff184189bcea38c919099622/transformed/linux-21.0.3-x64.tar.gz]
[2024-06-10T15:53:17,943][INFO ][o.o.n.Node               ] [testBulkWritesRejected] JVM arguments [-Dgradle.dist.lib=/home/ubuntu/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib, -Dgradle.user.home=/home/ubuntu/.gradle, -Dgradle.worker.jar=/home/ubuntu/.gradle/caches/8.7/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.coverage=true, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=3BEC575734C5562E, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:ReservedCodeCacheSize=64m, -esa, --add-modules=jdk.incubator.vector, -XX:HeapDumpPath=/var/jenkins/workspace/gradle-check/search/server/build/heapdump, -javaagent:/var/jenkins/workspace/gradle-check/search/server/build/tmp/.cache/expanded/zip_b7e3e3efc9fa7b01ca2c5374419709e6/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea]
[2024-06-10T15:53:17,943][WARN ][o.o.n.Node               ] [testBulkWritesRejected] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] no modules loaded
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.node.NodeMocksPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.search.MockSearchService$TestPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-06-10T15:53:17,944][INFO ][o.o.e.ExtensionsManager  ] [testBulkWritesRejected] ExtensionsManager initialized
[2024-06-10T15:53:17,950][INFO ][o.o.e.NodeEnvironment    ] [testBulkWritesRejected] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [251.8gb], net total_space [290.5gb], types [ext4]
[2024-06-10T15:53:17,950][INFO ][o.o.e.NodeEnvironment    ] [testBulkWritesRejected] heap size [3gb], compressed ordinary object pointers [true]
[2024-06-10T15:53:17,957][INFO ][o.o.n.Node               ] [testBulkWritesRejected] node name [node_t2], node ID [AbPwwa0ISaWjQUdnc2vOqg], cluster name [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-06-10T15:53:17,972][INFO ][o.o.d.DiscoveryModule    ] [testBulkWritesRejected] using discovery type [zen] and seed hosts providers [settings, file]
[2024-06-10T15:53:17,980][WARN ][o.o.g.DanglingIndicesState] [testBulkWritesRejected] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-06-10T15:53:17,990][INFO ][o.o.n.Node               ] [testBulkWritesRejected] initialized
[2024-06-10T15:53:17,992][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#3]]] starting ...
[2024-06-10T15:53:17,995][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#3]]] publish_address {127.0.0.1:35667}, bound_addresses {[::1]:44199}, {127.0.0.1:35667}
[2024-06-10T15:53:17,996][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#3]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:53:18,000][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#3]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:53:18,000][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#3]]] Remote clusters initialized successfully.
[2024-06-10T15:53:18,064][INFO ][o.o.c.s.MasterService    ] [node_t0] node-join[{node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true} join existing leader], term: 1, version: 6, delta: added {{node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:18,074][INFO ][o.o.c.s.ClusterApplierService] [node_t1] added {{node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:18,103][INFO ][o.o.c.s.ClusterApplierService] [node_t2] cluster-manager node changed {previous [], current [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true},{node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:18,103][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#3]]] started
[2024-06-10T15:53:18,103][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 6, reason: Publication{term=1, version=6}
[2024-06-10T15:53:18,105][INFO ][o.o.d.PeerFinder         ] [node_t2] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:53:18,114][INFO ][o.o.n.Node               ] [testBulkWritesRejected] version[3.0.0-SNAPSHOT], pid[273801], build[unknown/ead733cf85d7433dd9b075048f43dbae273e5f76/2024-06-10T21:23:34.135290907Z], OS[Linux/5.15.0-1056-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-06-10T15:53:18,114][INFO ][o.o.n.Node               ] [testBulkWritesRejected] JVM home [/home/ubuntu/.gradle/caches/transforms-4/694b5d2aff184189bcea38c919099622/transformed/linux-21.0.3-x64.tar.gz]
[2024-06-10T15:53:18,114][INFO ][o.o.n.Node               ] [testBulkWritesRejected] JVM arguments [-Dgradle.dist.lib=/home/ubuntu/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib, -Dgradle.user.home=/home/ubuntu/.gradle, -Dgradle.worker.jar=/home/ubuntu/.gradle/caches/8.7/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djava.security.manager=allow, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.coverage=true, -Dtests.gradle=true, -Dtests.jvm.argline=-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=3BEC575734C5562E, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:ReservedCodeCacheSize=64m, -esa, --add-modules=jdk.incubator.vector, -XX:HeapDumpPath=/var/jenkins/workspace/gradle-check/search/server/build/heapdump, -javaagent:/var/jenkins/workspace/gradle-check/search/server/build/tmp/.cache/expanded/zip_b7e3e3efc9fa7b01ca2c5374419709e6/jacocoagent.jar=destfile=../../jacoco/internalClusterTest.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -Xms512m, -Xmx3g, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp, -Duser.country, -Duser.language=en, -Duser.variant, -ea]
[2024-06-10T15:53:18,114][WARN ][o.o.n.Node               ] [testBulkWritesRejected] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] no modules loaded
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.index.mapper.MockFieldFilterPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.node.NodeMocksPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.search.MockSearchService$TestPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.p.PluginsService     ] [testBulkWritesRejected] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-06-10T15:53:18,115][INFO ][o.o.e.ExtensionsManager  ] [testBulkWritesRejected] ExtensionsManager initialized
[2024-06-10T15:53:18,123][INFO ][o.o.e.NodeEnvironment    ] [testBulkWritesRejected] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [251.8gb], net total_space [290.5gb], types [ext4]
[2024-06-10T15:53:18,123][INFO ][o.o.e.NodeEnvironment    ] [testBulkWritesRejected] heap size [3gb], compressed ordinary object pointers [true]
[2024-06-10T15:53:18,130][INFO ][o.o.n.Node               ] [testBulkWritesRejected] node name [node_t3], node ID [CfPc-NYuTkudPVFPasMI9w], cluster name [TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster], roles []
[2024-06-10T15:53:18,144][INFO ][o.o.d.DiscoveryModule    ] [testBulkWritesRejected] using discovery type [zen] and seed hosts providers [settings, file]
[2024-06-10T15:53:18,151][WARN ][o.o.g.DanglingIndicesState] [testBulkWritesRejected] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-06-10T15:53:18,164][INFO ][o.o.n.Node               ] [testBulkWritesRejected] initialized
[2024-06-10T15:53:18,170][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#4]]] starting ...
[2024-06-10T15:53:18,174][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#4]]] publish_address {127.0.0.1:33727}, bound_addresses {[::1]:40323}, {127.0.0.1:33727}
[2024-06-10T15:53:18,174][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#4]]] remote backed storage repository with name [test-remote-store-repo] and type [reloadable-fs] created
[2024-06-10T15:53:18,187][INFO ][o.o.n.r.RemoteStoreNodeService] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#4]]] successfully verified [test-remote-store-repo] repository
[2024-06-10T15:53:18,284][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:18,296][INFO ][o.o.p.PluginsService     ] [node_t2] PluginService:onIndexModule index:[test-idx-1/1VhS7U_8Rzyrnnb6YmyJqQ]
[2024-06-10T15:53:18,307][ERROR][o.o.i.s.IndexShard       ] [node_t2] [test-idx-1][0] skip local recovery as no index commit found
[2024-06-10T15:53:18,308][INFO ][o.o.c.s.MasterService    ] [node_t0] node-join[{node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true} join existing leader], term: 1, version: 9, delta: added {{node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:18,317][INFO ][o.o.c.s.ClusterApplierService] [node_t1] added {{node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true}}, term: 1, version: 9, reason: ApplyCommitRequest{term=1, version=9, sourceNode={node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:18,317][INFO ][o.o.c.s.ClusterApplierService] [node_t2] added {{node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true}}, term: 1, version: 9, reason: ApplyCommitRequest{term=1, version=9, sourceNode={node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:18,317][INFO ][o.o.c.s.ClusterApplierService] [node_t3] cluster-manager node changed {previous [], current [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true},{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true},{node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true}}, term: 1, version: 9, reason: ApplyCommitRequest{term=1, version=9, sourceNode={node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}}
[2024-06-10T15:53:18,318][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[697]-CLUSTER_SEED=[-1387441414082369818]-HASH=[24E97034F0A]-cluster[T#4]]] started
[2024-06-10T15:53:18,321][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true}}, term: 1, version: 9, reason: Publication{term=1, version=9}
[2024-06-10T15:53:18,322][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:18,322][INFO ][o.o.d.PeerFinder         ] [node_t3] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true}]
[2024-06-10T15:53:18,374][INFO ][o.o.i.r.RecoverySourceHandler] [node_t1] [test-idx-1][0][recover to node_t2] finalizing recovery took [4.2ms]
[2024-06-10T15:53:18,375][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test-idx-1][0]]]).
[2024-06-10T15:53:18,468][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:18,477][INFO ][o.o.p.PluginsService     ] [node_t0] PluginService:onIndexModule index:[test-idx-1/1VhS7U_8Rzyrnnb6YmyJqQ]
[2024-06-10T15:53:18,479][INFO ][o.o.c.m.MetadataMappingService] [node_t0] [test-idx-1/1VhS7U_8Rzyrnnb6YmyJqQ] create_mapping
[2024-06-10T15:53:20,029][WARN ][o.o.i.SegmentReplicationPressureService] [node_t1] Rejecting write requests for shard, stale shards [100.0%] shards: [SegmentReplicationShardStats{allocationId=VX-yTJhoTyW4WkPmsN4fUg, checkpointsBehindCount=3, bytesBehindCount=45036, currentReplicationLagMillis=1378, currentReplicationTimeMillis=1366, lastCompletedReplicationTimeMillis=0, currentReplicationState=null}]
[2024-06-10T15:53:20,663][INFO ][o.o.a.a.c.r.c.TransportCleanupRepositoryAction] [node_t0] Running cleanup operations on repository [test-remote-store-repo][-1]
[2024-06-10T15:53:20,684][INFO ][o.o.a.a.c.r.c.TransportCleanupRepositoryAction] [node_t0] Done with repository cleanup on [test-remote-store-repo][-1] with result [{"deleted_bytes":0,"deleted_blobs":0}]
[2024-06-10T15:53:20,686][INFO ][o.o.r.SegmentReplicationWithRemoteStorePressureIT] [testBulkWritesRejected] after test
[2024-06-10T15:53:20,686][INFO ][o.o.t.OpenSearchTestClusterRule] [testBulkWritesRejected] [SegmentReplicationWithRemoteStorePressureIT#testBulkWritesRejected]: cleaning up after test
[2024-06-10T15:53:20,693][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:20,704][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_t0] [test-idx-1/1VhS7U_8Rzyrnnb6YmyJqQ] deleting index
[2024-06-10T15:53:20,704][WARN ][o.o.c.r.a.AllocationService] [node_t0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-06-10T15:53:20,799][INFO ][o.o.i.t.t.TranslogTransferManager] [node_t1] [test-idx-1][0] Deleted all remote translog data at path=[p10101011110111][1VhS7U_8Rzyrnnb6YmyJqQ][0][translog][metadata]
[2024-06-10T15:53:20,804][INFO ][o.o.i.t.t.TranslogTransferManager] [node_t1] [test-idx-1][0] Deleted all remote translog data at path=[410101101011101][1VhS7U_8Rzyrnnb6YmyJqQ][0][translog][data]
[2024-06-10T15:53:20,819][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopping ...
[2024-06-10T15:53:20,820][INFO ][o.o.c.c.FollowersChecker ] [testBulkWritesRejected] FollowerChecker{discoveryNode={node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-06-10T15:53:20,820][INFO ][o.o.c.c.FollowersChecker ] [testBulkWritesRejected] FollowerChecker{discoveryNode={node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-06-10T15:53:20,820][INFO ][o.o.c.c.FollowersChecker ] [testBulkWritesRejected] FollowerChecker{discoveryNode={node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-06-10T15:53:20,820][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t1}{34jq6quTRl-trBvBScha8g}{N6Bom_J8QFuF51Skd8pF6g}{127.0.0.1}{127.0.0.1:42745}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-06-10T15:53:20,821][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t2}{AbPwwa0ISaWjQUdnc2vOqg}{G91-DWJfShmDLOSvCKdzog}{127.0.0.1}{127.0.0.1:35667}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-06-10T15:53:20,821][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t3}{CfPc-NYuTkudPVFPasMI9w}{ORdvpkZzTxuH8p81KGexEA}{127.0.0.1}{127.0.0.1:33727}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-06-10T15:53:20,821][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopped
[2024-06-10T15:53:20,821][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closing ...
[2024-06-10T15:53:20,821][INFO ][o.o.c.c.Coordinator      ] [node_t3] cluster-manager node [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:45807][disconnected] disconnected
[2024-06-10T15:53:20,821][INFO ][o.o.c.c.Coordinator      ] [node_t2] cluster-manager node [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:45807][disconnected] disconnected
[2024-06-10T15:53:20,821][INFO ][o.o.c.c.Coordinator      ] [node_t1] cluster-manager node [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:45807][disconnected] disconnected
[2024-06-10T15:53:20,821][INFO ][o.o.c.s.ClusterApplierService] [node_t2] cluster-manager node changed {previous [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 15, reason: becoming candidate: onLeaderFailure
[2024-06-10T15:53:20,822][INFO ][o.o.c.s.ClusterApplierService] [node_t1] cluster-manager node changed {previous [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 15, reason: becoming candidate: onLeaderFailure
[2024-06-10T15:53:20,822][INFO ][o.o.c.s.ClusterApplierService] [node_t3] cluster-manager node changed {previous [{node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 15, reason: becoming candidate: onLeaderFailure
[2024-06-10T15:53:20,822][WARN ][o.o.c.NodeConnectionsService] [node_t3] failed to connect to {node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:45807] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1098) ~[main/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	... 9 more
[2024-06-10T15:53:20,822][WARN ][o.o.c.NodeConnectionsService] [node_t1] failed to connect to {node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:45807] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1098) ~[main/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	... 9 more
[2024-06-10T15:53:20,822][WARN ][o.o.c.NodeConnectionsService] [node_t2] failed to connect to {node_t0}{WBopOJJxSy-VUkeNNw9K0w}{UiVK0bmyQ8yRWFHNhJhkag}{127.0.0.1}{127.0.0.1:45807}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:45807] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1098) ~[main/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-3.0.0-SNAPSHOT.jar:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[framework-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[opensearch-nio-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	... 9 more
[2024-06-10T15:53:20,826][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closed
[2024-06-10T15:53:20,827][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopping ...
[2024-06-10T15:53:20,828][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopped
[2024-06-10T15:53:20,828][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closing ...
[2024-06-10T15:53:20,830][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closed
[2024-06-10T15:53:20,832][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopping ...
[2024-06-10T15:53:20,833][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopped
[2024-06-10T15:53:20,833][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closing ...
[2024-06-10T15:53:20,837][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closed
[2024-06-10T15:53:20,838][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopping ...
[2024-06-10T15:53:20,839][INFO ][o.o.n.Node               ] [testBulkWritesRejected] stopped
[2024-06-10T15:53:20,839][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closing ...
[2024-06-10T15:53:20,840][INFO ][o.o.n.Node               ] [testBulkWritesRejected] closed
[2024-06-10T15:53:20,840][INFO ][o.o.t.OpenSearchTestClusterRule] [testBulkWritesRejected] [SegmentReplicationWithRemoteStorePressureIT#testBulkWritesRejected]: cleaned up after test

@reta reta mentioned this issue Jun 10, 2024
Merged
9 tasks
@reta
Copy link
Collaborator

reta commented Jun 19, 2024

Closing in favour of #14330

@reta reta closed this as completed Jun 19, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Segment Replication Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mch2 mch2

Labels
bug Something isn't working distributed framework flaky-test Random test failure that succeeds on second run :test Adding or fixing a test >test-failure Test failure from CI, local build, etc.
Projects
Segment Replication
Status: Done
Milestone
No milestone
6 participants
@reta @Xtansia @mch2 @andrross @tlfeng @dreamer-89