Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LegacyIndexStatsIT fails with ReleasableBytesReference #77634

Closed
nik9000 opened this issue Sep 13, 2021 · 1 comment · Fixed by #77641
Closed

LegacyIndexStatsIT fails with ReleasableBytesReference #77634

nik9000 opened this issue Sep 13, 2021 · 1 comment · Fixed by #77641
Assignees
@DaveCTurner
Labels
:Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@nik9000
Copy link
Member

nik9000 commented Sep 13, 2021

Build scan: None taken

Repro line:

./gradlew ':server:internalClusterTest' \
  --tests "org.elasticsearch.indices.stats.LegacyIndexStatsIT" \
  -Dtests.seed=EA84AA48D964012D -Dtests.locale=en-US \
  -Dtests.timezone=America/New_York -Druntime.java=16

Reproduces locally?: No

Applicable branches: 7.x only? It triggered while I was testing #77633 locally. Didn't reproduce.

Failure history:
Didn't find anything.

Failure excerpt:

> Task :server:internalClusterTest
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.

org.elasticsearch.indices.stats.LegacyIndexStatsIT > classMethod FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3555, name=elasticsearch[node_s2][generic][T#3], state=RUNNABLE, group=TGRP-LegacyIndexStatsIT]

        Caused by:
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([EA84AA48D964012D]:0)
            at org.elasticsearch.common.bytes.ReleasableBytesReference.get(ReleasableBytesReference.java:96)
            at org.elasticsearch.common.compress.DeflateCompressor.isCompressed(DeflateCompressor.java:50)
            at org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:29)
            at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:141)
            at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:102)
            at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61)
            at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:881)
            at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
            at java.base/java.lang.Thread.run(Thread.java:831)


Suite: Test class org.elasticsearch.indices.stats.LegacyIndexStatsIT
  1> [2021-09-13T10:33:50,300][INFO ][o.e.i.s.LegacyIndexStatsIT] [testFieldDataFieldsParam] before test
  1> [2021-09-13T10:33:50,301][INFO ][o.e.i.s.LegacyIndexStatsIT] [testFieldDataFieldsParam] [LegacyIndexStatsIT#testFieldDataFieldsParam]: setting up test
  1> [2021-09-13T10:33:50,302][INFO ][o.e.t.InternalTestCluster] [testFieldDataFieldsParam] Setup InternalTestCluster [SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster] with seed [85DC3084EDCDE682] using [3] dedicated masters, [3] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed])
  1> [2021-09-13T10:33:50,304][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] version[7.16.0], pid[1702091], build[unknown/unknown/unknown/unknown], OS[Linux/5.13.9-arch1-1/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
  1> [2021-09-13T10:33:50,304][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM home [/home/nik9000/.gradle/caches/transforms-3/49594bfb29fc78559d5278c1ea8edf22/transformed/linux-16.0.2-x64.tar.gz]
  1> [2021-09-13T10:33:50,305][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,305][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM arguments [-Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/home/nik9000/.gradle/wrapper/dists/gradle-7.2-all/260hg96vuh6ex27h9vo47iv4d/gradle-7.2/lib, -Dgradle.user.home=/home/nik9000/.gradle, -Dgradle.worker.jar=/home/nik9000/.gradle/caches/7.2/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.internal.worker.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=EA84AA48D964012D, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:+UseG1GC, --illegal-access=deny, --add-opens=java.base/java.security.cert=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/javax.net.ssl=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.time=ALL-UNNAMED, --add-opens=java.management/java.lang.management=ALL-UNNAMED, -XX:HeapDumpPath=/home/nik9000/Code/Elastic/elasticsearch/server/build/heapdump, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2021-09-13T10:33:50,305][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] setting [path.shared_data] is deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,305][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring multiple [path.data] paths is deprecated. Use RAID or other system level features for utilizing multiple disks. This feature will be removed in 8.0.
  1> [2021-09-13T10:33:50,306][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring [path.data] with a list is deprecated. Instead specify as a string value.
  1> [2021-09-13T10:33:50,306][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:50,307][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2021-09-13T10:33:50,307][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2021-09-13T10:33:50,307][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2021-09-13T10:33:50,307][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2021-09-13T10:33:50,307][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2021-09-13T10:33:50,307][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:50,312][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] using [3] data paths, mounts [[/ (/dev/sda3)]], net usable_space [147.2gb], net total_space [911.7gb], types [ext4]
  1> [2021-09-13T10:33:50,312][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] heap size [512mb], compressed ordinary object pointers [true]
  1> [2021-09-13T10:33:50,317][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] node name [node_s0], node ID [uNjo_NBhRw-1oIndJcnXBw], cluster name [SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster], roles [master, remote_cluster_client, ingest]
  1> [2021-09-13T10:33:50,376][INFO ][o.e.d.DiscoveryModule    ] [testFieldDataFieldsParam] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2021-09-13T10:33:50,384][INFO ][o.e.g.DanglingIndicesState] [testFieldDataFieldsParam] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2021-09-13T10:33:50,403][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] initialized
  1> [2021-09-13T10:33:50,404][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] version[7.16.0], pid[1702091], build[unknown/unknown/unknown/unknown], OS[Linux/5.13.9-arch1-1/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
  1> [2021-09-13T10:33:50,405][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM home [/home/nik9000/.gradle/caches/transforms-3/49594bfb29fc78559d5278c1ea8edf22/transformed/linux-16.0.2-x64.tar.gz]
  1> [2021-09-13T10:33:50,405][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,406][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM arguments [-Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/home/nik9000/.gradle/wrapper/dists/gradle-7.2-all/260hg96vuh6ex27h9vo47iv4d/gradle-7.2/lib, -Dgradle.user.home=/home/nik9000/.gradle, -Dgradle.worker.jar=/home/nik9000/.gradle/caches/7.2/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.internal.worker.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=EA84AA48D964012D, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:+UseG1GC, --illegal-access=deny, --add-opens=java.base/java.security.cert=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/javax.net.ssl=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.time=ALL-UNNAMED, --add-opens=java.management/java.lang.management=ALL-UNNAMED, -XX:HeapDumpPath=/home/nik9000/Code/Elastic/elasticsearch/server/build/heapdump, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2021-09-13T10:33:50,416][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] setting [path.shared_data] is deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,417][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring multiple [path.data] paths is deprecated. Use RAID or other system level features for utilizing multiple disks. This feature will be removed in 8.0.
  1> [2021-09-13T10:33:50,417][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring [path.data] with a list is deprecated. Instead specify as a string value.
  1> [2021-09-13T10:33:50,418][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:50,418][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2021-09-13T10:33:50,418][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2021-09-13T10:33:50,418][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2021-09-13T10:33:50,418][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2021-09-13T10:33:50,418][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2021-09-13T10:33:50,419][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:50,424][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] using [3] data paths, mounts [[/ (/dev/sda3)]], net usable_space [147.2gb], net total_space [911.7gb], types [ext4]
  1> [2021-09-13T10:33:50,425][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] heap size [512mb], compressed ordinary object pointers [true]
  1> [2021-09-13T10:33:50,432][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] node name [node_s1], node ID [egJGYd-TRd-7q_hTlQvZ6w], cluster name [SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster], roles [master, remote_cluster_client, ingest]
  1> [2021-09-13T10:33:50,527][INFO ][o.e.d.DiscoveryModule    ] [testFieldDataFieldsParam] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2021-09-13T10:33:50,538][INFO ][o.e.g.DanglingIndicesState] [testFieldDataFieldsParam] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2021-09-13T10:33:50,554][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] initialized
  1> [2021-09-13T10:33:50,561][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] version[7.16.0], pid[1702091], build[unknown/unknown/unknown/unknown], OS[Linux/5.13.9-arch1-1/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
  1> [2021-09-13T10:33:50,561][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM home [/home/nik9000/.gradle/caches/transforms-3/49594bfb29fc78559d5278c1ea8edf22/transformed/linux-16.0.2-x64.tar.gz]
  1> [2021-09-13T10:33:50,561][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,562][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM arguments [-Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/home/nik9000/.gradle/wrapper/dists/gradle-7.2-all/260hg96vuh6ex27h9vo47iv4d/gradle-7.2/lib, -Dgradle.user.home=/home/nik9000/.gradle, -Dgradle.worker.jar=/home/nik9000/.gradle/caches/7.2/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.internal.worker.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=EA84AA48D964012D, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:+UseG1GC, --illegal-access=deny, --add-opens=java.base/java.security.cert=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/javax.net.ssl=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.time=ALL-UNNAMED, --add-opens=java.management/java.lang.management=ALL-UNNAMED, -XX:HeapDumpPath=/home/nik9000/Code/Elastic/elasticsearch/server/build/heapdump, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2021-09-13T10:33:50,562][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] setting [path.shared_data] is deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,562][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring multiple [path.data] paths is deprecated. Use RAID or other system level features for utilizing multiple disks. This feature will be removed in 8.0.
  1> [2021-09-13T10:33:50,562][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring [path.data] with a list is deprecated. Instead specify as a string value.
  1> [2021-09-13T10:33:50,563][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:50,563][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2021-09-13T10:33:50,563][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2021-09-13T10:33:50,563][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2021-09-13T10:33:50,564][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2021-09-13T10:33:50,564][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2021-09-13T10:33:50,564][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:50,595][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] using [3] data paths, mounts [[/ (/dev/sda3)]], net usable_space [147.2gb], net total_space [911.7gb], types [ext4]
  1> [2021-09-13T10:33:50,595][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] heap size [512mb], compressed ordinary object pointers [true]
  1> [2021-09-13T10:33:50,599][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] node name [node_s2], node ID [8sVOvGH-TXGTrJxtA274Wg], cluster name [SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster], roles [master, remote_cluster_client, ingest]
  1> [2021-09-13T10:33:50,642][INFO ][o.e.d.DiscoveryModule    ] [testFieldDataFieldsParam] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2021-09-13T10:33:50,649][INFO ][o.e.g.DanglingIndicesState] [testFieldDataFieldsParam] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2021-09-13T10:33:50,681][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] initialized
  1> [2021-09-13T10:33:50,684][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] version[7.16.0], pid[1702091], build[unknown/unknown/unknown/unknown], OS[Linux/5.13.9-arch1-1/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
  1> [2021-09-13T10:33:50,685][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM home [/home/nik9000/.gradle/caches/transforms-3/49594bfb29fc78559d5278c1ea8edf22/transformed/linux-16.0.2-x64.tar.gz]
  1> [2021-09-13T10:33:50,685][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,685][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM arguments [-Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/home/nik9000/.gradle/wrapper/dists/gradle-7.2-all/260hg96vuh6ex27h9vo47iv4d/gradle-7.2/lib, -Dgradle.user.home=/home/nik9000/.gradle, -Dgradle.worker.jar=/home/nik9000/.gradle/caches/7.2/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.internal.worker.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=EA84AA48D964012D, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:+UseG1GC, --illegal-access=deny, --add-opens=java.base/java.security.cert=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/javax.net.ssl=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.time=ALL-UNNAMED, --add-opens=java.management/java.lang.management=ALL-UNNAMED, -XX:HeapDumpPath=/home/nik9000/Code/Elastic/elasticsearch/server/build/heapdump, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2021-09-13T10:33:50,685][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] setting [path.shared_data] is deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,686][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring multiple [path.data] paths is deprecated. Use RAID or other system level features for utilizing multiple disks. This feature will be removed in 8.0.
  1> [2021-09-13T10:33:50,686][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring [path.data] with a list is deprecated. Instead specify as a string value.
  1> [2021-09-13T10:33:50,686][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:50,686][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2021-09-13T10:33:50,687][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2021-09-13T10:33:50,687][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2021-09-13T10:33:50,687][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2021-09-13T10:33:50,687][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2021-09-13T10:33:50,687][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:50,693][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] using [3] data paths, mounts [[/ (/dev/sda3)]], net usable_space [147.2gb], net total_space [911.7gb], types [ext4]
  1> [2021-09-13T10:33:50,693][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] heap size [512mb], compressed ordinary object pointers [true]
  1> [2021-09-13T10:33:50,696][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] node name [node_s3], node ID [Zb7KFZpfQIyyz4qkfybtOA], cluster name [SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster], roles [data_frozen, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
  1> [2021-09-13T10:33:50,731][INFO ][o.e.d.DiscoveryModule    ] [testFieldDataFieldsParam] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2021-09-13T10:33:50,738][INFO ][o.e.g.DanglingIndicesState] [testFieldDataFieldsParam] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2021-09-13T10:33:50,757][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] initialized
  1> [2021-09-13T10:33:50,767][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] version[7.16.0], pid[1702091], build[unknown/unknown/unknown/unknown], OS[Linux/5.13.9-arch1-1/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
  1> [2021-09-13T10:33:50,769][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM home [/home/nik9000/.gradle/caches/transforms-3/49594bfb29fc78559d5278c1ea8edf22/transformed/linux-16.0.2-x64.tar.gz]
  1> [2021-09-13T10:33:50,769][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,769][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM arguments [-Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/home/nik9000/.gradle/wrapper/dists/gradle-7.2-all/260hg96vuh6ex27h9vo47iv4d/gradle-7.2/lib, -Dgradle.user.home=/home/nik9000/.gradle, -Dgradle.worker.jar=/home/nik9000/.gradle/caches/7.2/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.internal.worker.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=EA84AA48D964012D, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:+UseG1GC, --illegal-access=deny, --add-opens=java.base/java.security.cert=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/javax.net.ssl=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.time=ALL-UNNAMED, --add-opens=java.management/java.lang.management=ALL-UNNAMED, -XX:HeapDumpPath=/home/nik9000/Code/Elastic/elasticsearch/server/build/heapdump, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2021-09-13T10:33:50,770][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] setting [path.shared_data] is deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,770][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring multiple [path.data] paths is deprecated. Use RAID or other system level features for utilizing multiple disks. This feature will be removed in 8.0.
  1> [2021-09-13T10:33:50,770][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring [path.data] with a list is deprecated. Instead specify as a string value.
  1> [2021-09-13T10:33:50,771][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:50,771][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2021-09-13T10:33:50,772][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2021-09-13T10:33:50,772][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2021-09-13T10:33:50,772][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2021-09-13T10:33:50,772][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2021-09-13T10:33:50,772][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:50,829][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] using [3] data paths, mounts [[/ (/dev/sda3)]], net usable_space [147.2gb], net total_space [911.7gb], types [ext4]
  1> [2021-09-13T10:33:50,832][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] heap size [512mb], compressed ordinary object pointers [true]
  1> [2021-09-13T10:33:50,836][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] node name [node_s4], node ID [VpuNjarURD6yaW83RF9yJA], cluster name [SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster], roles [data_frozen, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
  1> [2021-09-13T10:33:50,874][INFO ][o.e.d.DiscoveryModule    ] [testFieldDataFieldsParam] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2021-09-13T10:33:50,881][INFO ][o.e.g.DanglingIndicesState] [testFieldDataFieldsParam] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2021-09-13T10:33:50,893][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] initialized
  1> [2021-09-13T10:33:50,895][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] version[7.16.0], pid[1702091], build[unknown/unknown/unknown/unknown], OS[Linux/5.13.9-arch1-1/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
  1> [2021-09-13T10:33:50,895][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM home [/home/nik9000/.gradle/caches/transforms-3/49594bfb29fc78559d5278c1ea8edf22/transformed/linux-16.0.2-x64.tar.gz]
  1> [2021-09-13T10:33:50,895][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,895][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] JVM arguments [-Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/home/nik9000/.gradle/wrapper/dists/gradle-7.2-all/260hg96vuh6ex27h9vo47iv4d/gradle-7.2/lib, -Dgradle.user.home=/home/nik9000/.gradle, -Dgradle.worker.jar=/home/nik9000/.gradle/caches/7.2/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,COMPAT, -Djna.nosys=true, -Dorg.gradle.internal.worker.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/tmp/internalClusterTest/work, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=EA84AA48D964012D, -Dtests.task=:server:internalClusterTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:+UseG1GC, --illegal-access=deny, --add-opens=java.base/java.security.cert=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/javax.net.ssl=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.time=ALL-UNNAMED, --add-opens=java.management/java.lang.management=ALL-UNNAMED, -XX:HeapDumpPath=/home/nik9000/Code/Elastic/elasticsearch/server/build/heapdump, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=/home/nik9000/Code/Elastic/elasticsearch/server/build/testrun/internalClusterTest/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2021-09-13T10:33:50,896][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] setting [path.shared_data] is deprecated and will be removed in a future release
  1> [2021-09-13T10:33:50,896][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring multiple [path.data] paths is deprecated. Use RAID or other system level features for utilizing multiple disks. This feature will be removed in 8.0.
  1> [2021-09-13T10:33:50,896][DEPRECATION][o.e.d.n.Node             ] [testFieldDataFieldsParam] Configuring [path.data] with a list is deprecated. Instead specify as a string value.
  1> [2021-09-13T10:33:50,897][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:50,897][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.script.MockScriptService$TestPlugin]
  1> [2021-09-13T10:33:50,897][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2021-09-13T10:33:50,897][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2021-09-13T10:33:50,897][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin]
  1> [2021-09-13T10:33:50,897][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.test.TestGeoShapeFieldMapperPlugin]
  1> [2021-09-13T10:33:50,898][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:50,903][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] using [3] data paths, mounts [[/ (/dev/sda3)]], net usable_space [147.2gb], net total_space [911.7gb], types [ext4]
  1> [2021-09-13T10:33:50,907][INFO ][o.e.e.NodeEnvironment    ] [testFieldDataFieldsParam] heap size [512mb], compressed ordinary object pointers [true]
  1> [2021-09-13T10:33:50,911][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] node name [node_s5], node ID [qbOVPpPrTiyBHbfCveJ0qg], cluster name [SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster], roles [data_frozen, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
  1> [2021-09-13T10:33:50,958][INFO ][o.e.d.DiscoveryModule    ] [testFieldDataFieldsParam] using discovery type [zen] and seed hosts providers [settings, file]
  1> [2021-09-13T10:33:50,965][INFO ][o.e.g.DanglingIndicesState] [testFieldDataFieldsParam] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2021-09-13T10:33:50,979][INFO ][o.e.n.Node               ] [testFieldDataFieldsParam] initialized
  1> [2021-09-13T10:33:51,000][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#1]]] starting ...
  1> [2021-09-13T10:33:51,014][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#2]]] starting ...
  1> [2021-09-13T10:33:51,031][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#1]]] publish_address {127.0.0.1:42495}, bound_addresses {[::1]:41293}, {127.0.0.1:42495}
  1> [2021-09-13T10:33:51,035][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#4]]] starting ...
  1> [2021-09-13T10:33:51,038][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#3]]] starting ...
  1> [2021-09-13T10:33:51,080][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#2]]] publish_address {127.0.0.1:34645}, bound_addresses {[::1]:46725}, {127.0.0.1:34645}
  1> [2021-09-13T10:33:51,093][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#5]]] starting ...
  1> [2021-09-13T10:33:51,100][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#6]]] starting ...
  1> [2021-09-13T10:33:51,108][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#3]]] publish_address {127.0.0.1:42361}, bound_addresses {[::1]:43791}, {127.0.0.1:42361}
  1> [2021-09-13T10:33:51,108][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#4]]] publish_address {127.0.0.1:33839}, bound_addresses {[::1]:40283}, {127.0.0.1:33839}
  1> [2021-09-13T10:33:51,150][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#5]]] publish_address {127.0.0.1:38593}, bound_addresses {[::1]:42887}, {127.0.0.1:38593}
  1> [2021-09-13T10:33:51,274][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#6]]] publish_address {127.0.0.1:36183}, bound_addresses {[::1]:41295}, {127.0.0.1:36183}
  1> [2021-09-13T10:33:51,397][INFO ][o.e.c.c.Coordinator      ] [node_s2] setting initial configuration to VotingConfiguration{8sVOvGH-TXGTrJxtA274Wg,{bootstrap-placeholder}-node_s1,uNjo_NBhRw-1oIndJcnXBw}
  1> [2021-09-13T10:33:51,590][INFO ][o.e.c.s.MasterService    ] [node_s2] elected-as-master ([2] nodes joined)[{node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr} elect leader, {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], node-join[{node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr} join existing leader], term: 1, version: 1, delta: master node changed {previous [], current [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}]}, added {{node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr}, {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr}}
  1> [2021-09-13T10:33:51,635][INFO ][o.e.c.c.CoordinationState] [node_s2] cluster UUID set to [YdDqBCRrQQ-ootUefYoNqw]
  1> [2021-09-13T10:33:51,640][INFO ][o.e.c.c.CoordinationState] [node_s0] cluster UUID set to [YdDqBCRrQQ-ootUefYoNqw]
  1> [2021-09-13T10:33:51,640][INFO ][o.e.c.c.CoordinationState] [node_s1] cluster UUID set to [YdDqBCRrQQ-ootUefYoNqw]
  1> [2021-09-13T10:33:51,656][INFO ][o.e.c.s.ClusterApplierService] [node_s0] master node changed {previous [], current [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}]}, added {{node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr}, {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:51,656][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#1]]] started
  1> [2021-09-13T10:33:51,675][INFO ][o.e.c.s.ClusterApplierService] [node_s1] master node changed {previous [], current [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}]}, added {{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}, {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:51,675][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#2]]] started
  1> [2021-09-13T10:33:51,677][INFO ][o.e.c.s.ClusterApplierService] [node_s2] master node changed {previous [], current [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}]}, added {{node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr}, {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr}}, term: 1, version: 1, reason: Publication{term=1, version=1}
  1> [2021-09-13T10:33:51,678][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#3]]] started
  1> [2021-09-13T10:33:51,679][INFO ][o.e.c.s.MasterService    ] [node_s2] node-join[{node_s4}{VpuNjarURD6yaW83RF9yJA}{9ixlAlpbQB6Km2YPHuwD9A}{127.0.0.1}{127.0.0.1:38593}{cdfhirsw} join existing leader, {node_s5}{qbOVPpPrTiyBHbfCveJ0qg}{tgliZ_O5QU2_IiLiuIFO0A}{127.0.0.1}{127.0.0.1:36183}{cdfhirsw} join existing leader], term: 1, version: 2, delta: added {{node_s5}{qbOVPpPrTiyBHbfCveJ0qg}{tgliZ_O5QU2_IiLiuIFO0A}{127.0.0.1}{127.0.0.1:36183}{cdfhirsw}, {node_s4}{VpuNjarURD6yaW83RF9yJA}{9ixlAlpbQB6Km2YPHuwD9A}{127.0.0.1}{127.0.0.1:38593}{cdfhirsw}}
  1> [2021-09-13T10:33:51,691][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s5}{qbOVPpPrTiyBHbfCveJ0qg}{tgliZ_O5QU2_IiLiuIFO0A}{127.0.0.1}{127.0.0.1:36183}{cdfhirsw}, {node_s4}{VpuNjarURD6yaW83RF9yJA}{9ixlAlpbQB6Km2YPHuwD9A}{127.0.0.1}{127.0.0.1:38593}{cdfhirsw}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:51,691][INFO ][o.e.c.s.ClusterApplierService] [node_s4] master node changed {previous [], current [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}]}, added {{node_s5}{qbOVPpPrTiyBHbfCveJ0qg}{tgliZ_O5QU2_IiLiuIFO0A}{127.0.0.1}{127.0.0.1:36183}{cdfhirsw}, {node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr}, {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}, {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:51,691][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s5}{qbOVPpPrTiyBHbfCveJ0qg}{tgliZ_O5QU2_IiLiuIFO0A}{127.0.0.1}{127.0.0.1:36183}{cdfhirsw}, {node_s4}{VpuNjarURD6yaW83RF9yJA}{9ixlAlpbQB6Km2YPHuwD9A}{127.0.0.1}{127.0.0.1:38593}{cdfhirsw}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:51,691][INFO ][o.e.c.s.ClusterApplierService] [node_s5] master node changed {previous [], current [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}]}, added {{node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr}, {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}, {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr}, {node_s4}{VpuNjarURD6yaW83RF9yJA}{9ixlAlpbQB6Km2YPHuwD9A}{127.0.0.1}{127.0.0.1:38593}{cdfhirsw}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:51,694][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#5]]] started
  1> [2021-09-13T10:33:51,694][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#6]]] started
  1> [2021-09-13T10:33:51,694][INFO ][o.e.c.s.ClusterApplierService] [node_s2] added {{node_s5}{qbOVPpPrTiyBHbfCveJ0qg}{tgliZ_O5QU2_IiLiuIFO0A}{127.0.0.1}{127.0.0.1:36183}{cdfhirsw}, {node_s4}{VpuNjarURD6yaW83RF9yJA}{9ixlAlpbQB6Km2YPHuwD9A}{127.0.0.1}{127.0.0.1:38593}{cdfhirsw}}, term: 1, version: 2, reason: Publication{term=1, version=2}
  1> [2021-09-13T10:33:51,771][INFO ][o.e.g.GatewayService     ] [node_s2] recovered [0] indices into cluster_state
  1> [2021-09-13T10:33:52,182][INFO ][o.e.c.s.MasterService    ] [node_s2] node-join[{node_s3}{Zb7KFZpfQIyyz4qkfybtOA}{SwwjK1SQRnitSvtpLNMu0Q}{127.0.0.1}{127.0.0.1:33839}{cdfhirsw} join existing leader], term: 1, version: 5, delta: added {{node_s3}{Zb7KFZpfQIyyz4qkfybtOA}{SwwjK1SQRnitSvtpLNMu0Q}{127.0.0.1}{127.0.0.1:33839}{cdfhirsw}}
  1> [2021-09-13T10:33:52,187][INFO ][o.e.c.s.ClusterApplierService] [node_s4] added {{node_s3}{Zb7KFZpfQIyyz4qkfybtOA}{SwwjK1SQRnitSvtpLNMu0Q}{127.0.0.1}{127.0.0.1:33839}{cdfhirsw}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:52,187][INFO ][o.e.c.s.ClusterApplierService] [node_s5] added {{node_s3}{Zb7KFZpfQIyyz4qkfybtOA}{SwwjK1SQRnitSvtpLNMu0Q}{127.0.0.1}{127.0.0.1:33839}{cdfhirsw}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:52,187][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s3}{Zb7KFZpfQIyyz4qkfybtOA}{SwwjK1SQRnitSvtpLNMu0Q}{127.0.0.1}{127.0.0.1:33839}{cdfhirsw}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:52,190][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s3}{Zb7KFZpfQIyyz4qkfybtOA}{SwwjK1SQRnitSvtpLNMu0Q}{127.0.0.1}{127.0.0.1:33839}{cdfhirsw}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:52,192][INFO ][o.e.c.s.ClusterApplierService] [node_s3] master node changed {previous [], current [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}]}, added {{node_s5}{qbOVPpPrTiyBHbfCveJ0qg}{tgliZ_O5QU2_IiLiuIFO0A}{127.0.0.1}{127.0.0.1:36183}{cdfhirsw}, {node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr}, {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}, {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr}, {node_s4}{VpuNjarURD6yaW83RF9yJA}{9ixlAlpbQB6Km2YPHuwD9A}{127.0.0.1}{127.0.0.1:38593}{cdfhirsw}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}}
  1> [2021-09-13T10:33:52,197][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[11]-CLUSTER_SEED=[-8801106224305281406]-HASH=[8400F4A91E31D]-cluster[T#4]]] started
  1> [2021-09-13T10:33:52,197][INFO ][o.e.c.s.ClusterApplierService] [node_s2] added {{node_s3}{Zb7KFZpfQIyyz4qkfybtOA}{SwwjK1SQRnitSvtpLNMu0Q}{127.0.0.1}{127.0.0.1:33839}{cdfhirsw}}, term: 1, version: 5, reason: Publication{term=1, version=5}
  1> [2021-09-13T10:33:52,200][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:52,200][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:52,231][DEPRECATION][o.e.d.c.s.Settings       ] [node_s2] [index.force_memory_term_dictionary] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2021-09-13T10:33:52,231][DEPRECATION][o.e.d.c.s.Settings       ] [node_s2] [index.force_memory_term_dictionary] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2021-09-13T10:33:52,232][DEPRECATION][o.e.d.c.s.Settings       ] [node_s2] [index.force_memory_term_dictionary] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2021-09-13T10:33:52,233][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s2] adding template [random_index_template] for index patterns [*]
  1> [2021-09-13T10:33:52,255][INFO ][o.e.i.s.LegacyIndexStatsIT] [testFieldDataFieldsParam] [LegacyIndexStatsIT#testFieldDataFieldsParam]: all set up test
  1> [2021-09-13T10:33:52,257][DEPRECATION][o.e.d.c.s.Settings       ] [node_s2] [index.force_memory_term_dictionary] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2021-09-13T10:33:52,259][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s2] [test1] creating index, cause [api], templates [random_index_template], shards [6]/[0]
  1> [2021-09-13T10:33:52,289][DEPRECATION][o.e.d.c.s.Settings       ] [node_s4] [index.force_memory_term_dictionary] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2021-09-13T10:33:52,289][DEPRECATION][o.e.d.c.s.Settings       ] [node_s5] [index.force_memory_term_dictionary] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2021-09-13T10:33:52,289][DEPRECATION][o.e.d.c.s.Settings       ] [node_s3] [index.force_memory_term_dictionary] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2021-09-13T10:33:52,319][WARN ][o.e.i.s.IndexShard       ] [node_s3] [test1][2] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
  1> [2021-09-13T10:33:52,321][WARN ][o.e.i.s.IndexShard       ] [node_s5] [test1][1] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
  1> [2021-09-13T10:33:52,322][WARN ][o.e.i.s.IndexShard       ] [node_s4] [test1][3] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
  1> [2021-09-13T10:33:52,322][WARN ][o.e.i.s.IndexShard       ] [node_s3] [test1][5] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
  1> [2021-09-13T10:33:52,324][WARN ][o.e.i.s.IndexShard       ] [node_s4] [test1][0] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
  1> [2021-09-13T10:33:52,327][WARN ][o.e.i.s.IndexShard       ] [node_s5] [test1][4] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=true]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
  1> [2021-09-13T10:33:52,363][INFO ][o.e.c.r.a.AllocationService] [node_s2] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test1][3], [test1][0], [test1][4], [test1][5]]]).
  1> [2021-09-13T10:33:52,394][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:52,394][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:52,445][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:52,445][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:52,575][INFO ][o.e.i.s.LegacyIndexStatsIT] [testFieldDataFieldsParam] [LegacyIndexStatsIT#testFieldDataFieldsParam]: cleaning up after test
  1> [2021-09-13T10:33:52,613][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] no modules loaded
  1> [2021-09-13T10:33:52,613][INFO ][o.e.p.PluginsService     ] [testFieldDataFieldsParam] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin]
  1> [2021-09-13T10:33:52,693][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s2] [test1/LPysSNQSQRW5yX_dLBZdLw] deleting index
  1> [2021-09-13T10:33:52,764][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s2] removing template [random_index_template]
  1> [2021-09-13T10:33:52,832][INFO ][o.e.i.s.LegacyIndexStatsIT] [testFieldDataFieldsParam] [LegacyIndexStatsIT#testFieldDataFieldsParam]: cleaned up after test
  1> [2021-09-13T10:33:52,832][INFO ][o.e.i.s.LegacyIndexStatsIT] [testFieldDataFieldsParam] after test
  1> [2021-09-13T10:33:52,835][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2021-09-13T10:33:52,843][INFO ][o.e.c.c.Coordinator      ] [node_s0] master node [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}] failed, restarting discovery
  1> org.elasticsearch.transport.NodeDisconnectedException: [node_s2][127.0.0.1:42361][disconnected] disconnected
  1> [2021-09-13T10:33:52,845][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2021-09-13T10:33:52,845][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2021-09-13T10:33:52,854][INFO ][o.e.c.s.MasterService    ] [node_s2] node-left[{node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr} reason: disconnected], term: 1, version: 12, delta: removed {{node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr}}
  1> [2021-09-13T10:33:52,854][INFO ][o.e.n.Node               ] [suite] closed
  1> [2021-09-13T10:33:52,855][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2021-09-13T10:33:52,859][INFO ][o.e.c.c.Coordinator      ] [node_s1] master node [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}] failed, restarting discovery
  1> org.elasticsearch.transport.NodeDisconnectedException: [node_s2][127.0.0.1:42361][disconnected] disconnected
  1> [2021-09-13T10:33:52,861][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2021-09-13T10:33:52,861][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2021-09-13T10:33:52,863][INFO ][o.e.n.Node               ] [suite] closed
  1> [2021-09-13T10:33:52,864][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2021-09-13T10:33:52,868][INFO ][o.e.c.s.ClusterApplierService] [node_s2] master node changed {previous [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}], current []}, term: 1, version: 11, reason: becoming candidate: Publication.onCompletion(false)
  1> [2021-09-13T10:33:52,868][WARN ][o.e.c.s.MasterService    ] [node_s2] failing [node-left[{node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr} reason: disconnected]]: failed to commit cluster state version [12]
  1> org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
  1>    at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1512) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1433) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:162) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication.access$500(Publication.java:31) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:361) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Coordinator$5.onFailure(Coordinator.java:1183) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.PublicationTransportHandler$PublicationContext.lambda$sendClusterStateDiff$8(PublicationTransportHandler.java:430) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:190) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:348) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:348) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) ~[main/:?]
  1>    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1263) ~[main/:?]
  1>    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1123) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[main/:?]
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum
  1>    at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:160) ~[main/:?]
  1>    ... 17 more
  1> [2021-09-13T10:33:52,868][ERROR][o.e.c.c.Coordinator      ] [node_s2] unexpected failure during [node-left]
  1> org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
  1>    at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1512) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1433) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:162) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication.access$500(Publication.java:31) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:361) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.Coordinator$5.onFailure(Coordinator.java:1183) ~[main/:?]
  1>    at org.elasticsearch.cluster.coordination.PublicationTransportHandler$PublicationContext.lambda$sendClusterStateDiff$8(PublicationTransportHandler.java:430) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:190) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:348) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:348) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) ~[main/:?]
  1>    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1263) ~[main/:?]
  1>    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1123) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) ~[main/:?]
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[main/:?]
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum
  1>    at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:160) ~[main/:?]
  1>    ... 17 more
  1> [2021-09-13T10:33:52,868][WARN ][o.e.c.NodeConnectionsService] [node_s2] failed to connect to {node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s1][127.0.0.1:34645] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,869][WARN ][o.e.c.NodeConnectionsService] [node_s2] failed to connect to {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:42495] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,874][INFO ][o.e.c.c.Coordinator      ] [node_s4] master node [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}] failed, restarting discovery
  1> org.elasticsearch.transport.NodeDisconnectedException: [node_s2][127.0.0.1:42361][disconnected] disconnected
  1> [2021-09-13T10:33:52,874][INFO ][o.e.c.c.Coordinator      ] [node_s3] master node [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}] failed, restarting discovery
  1> org.elasticsearch.transport.NodeDisconnectedException: [node_s2][127.0.0.1:42361][disconnected] disconnected
  1> [2021-09-13T10:33:52,874][INFO ][o.e.c.c.Coordinator      ] [node_s5] master node [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}] failed, restarting discovery
  1> org.elasticsearch.transport.NodeDisconnectedException: [node_s2][127.0.0.1:42361][disconnected] disconnected
  1> [2021-09-13T10:33:52,875][INFO ][o.e.c.s.ClusterApplierService] [node_s3] master node changed {previous [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}], current []}, term: 1, version: 11, reason: becoming candidate: onLeaderFailure
  1> [2021-09-13T10:33:52,875][INFO ][o.e.c.s.ClusterApplierService] [node_s5] master node changed {previous [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}], current []}, term: 1, version: 11, reason: becoming candidate: onLeaderFailure
  1> [2021-09-13T10:33:52,875][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2021-09-13T10:33:52,875][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2021-09-13T10:33:52,875][INFO ][o.e.c.s.ClusterApplierService] [node_s4] master node changed {previous [{node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr}], current []}, term: 1, version: 11, reason: becoming candidate: onLeaderFailure
  1> [2021-09-13T10:33:52,875][WARN ][o.e.c.NodeConnectionsService] [node_s3] failed to connect to {node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s1][127.0.0.1:34645] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,875][WARN ][o.e.c.NodeConnectionsService] [node_s4] failed to connect to {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s2][127.0.0.1:42361] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,876][WARN ][o.e.c.NodeConnectionsService] [node_s4] failed to connect to {node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s1][127.0.0.1:34645] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,875][WARN ][o.e.c.NodeConnectionsService] [node_s5] failed to connect to {node_s1}{egJGYd-TRd-7q_hTlQvZ6w}{iyCw83U8TyyWeUkUGJhH0g}{127.0.0.1}{127.0.0.1:34645}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s1][127.0.0.1:34645] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,877][WARN ][o.e.c.NodeConnectionsService] [node_s5] failed to connect to {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s2][127.0.0.1:42361] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,877][WARN ][o.e.c.NodeConnectionsService] [node_s3] failed to connect to {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:42495] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,876][WARN ][o.e.c.NodeConnectionsService] [node_s4] failed to connect to {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:42495] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,878][INFO ][o.e.n.Node               ] [suite] closed
  1> [2021-09-13T10:33:52,888][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2021-09-13T10:33:52,890][WARN ][o.e.c.NodeConnectionsService] [node_s3] failed to connect to {node_s2}{8sVOvGH-TXGTrJxtA274Wg}{s0QPGmSCRsOdxq4mE_8KhA}{127.0.0.1}{127.0.0.1:42361}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s2][127.0.0.1:42361] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,892][WARN ][o.e.c.NodeConnectionsService] [node_s5] failed to connect to {node_s0}{uNjo_NBhRw-1oIndJcnXBw}{5P4Uus8AQR-jcYjdYOljVA}{127.0.0.1}{127.0.0.1:42495}{imr} (tried [1] times)
  1> org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:42495] connect_exception
  1>    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:970) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[main/:?]
  1>    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1>    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
  1>    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112) ~[framework-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:440) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:429) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:250) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    at java.lang.Thread.run(Thread.java:831) [?:?]
  1> Caused by: java.net.ConnectException: Connection refused
  1>    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
  1>    at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
  1>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
  1>    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128) ~[elasticsearch-nio-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
  1>    ... 9 more
  1> [2021-09-13T10:33:52,917][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2021-09-13T10:33:52,917][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2021-09-13T10:33:52,927][INFO ][o.e.n.Node               ] [suite] closed
  2> sep 13, 2021 10:33:52 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
  2> ADVERTENCIA: Uncaught exception in thread: Thread[elasticsearch[node_s2][generic][T#3],5,TGRP-LegacyIndexStatsIT]
  2> java.lang.AssertionError
  2>    at __randomizedtesting.SeedInfo.seed([EA84AA48D964012D]:0)
  2>    at org.elasticsearch.common.bytes.ReleasableBytesReference.get(ReleasableBytesReference.java:96)
  2>    at org.elasticsearch.common.compress.DeflateCompressor.isCompressed(DeflateCompressor.java:50)
  2>    at org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:29)
  2>    at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:141)
  2>    at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:102)
  2>    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61)
  2>    at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:881)
  2>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
  2>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
  2>    at java.base/java.lang.Thread.run(Thread.java:831)

  1> [2021-09-13T10:33:52,938][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2021-09-13T10:33:52,944][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2021-09-13T10:33:52,944][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2021-09-13T10:33:52,946][INFO ][o.e.n.Node               ] [suite] closed
  1> [2021-09-13T10:33:52,952][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2021-09-13T10:33:52,956][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2021-09-13T10:33:52,956][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2021-09-13T10:33:52,959][INFO ][o.e.n.Node               ] [suite] closed
  2> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3555, name=elasticsearch[node_s2][generic][T#3], state=RUNNABLE, group=TGRP-LegacyIndexStatsIT]

        Caused by:
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([EA84AA48D964012D]:0)
            at org.elasticsearch.common.bytes.ReleasableBytesReference.get(ReleasableBytesReference.java:96)
            at org.elasticsearch.common.compress.DeflateCompressor.isCompressed(DeflateCompressor.java:50)
            at org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:29)
            at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:141)
            at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:102)
            at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61)
            at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:881)
            at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
            at java.base/java.lang.Thread.run(Thread.java:831)
  2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.stats.LegacyIndexStatsIT" -Dtests.seed=EA84AA48D964012D -Dtests.locale=en-US -Dtests.timezone=America/New_York -Druntime.java=16
  2> NOTE: test params are: codec=Asserting(Lucene87): {index_uuid=Lucene84, bar=PostingsFormat(name=Asserting), _field_names=PostingsFormat(name=Asserting), baz=Lucene84, _id=PostingsFormat(name=Asserting), type=PostingsFormat(name=Asserting)}, docValues:{_seq_no=DocValuesFormat(name=Lucene80), _primary_term=DocValuesFormat(name=Lucene80), _version=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=288, maxMBSortInHeap=5.856081002098787, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es-PA, timezone=America/St_Lucia
  2> NOTE: Linux 5.13.9-arch1-1 amd64/Eclipse Foundation 16.0.2 (64-bit)/cpus=12,threads=1,free=408530552,total=536870912
  2> NOTE: All tests run in this JVM: [DocumentActionsIT, PersistentTasksExecutorIT, QueryStringIT, QueryProfilerIT, SearchRedStateIndexIT, SearchWithRandomExceptionsIT, GeoBoundingBoxQueryGeoShapeIT, GeoPolygonIT, SourceFetchingIT, FunctionScorePluginIT, HistogramIT, DoubleTermsIT, LongTermsIT, DiversifiedSamplerIT, FilterIT, StatsIT, MedianAbsoluteDeviationIT, FiltersAggsRewriteIT, DateDerivativeIT, DerivativeIT, MissingValueIT, CustomHighlighterSearchIT, SearchWithRejectionsIT, SearchFieldsIT, MatchPhraseQueryIT, CorruptedFileIT, SearchIdleIT, WaitUntilRefreshIT, LegacyIndexStatsIT]
@nik9000 nik9000 added the >test-failure Triaged test failures from CI label Sep 13, 2021
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Sep 13, 2021
@DaveCTurner
Avoid early release of local forking requests
99bbb4b 
Today we protect against releasing a request from a remote node until
its handler has completed, but we do not have the same protection for
requests from the local node. This commit adds the missing refcounting.

Relates elastic#77407
Closes elastic#77634
@DaveCTurner DaveCTurner mentioned this issue Sep 13, 2021
Merged
@DaveCTurner DaveCTurner self-assigned this Sep 13, 2021
@DaveCTurner DaveCTurner added the :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. label Sep 13, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Sep 13, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

elasticsearchmachine pushed a commit that referenced this issue Sep 13, 2021
@DaveCTurner
Avoid early release of local forking requests (#77641)
89dd3c4 
Today we protect against releasing a request from a remote node until
its handler has completed, but we do not have the same protection for
requests from the local node. This commit adds the missing refcounting.

Relates #77407
Closes #77634
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Sep 13, 2021
@DaveCTurner
Avoid early release of local forking requests (elastic#77641)
e31809f 
Today we protect against releasing a request from a remote node until
its handler has completed, but we do not have the same protection for
requests from the local node. This commit adds the missing refcounting.

Relates elastic#77407
Closes elastic#77634
elasticsearchmachine pushed a commit that referenced this issue Sep 13, 2021
@DaveCTurner
Avoid early release of local forking requests (#77641) (#77645)
d7bcd55 
Today we protect against releasing a request from a remote node until
its handler has completed, but we do not have the same protection for
requests from the local node. This commit adds the missing refcounting.

Relates #77407
Closes #77634
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
:Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Avoid early release of local forking requests DaveCTurner/elasticsearch
3 participants
@nik9000 @DaveCTurner @elasticmachine