Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] SearchWithRandomIOExceptionsIT testRandomDirectoryIOExceptions failing #114824

Closed
elasticsearchmachine opened this issue Oct 15, 2024 · 9 comments
Closed

[CI] SearchWithRandomIOExceptionsIT testRandomDirectoryIOExceptions failing #114824

elasticsearchmachine opened this issue Oct 15, 2024 · 9 comments
Assignees
@original-brownbear
Labels
medium-risk An open issue or test failure that is a medium risk to future releases :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Oct 15, 2024 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT.testRandomDirectoryIOExceptions" -Dtests.seed=631C2F340601E2A2 -Dtests.locale=ms-Latn-MY -Dtests.timezone=Asia/Kamchatka -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Shard [test][0] is still locked after 5 sec waiting

Issue Reasons:

  • [8.x] 5 failures in test testRandomDirectoryIOExceptions (0.7% fail rate in 738 executions)
  • [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (10.0% fail rate in 20 executions)
  • [8.x] 2 failures in pipeline elasticsearch-periodic (10.0% fail rate in 20 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Search/Search Search-related issues that do not fall into other categories >test-failure Triaged test failures from CI Team:Search Meta label for search team needs:risk Requires assignment of a risk label (low, medium, blocker) labels Oct 15, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-search (Team:Search)

@benwtrent
Copy link
Member

benwtrent commented Oct 29, 2024 •
edited
Loading

Its been failing since April or before. All the failures are effectively the same.

Expected 0 bytes for requests in-flight but got N bytes; where N is some non zero number near 100.

Some other full stacks:


2> java.lang.AssertionError: All incoming requests on node [node_s1] should have finished. Expected 0 bytes for requests in-flight but got 99 bytes; pending tasks [[]] |  
-- | --
  | at org.elasticsearch.test.InternalTestCluster.lambda$assertRequestsFinished$37(InternalTestCluster.java:2548) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1429) |  
  | at org.elasticsearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2539) |  
  | at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2512) |  
  | at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:611) |  
  | at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2420) |  
  | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) |  
  | at java.base/java.lang.reflect.Method.invoke(Method.java:580) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20) |  
  | at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) |  
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) |  
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) |  
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) |  
  | at java.base/java.lang.Thread.run(Thread.java:1570) |  
  |   |  
  | java.lang.AssertionError: searcher was not warmed up yet for source[field_has_value] |  
  | at __randomizedtesting.SeedInfo.seed([E1A2C0470C9B7EFC]:0) |  
  | at org.elasticsearch.index.engine.InternalEngine.assertSearcherIsWarmedUp(InternalEngine.java:523) |  
  | at org.elasticsearch.index.engine.Engine$1.acquireSearcherInternal(Engine.java:831) |  
  | at org.elasticsearch.index.engine.Engine$SearcherSupplier.acquireSearcher(Engine.java:1484) |  
  | at org.elasticsearch.index.engine.Engine.acquireSearcher(Engine.java:882) |  
  | at org.elasticsearch.index.engine.Engine.acquireSearcher(Engine.java:875) |  
  | at org.elasticsearch.index.engine.Engine.acquireSearcher(Engine.java:871) |  
  | at org.elasticsearch.index.shard.IndexShard$RefreshFieldHasValueListener.afterRefresh(IndexShard.java:4084) |  
  | at org.apache.lucene.search.ReferenceManager.notifyRefreshListenersRefreshed(ReferenceManager.java:275) |  
  | at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:182) |  
  | at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) |  
  | at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2056) |  
  | at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2027) |  
  | at org.elasticsearch.index.engine.Engine.lambda$externalRefresh$11(Engine.java:1252) |  
  | at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:356) |  
  | at org.elasticsearch.index.engine.Engine.externalRefresh(Engine.java:1250) |  
  | at org.elasticsearch.index.shard.IndexShard.externalRefresh(IndexShard.java:1296) |  
  | at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnReplica(TransportShardRefreshAction.java:103) |  
  | at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnReplica(TransportShardRefreshAction.java:38) |  
  | at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:676) |  
  | at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:647) |  
  | at org.elasticsearch.index.shard.IndexShard.lambda$innerAcquireReplicaOperationPermit$44(IndexShard.java:3788) |  
  | at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:219) |  
  | at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400) |  
  | at org.elasticsearch.index.shard.IndexShardOperationPermits.innerAcquire(IndexShardOperationPermits.java:255) |  
  | at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:203) |  
  | at org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$42(IndexShard.java:3718) |  
  | at org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:3830) |  
  | at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:3712) |  
  | at org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1120) |  
  | at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:771) |  
  | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) |  
  | at org.elasticsearch.action.support.replication.TransportReplicationAction.handleReplicaRequest(TransportReplicationAction.java:625) |  
  | at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:76) |  
  | at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:289) |  
  | at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:302) |  
  | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) |  
  | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) |  
  | at java.base/java.lang.Thread.run(Thread.java:1570)

@benwtrent benwtrent added medium-risk An open issue or test failure that is a medium risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Oct 29, 2024
@benwtrent
Copy link
Member

The most common thing we waiting on is the health_check persistent task or empty (I think the empty is a race condition, where a task finished between us checking and then verifying in flight tasks).

@javanna javanna added :Search Foundations/Search Catch all for Search Foundations and removed :Search/Search Search-related issues that do not fall into other categories labels Nov 7, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch label Nov 7, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@elasticsearchmachine elasticsearchmachine removed the Team:Search Meta label for search team label Nov 7, 2024
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in test testRandomDirectoryIOExceptions (0.5% fail rate in 404 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Nov 11, 2024
@elasticsearchmachine
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
14944f2 
…stRandomDirectoryIOExceptions #114824
@benwtrent
Copy link
Member

The latest failures

2> REPRODUCE WITH: ./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT.testRandomDirectoryIOExceptions" -Dtests.seed=F8F0E5FB8684B939 -Dtests.locale=ee -Dtests.timezone=America/Tortola -Druntime.java=23 |  
-- | --
  | 2> REPRODUCE WITH: ./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT.testRandomDirectoryIOExceptions" -Dtests.seed=F8F0E5FB8684B939 -Dtests.locale=ee -Dtests.timezone=America/Tortola -Druntime.java=23 |  
  | 2> java.lang.AssertionError: All incoming requests on node [node_s4] should have finished. Expected 0 bytes for requests in-flight but got 78 bytes; pending tasks [[]] |  
  | at org.elasticsearch.test.InternalTestCluster.lambda$assertRequestsFinished$37(InternalTestCluster.java:2548) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1439) |  
  | at org.elasticsearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2539) |  
  | at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2512) |  
  | at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:611) |  
  | at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2422) |  
  | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) |  
  | at java.base/java.lang.reflect.Method.invoke(Method.java:580) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20) |  
  | at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) |  
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) |  
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) |  
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) |  
  | at java.base/java.lang.Thread.run(Thread.java:1575) |  
  |   |  
  | java.lang.AssertionError: searcher was not warmed up yet for source[field_has_value] |  
  | at __randomizedtesting.SeedInfo.seed([F8F0E5FB8684B939]:0) |  
  | at org.elasticsearch.index.engine.InternalEngine.assertSearcherIsWarmedUp(InternalEngine.java:523) |  
  | at org.elasticsearch.index.engine.Engine$1.acquireSearcherInternal(Engine.java:828) |  
  | at org.elasticsearch.index.engine.Engine$SearcherSupplier.acquireSearcher(Engine.java:1481) |  
  | at org.elasticsearch.index.engine.Engine.acquireSearcher(Engine.java:879) |  
  | at org.elasticsearch.index.engine.Engine.acquireSearcher(Engine.java:872) |  
  | at org.elasticsearch.index.engine.Engine.acquireSearcher(Engine.java:868) |  
  | at org.elasticsearch.index.shard.IndexShard$RefreshFieldHasValueListener.afterRefresh(IndexShard.java:4083) |  
  | at org.apache.lucene.search.ReferenceManager.notifyRefreshListenersRefreshed(ReferenceManager.java:275) |  
  | at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:182) |  
  | at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) |  
  | at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2056) |  
  | at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2027) |  
  | at org.elasticsearch.index.shard.IndexShard.finalizeRecovery(IndexShard.java:2309) |  
  | at org.elasticsearch.indices.recovery.RecoveryTarget.lambda$finalizeRecovery$2(RecoveryTarget.java:401) |  
  | at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:356) |  
  | at org.elasticsearch.indices.recovery.RecoveryTarget.finalizeRecovery(RecoveryTarget.java:382) |  
  | at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$5.handleRequest(PeerRecoveryTargetService.java:214) |  
  | at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$5.handleRequest(PeerRecoveryTargetService.java:207) |  
  | at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:661) |  
  | at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:648) |  
  | at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:76) |  
  | at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:289) |  
  | at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:302) |  
  | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) |  
  | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) |  
  | at java.base/java.lang.Thread.run(Thread.java:1575)

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Nov 11, 2024
@original-brownbear
Unmute a lot of fixed tests from search race condition
d8a36b6 
closes elastic#115664 elastic#113430 elastic#115717 elastic#115705 elastic#115970 elastic#115988 elastic#115810 elastic#116027
 elastic#115754 elastic#116097 elastic#115818 elastic#116377 elastic#114824
@original-brownbear original-brownbear mentioned this issue Nov 11, 2024
Merged
original-brownbear added a commit that referenced this issue Nov 11, 2024
@original-brownbear
Unmute a lot of fixed tests from search race condition (#116587)
a21d375 
These are all fixed by #116264

closes #115664 #113430 #115717 #115705 #115970 #115988 #115810 #116027
 #115754 #116097 #115818 #116377 #114824
jozala pushed a commit that referenced this issue Nov 13, 2024
@elasticsearchmachine @jozala
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
7983e96 
…stRandomDirectoryIOExceptions #114824
jozala pushed a commit that referenced this issue Nov 13, 2024
@original-brownbear @jozala
Unmute a lot of fixed tests from search race condition (#116587)
d2e8a30 
These are all fixed by #116264

closes #115664 #113430 #115717 #115705 #115970 #115988 #115810 #116027
 #115754 #116097 #115818 #116377 #114824
afoucret pushed a commit to afoucret/elasticsearch that referenced this issue Nov 14, 2024
@elasticsearchmachine @afoucret
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
ee3bbc3 
…stRandomDirectoryIOExceptions elastic#114824
afoucret pushed a commit to afoucret/elasticsearch that referenced this issue Nov 14, 2024
@original-brownbear @afoucret
Unmute a lot of fixed tests from search race condition (elastic#116587)
776dfba 
These are all fixed by elastic#116264

closes elastic#115664 elastic#113430 elastic#115717 elastic#115705 elastic#115970 elastic#115988 elastic#115810 elastic#116027
 elastic#115754 elastic#116097 elastic#115818 elastic#116377 elastic#114824
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in test testRandomDirectoryIOExceptions (0.4% fail rate in 547 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Nov 15, 2024
@elasticsearchmachine
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
4d210c8 
…stRandomDirectoryIOExceptions #114824
@original-brownbear original-brownbear self-assigned this Nov 15, 2024
salvatore-campagna pushed a commit to salvatore-campagna/elasticsearch that referenced this issue Nov 18, 2024
@elasticsearchmachine @salvatore-campagna
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
fdd18bc 
…stRandomDirectoryIOExceptions elastic#114824
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.x

Mute Reasons:

  • [8.x] 5 failures in test testRandomDirectoryIOExceptions (0.7% fail rate in 738 executions)
  • [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (10.0% fail rate in 20 executions)
  • [8.x] 2 failures in pipeline elasticsearch-periodic (10.0% fail rate in 20 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Nov 18, 2024
@elasticsearchmachine
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
9984f7a 
…stRandomDirectoryIOExceptions #114824
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
@elasticsearchmachine @alexey-ivanov-es
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
da8e8d1 
…stRandomDirectoryIOExceptions elastic#114824
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
@original-brownbear @alexey-ivanov-es
Unmute a lot of fixed tests from search race condition (elastic#116587)
94ffee1 
These are all fixed by elastic#116264

closes elastic#115664 elastic#113430 elastic#115717 elastic#115705 elastic#115970 elastic#115988 elastic#115810 elastic#116027
 elastic#115754 elastic#116097 elastic#115818 elastic#116377 elastic#114824
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
@elasticsearchmachine @alexey-ivanov-es
Mute org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT te…
4011029 
…stRandomDirectoryIOExceptions elastic#114824
@javanna
Copy link
Member

javanna commented Dec 9, 2024

Unmuted by #116587

@javanna javanna closed this as completed Dec 9, 2024
@cbuescher cbuescher mentioned this issue Jan 10, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
medium-risk An open issue or test failure that is a medium risk to future releases :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@javanna @benwtrent @original-brownbear @elasticsearchmachine