Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] MockDiskUsagesIT.testAutomaticReleaseOfIndexBlock fails #45338

Closed
dimitris-athanasiou opened this issue Aug 8, 2019 · 4 comments · Fixed by #45342 or #47387
Closed

[CI] MockDiskUsagesIT.testAutomaticReleaseOfIndexBlock fails #45338

dimitris-athanasiou opened this issue Aug 8, 2019 · 4 comments · Fixed by #45342 or #47387
Assignees
@DaveCTurner
Labels
:Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >test-failure Triaged test failures from CI

Comments

@dimitris-athanasiou
Copy link
Contributor

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+release-tests/232/console

Build scan: https://gradle.com/s/wnz3akbwhhnne

Reproduce with:

./gradlew :server:integTest --tests "org.elasticsearch.cluster.routing.allocation.decider.MockDiskUsagesIT.testAutomaticReleaseOfIndexBlock" -Dtests.seed=815A6FA6949ACF83 -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=mt-MT -Dtests.timezone=Asia/Ashgabat -Dcompiler.java=12 -Druntime.java=8

Failure:

java.lang.AssertionError: Request executed with success but a ClusterBlockException was expectedClose stacktrace
at __randomizedtesting.SeedInfo.seed([815A6FA6949ACF83:CAD3581BABADC932]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertBlocked(ElasticsearchAssertions.java:166)
at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertBlocked(ElasticsearchAssertions.java:191)
at org.elasticsearch.cluster.routing.allocation.decider.MockDiskUsagesIT.testAutomaticReleaseOfIndexBlock(MockDiskUsagesIT.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)

Could not reproduce locally.
@dimitris-athanasiou dimitris-athanasiou added >test-failure Triaged test failures from CI :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) labels Aug 8, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dimitris-athanasiou
Copy link
Contributor Author

This test was introduced recently in #42559

dimitris-athanasiou added a commit that referenced this issue Aug 8, 2019
@dimitris-athanasiou
Mute testAutomaticReleaseOfIndexBlock
e53bb05 
Relates #45338
@dimitris-athanasiou
Copy link
Contributor Author

Muted the test in both master and 7.x by 42e8b36 and e53bb05 respectively.

dimitris-athanasiou added a commit that referenced this issue Aug 8, 2019
@dimitris-athanasiou
Mute testAutomaticReleaseOfIndexBlock
42e8b36 
Relates #45338
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Aug 8, 2019
@DaveCTurner
Add missing wait to testAutomaticReleaseOfIndexBlock
8a49673 
Today the test waits for one of the shards to be blocked, but this does not
mean that the block has been applied on all nodes, so a subsequent indexing
operation may still go through.

Fixes elastic#45338
@DaveCTurner DaveCTurner mentioned this issue Aug 8, 2019
Merged
original-brownbear pushed a commit to original-brownbear/elasticsearch that referenced this issue Aug 8, 2019
@DaveCTurner @original-brownbear
Add missing wait to testAutomaticReleaseOfIndexBlock (elastic#45342)
8033f90 
Today the test waits for one of the shards to be blocked, but this does not
mean that the block has been applied on all nodes, so a subsequent indexing
operation may still go through.

Fixes elastic#45338
original-brownbear pushed a commit that referenced this issue Aug 8, 2019
@DaveCTurner @original-brownbear
Add missing wait to testAutomaticReleaseOfIndexBlock (#45342)
79e4375 
Today the test waits for one of the shards to be blocked, but this does not
mean that the block has been applied on all nodes, so a subsequent indexing
operation may still go through.

Fixes #45338
@original-brownbear original-brownbear mentioned this issue Aug 8, 2019
Merged
original-brownbear added a commit that referenced this issue Aug 8, 2019
@original-brownbear
Add missing wait to testAutomaticReleaseOfIndexBlock (#45342) (#45351)
b19de55 
Today the test waits for one of the shards to be blocked, but this does not
mean that the block has been applied on all nodes, so a subsequent indexing
operation may still go through.

Fixes #45338
@dakrone
Copy link
Member

dakrone commented Oct 1, 2019

This failed again today:

  2> java.lang.AssertionError: Request executed with success but a ClusterBlockException was expected
        at __randomizedtesting.SeedInfo.seed([CE5B07216BD6F25A:85D2309C54E1F4EB]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertBlocked(ElasticsearchAssertions.java:166)
        at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertBlocked(ElasticsearchAssertions.java:191)
        at org.elasticsearch.cluster.routing.allocation.decider.MockDiskUsagesIT.testAutomaticReleaseOfIndexBlock(MockDiskUsagesIT.java:197)

Reproduction line (this did not reproduce for me) is:

./gradlew ':server:integTest' --tests "org.elasticsearch.cluster.routing.allocation.decider.MockDiskUsagesIT.testAutomaticReleaseOfIndexBlock" -Dtests.seed=CE5B07216BD6F25A -Dtests.security.manager=true -Dtests.locale=en-GD -Dtests.timezone=America/Catamarca -Dcompiler.java=12 -Druntime.java=12

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java12,nodes=general-purpose/240/console
https://gradle-enterprise.elastic.co/s/cyiv347vuav34

@dakrone dakrone reopened this Oct 1, 2019
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Oct 1, 2019
@DaveCTurner
Keep nodes above watermark in testAutomaticReleaseOfIndexBlock
bf62a06 
Today the comment boldly claims that this line of code keeps nodes above the
10-byte low watermark when in fact this is not true at all. This change fixes
this so that it really does keep nodes above the low watermark.

Fixes elastic#45338. Again.
@DaveCTurner DaveCTurner mentioned this issue Oct 1, 2019
Merged
DaveCTurner added a commit that referenced this issue Oct 1, 2019
@DaveCTurner
Keep nodes above watermark in testAutomaticReleaseOfIndexBlock (#47387)
650c0c6 
Today the comment boldly claims that this line of code keeps nodes above the
10-byte low watermark when in fact this is not true at all. This change fixes
this so that it really does keep nodes above the low watermark.

Fixes #45338. Again.
DaveCTurner added a commit that referenced this issue Oct 1, 2019
@DaveCTurner
Keep nodes above watermark in testAutomaticReleaseOfIndexBlock (#47387)
99b25d3 
Today the comment boldly claims that this line of code keeps nodes above the
10-byte low watermark when in fact this is not true at all. This change fixes
this so that it really does keep nodes above the low watermark.

Fixes #45338. Again.
DaveCTurner added a commit that referenced this issue Oct 1, 2019
@DaveCTurner
Keep nodes above watermark in testAutomaticReleaseOfIndexBlock (#47387)
4f31275 
Today the comment boldly claims that this line of code keeps nodes above the
10-byte low watermark when in fact this is not true at all. This change fixes
this so that it really does keep nodes above the low watermark.

Fixes #45338. Again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
:Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
4 participants
@dakrone @DaveCTurner @dimitris-athanasiou @elasticmachine