Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NoOpEngineTests failure #51303

Closed
astefan opened this issue Jan 22, 2020 · 6 comments · Fixed by #51331 or #53345
Closed

NoOpEngineTests failure #51303

astefan opened this issue Jan 22, 2020 · 6 comments · Fixed by #51331 or #53345
Assignees
@dnhatn
Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test-failure Triaged test failures from CI

Comments

@astefan
Copy link
Contributor

astefan commented Jan 22, 2020

This one reproduces with

gradlew :server:test --tests "org.elasticsearch.index.engine.NoOpEngineTests.testNoOpEngineStats" -Dtests.seed=C1530B8CE0376A32 -Dtests.security.manager=true -Dtests.locale=be-BY -Dtests.timezone=SystemV/AST4ADT -Dcompiler.java=13

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu-16.04&&immutable/528/console
Build Scans: https://gradle-enterprise.elastic.co/s/zhi7ubzxpt5ye

java.lang.AssertionError: expected:<5> but was:<4>
	at __randomizedtesting.SeedInfo.seed([C1530B8CE0376A32:20CD69E2C68E02D3]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.elasticsearch.index.engine.NoOpEngineTests.testNoOpEngineStats(NoOpEngineTests.java:150)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)
@astefan astefan added >test-failure Triaged test failures from CI :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jan 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Engine)

@tlrx
Copy link
Member

tlrx commented Jan 22, 2020

I suspect that this is related to #51192

@dnhatn dnhatn self-assigned this Jan 22, 2020
@dnhatn dnhatn mentioned this issue Jan 22, 2020
Merged
dnhatn added a commit that referenced this issue Jan 23, 2020
@dnhatn
Do not wrap soft-deletes reader for segment stats (#51331)
5132715 
IndexWriter might not filter out fully deleted segments if retention 
leases exist or the number of the retaining operations is non-zero.
SoftDeletesDirectoryReaderWrapper, however, always filters out fully
deleted segments.

This change uses the original directory reader when calculating segment 
stats instead.

Relates #51192
Closes #51303
dnhatn added a commit that referenced this issue Jan 23, 2020
@dnhatn
Do not wrap soft-deletes reader for segment stats (#51331)
acf84b6 
IndexWriter might not filter out fully deleted segments if retention
leases exist or the number of the retaining operations is non-zero.
SoftDeletesDirectoryReaderWrapper, however, always filters out fully
deleted segments.

This change uses the original directory reader when calculating segment
stats instead.

Relates #51192
Closes #51303
dnhatn added a commit that referenced this issue Jan 23, 2020
@dnhatn
Do not wrap soft-deletes reader for segment stats (#51331)
96b3d6b 
IndexWriter might not filter out fully deleted segments if retention
leases exist or the number of the retaining operations is non-zero.
SoftDeletesDirectoryReaderWrapper, however, always filters out fully
deleted segments.

This change uses the original directory reader when calculating segment
stats instead.

Relates #51192
Closes #51303
dnhatn added a commit that referenced this issue Jan 23, 2020
@dnhatn
Do not wrap soft-deletes reader for segment stats (#51331)
8b00fd0 
IndexWriter might not filter out fully deleted segments if retention
leases exist or the number of the retaining operations is non-zero.
SoftDeletesDirectoryReaderWrapper, however, always filters out fully
deleted segments.

This change uses the original directory reader when calculating segment
stats instead.

Relates #51192
Closes #51303
@probakowski
Copy link
Contributor

Another instance of this error: https://gradle-enterprise.elastic.co/s/tp7px2dwmroio

This reproduces with: ./gradlew ':server:test' --tests "org.elasticsearch.index.engine.NoOpEngineTests.testNoOpEngineStats" -Dtests.seed=DDB7E0B7006591FA -Dtests.security.manager=true -Dtests.locale=id-ID -Dtests.timezone=America/Indiana/Petersburg

Error:

java.lang.AssertionError: expected:<915> but was:<1834>
        at __randomizedtesting.SeedInfo.seed([DDB7E0B7006591FA:3C2982D926DCF91B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.junit.Assert.assertEquals(Assert.java:631)
        at org.elasticsearch.index.engine.NoOpEngineTests.testNoOpEngineStats(NoOpEngineTests.java:146)

@probakowski probakowski reopened this Mar 4, 2020
@mayya-sharipova
Copy link
Contributor

mayya-sharipova commented Mar 6, 2020 •
edited
Loading

Another instance of this test failure on 7.x windows today:

Log:https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2019/479/console
Build Scans: https://gradle-enterprise.elastic.co/s/tejy2wezodv4y

REPRODUCE WITH: gradlew ':server:test' --tests "org.elasticsearch.index.engine.NoOpEngineTests.testNoOpEngineStats"
-Dtests.seed=7294AABC54550A83
-Dtests.security.manager=true
-Dtests.locale=hu-HU
-Dtests.timezone=Atlantic/Faeroe
-Dcompiler.java=13

java.lang.AssertionError: expected:<414> but was:<830>
	at __randomizedtesting.SeedInfo.seed([7294AABC54550A83:930AC8D272EC6262]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.elasticsearch.index.engine.NoOpEngineTests.testNoOpEngineStats(NoOpEngineTests.java:146)

Reproduces for me as well on my Mac laptop.

@tlrx
Copy link
Member

tlrx commented Mar 9, 2020

@dnhatn would you have a look please? It only reproduces in 7.x and I suspect some combination of #51905 with #51331, but I had a look and 'm not sure how to fix it.

@dnhatn
Copy link
Member

dnhatn commented Mar 9, 2020

@tlrx Looking...

@dnhatn dnhatn mentioned this issue Mar 10, 2020
Merged
dnhatn added a commit that referenced this issue Mar 10, 2020
@dnhatn
Fix doc_stats and segment_stats of ReadOnlyEngine (#53345)
352e59c 
We can't always have the same segment stats and doc stats between 
InternalEngine and ReadOnlyEngine if there are some fully deleted 
segments. ReadOnlyEngine always filters out them. InternalEngine,
however, will keep them if peer recovery retention leases exist or the
number of the retaining operations is non-zero.

This change reverts the fix in #51331 and uses the wrapped reader to 
calculate the segment stats and doc stats. For the test, we need to
disable the extra retaining soft-deletes operations.

Closes #51303
dnhatn added a commit that referenced this issue Mar 11, 2020
@dnhatn
Fix doc_stats and segment_stats of ReadOnlyEngine (#53345)
24f1147 
We can't always have the same segment stats and doc stats between
InternalEngine and ReadOnlyEngine if there are some fully deleted
segments. ReadOnlyEngine always filters out them. InternalEngine,
however, will keep them if peer recovery retention leases exist or the
number of the retaining operations is non-zero.

This change reverts the fix in #51331 and uses the wrapped reader to
calculate the segment stats and doc stats. For the test, we need to
disable the extra retaining soft-deletes operations.

Closes #51303
dnhatn added a commit that referenced this issue Mar 11, 2020
@dnhatn
Fix doc_stats and segment_stats of ReadOnlyEngine (#53345)
4ceabfb 
We can't always have the same segment stats and doc stats between
InternalEngine and ReadOnlyEngine if there are some fully deleted
segments. ReadOnlyEngine always filters out them. InternalEngine,
however, will keep them if peer recovery retention leases exist or the
number of the retaining operations is non-zero.

This change reverts the fix in #51331 and uses the wrapped reader to
calculate the segment stats and doc stats. For the test, we need to
disable the extra retaining soft-deletes operations.

Closes #51303
@codebrain codebrain mentioned this issue Apr 1, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
6 participants
@tlrx @astefan @probakowski @mayya-sharipova @dnhatn @elasticmachine