[CI] InternalAutoDateHistogramTests#testReduceRandom reproducable failure

[cbuescher]

Failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/6700/console

Reproducable locally with this seed:

REPRODUCE WITH: ./gradlew :server:test \
  -Dtests.seed=6345BDE5AED7493F \
  -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests \
  -Dtests.method="testReduceRandom" \
  -Dtests.security.manager=true \
  -Dtests.locale=lv \
  -Dtests.timezone=HST

Error:

java.lang.AssertionError: expected:<{1532036050000=59, 1532036051000=58, 1532036052000=3, 1532036053000=71, 1532036054000=20, 1532036055000=48, 1532036110000=100, 1532036170000=49, 1532036230000=92}> but was:<{1532036050000=259, 1532036080000=0, 1532036110000=100, 1532036140000=0, 1532036170000=49, 1532036200000=0, 1532036230000=92}>
	at __randomizedtesting.SeedInfo.seed([6345BDE5AED7493F:A37811F4888AD069]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:107)
	at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:45)
	at org.elasticsearch.test.InternalAggregationTestCase.testReduceRandom(InternalAggregationTestCase.java:264)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)

Other seeds seem to be okay, so I don't think the test needs to be muted right away.

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@pcsanwald are you up for taking a look at this or would you like me to look into it?

[pcsanwald]

I’m happy to have a look

[colings86]

Thanks @pcsanwald

[pcsanwald]

I'm still investigating, but, some more detail on the cause:

java.lang.AssertionError: 
Expected :{1532090899000=59, 1532090900000=58, 1532090901000=3, 1532090902000=71, 1532090903000=20, 1532090904000=48, 1532090959000=100, 1532091019000=49, 1532091079000=92}
Actual   :{1532090899000=259, 1532090929000=0, 1532090959000=100, 1532090989000=0, 1532091019000=49, 1532091049000=0, 1532091079000=92}

The problem is a mismatch between the interval expected by the test (which is 1s intervals) and the interval actually returned by the aggregator (which is 30s intervals). The total number of documents aggregated is the same. Still working out which number this seed is producing that is causing this behavior, as I agree with @cbuescher that the problem is not reproducible with other seeds I've tried.

[pcsanwald]

I've muted the test in 320f1d2. @colings86 and I identified the problem and I'll open a PR against this issue when ready.

[davidkyle]

Reopening this issue as CI has thrown up a couple more failures. Sadly these do not reproduce

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/220/console

java.lang.AssertionError: expected:<{1533427200000=1510}> but was:<{1533427200000=1460, 1533513600000=50}>
	at __randomizedtesting.SeedInfo.seed([A53DF863CAC51132:65005472EC988864]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:137)
	at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:43)
	at org.elasticsearch.test.InternalAggregationTestCase.testReduceRandom(InternalAggregationTestCase.java:264)

./gradlew :server:test \
  -Dtests.seed=A53DF863CAC51132 \
  -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests \
  -Dtests.method="testReduceRandom" \
  -Dtests.security.manager=true \
  -Dtests.locale=bas \
  -Dtests.timezone=Australia/LHI

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/1798/console

java.lang.AssertionError: expected:<{1533254400000=655}> but was:<{1533254400000=279, 1533340800000=376}>
	at __randomizedtesting.SeedInfo.seed([263ED3D5E0617B78:E6037FC4C63CE22E]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:137)
	at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:43)
	at org.elasticsearch.test.InternalAggregationTestCase.testReduceRandom(InternalAggregationTestCase.java:264)

gradlew :server:test \
  -Dtests.seed=263ED3D5E0617B78 \
  -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests \
  -Dtests.method="testReduceRandom" \
  -Dtests.security.manager=true \
  -Dtests.locale=sq \
  -Dtests.timezone=Hongkong

[pcsanwald]

thanks @davidkyle I will have a look at these failures today.

[DaveCTurner]

I encountered this issue on a PR build which yielded this REPRODUCE WITH line that does reproduce on master (6fe6247):

./gradlew :server:test -Dtests.seed=55C6AF1793E2275B -Dtests.class=org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests -Dtests.method="testReduceRandom" -Dtests.security.manager=true -Dtests.locale=en-IN -Dtests.timezone=Pacific/Chuuk

The failing stack trace is:

   > Throwable #1: java.lang.AssertionError: expected:<{1533600000000=585}> but was:<{1533600000000=372, 1533686400000=213}>
   >    at __randomizedtesting.SeedInfo.seed([55C6AF1793E2275B:95FB0306B5BFBE0D]:0)
   >    at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:137)
   >    at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogramTests.assertReduced(InternalAutoDateHistogramTests.java:43)
   >    at org.elasticsearch.test.InternalAggregationTestCase.testReduceRandom(InternalAggregationTestCase.java:264)
   >    at java.lang.Thread.run(Thread.java:748)

[pcsanwald]

thanks @DaveCTurner, this is helpful. I've muted the offending test in 3ce984d while I work on this fix.