WatcherRestIT painless/10_basic/Test execute watch api with rest_total_hits_as_int failure

[jdconrad]

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+fast+part2/3/console

Did not reproduce with:

./gradlew :x-pack:qa:smoke-test-watcher:integTestRunner --tests "org.elasticsearch.smoketest.WatcherRestIT.test {yaml=painless/10_basic/Test execute watch api with rest_total_hits_as_int}" -Dtests.seed=6D9CCFE8486B1613 -Dtests.security.manager=true -Dtests.locale=mk -Dtests.timezone=Europe/Zurich -Dcompiler.java=12 -Druntime.java=11

Output:

09:45:13 > Task :x-pack:qa:smoke-test-watcher:integTestRunner
09:45:13 
09:45:13 
09:45:13 org.elasticsearch.smoketest.WatcherRestIT > test {yaml=painless/10_basic/Test execute watch api with rest_total_hits_as_int} FAILED
09:45:13     java.lang.AssertionError: Failure at [painless/10_basic:232]: watch_record.state didn't match expected value:
09:45:13                 watch_record.state: expected [executed] but was [not_executed_already_queued]
09:45:13 
09:45:13         Caused by:
09:45:13         java.lang.AssertionError: watch_record.state didn't match expected value:
09:45:13                     watch_record.state: expected [executed] but was [not_executed_already_queued]

[elasticmachine]

Pinging @elastic/es-core-features

[danielmitterdorfer]

Another instance in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/121/console (build id: 20190710093200-247A83CB)

10:02:04 org.elasticsearch.smoketest.WatcherRestIT > test {yaml=painless/10_basic/Test execute watch api with rest_total_hits_as_int} FAILED
10:02:04     java.lang.AssertionError: Failure at [painless/10_basic:232]: watch_record.state didn't match expected value:
10:02:04                 watch_record.state: expected [executed] but was [not_executed_already_queued]
10:02:04 
10:02:04         Caused by:
10:02:04         java.lang.AssertionError: watch_record.state didn't match expected value:
10:02:04                     watch_record.state: expected [executed] but was [not_executed_already_queued]

Reproduction line:

./gradlew :x-pack:qa:smoke-test-watcher:integTestRunner --tests "org.elasticsearch.smoketest.WatcherRestIT.test {yaml=painless/10_basic/Test execute watch api with rest_total_hits_as_int}" -Dtests.seed=EA25ECF3AB16AEEE -Dtests.security.manager=true -Dtests.locale=kam -Dtests.timezone=America/Louisville -Dcompiler.java=12 -Druntime.java=11

(does not reproduce locally)

Frequency: This failure started to occur on June 28, 2019 and happened 26 times in total.

[danielmitterdorfer]

I've muted this test now in #44159.

[henningandersen]

Got same failure on my PR build but for another variant of the test:
org.elasticsearch.smoketest.WatcherRestIT.test {yaml=painless/10_basic/Test execute watch api}

java.lang.AssertionError: Failure at [painless/10_basic:109]: watch_record.state didn't match expected value:
            watch_record.state: expected [executed] but was [not_executed_already_queued]

https://scans.gradle.com/s/bpuyre7xomnd2/tests/6ej7h6toy4af6-6g5rmtb4agtbo

[ebadyano]

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/349/console

[mayya-sharipova]

Another failure on 7.x today. Doesn't reproduce for me.

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java12,nodes=general-purpose/160/console
Build scan: https://gradle-enterprise.elastic.co/s/4gbxcngieqbjw