[CI] MixedClusterClientYamlTestSuiteIT test {p0=field_caps/10_basic/Get object and nested field caps} failing #80335
Comments
martijnvg
added
Team:Search
|
Pinging @elastic/es-search (Team:Search) |
|
@martijnvg I was looking if this error ever occured in our main CI tests but couldn't find anything similar so far, you saw this in #80330, right? In terms of reproduction: that PR was running bwc pn 7.16 against 7.7 if I understand the repro line correctly? |
|
I found one similar looking error in a CI run from Oct 19th now, but not many more: |
|
Could not reproduce this locally yet despite many runs with different seeds, I will also try running the same seed multiple times. The "indices" section should only be left out if either all indices where mapped the same way or if "test3" didn't exist, but its clearly there and should be mapped differently. |
|
I'm looking at trying to reproduce this with some unexpected ordering in which the |
|
Maybe not directly related, but in a failure from Nov 7th. I see an index shard failure in this tests method, also in a mixed cluster scenario: |
|
The two ones mentioned above on Nov 10th are slightly different failures though in other test cases in the same base test. And from https://gradle-enterprise.elastic.co/s/yh2sbheezc264 (this is 10_basic/Field caps with include_unmapped) |
|
I've looked a bit more at the failure history of Here we have an see a failure in the field_caps response for the missing index: Since the start of these series of occasaional failures lies shortly after the merge of #79212 I suspect some changes in error handling/propagation to come into play here. I'm still looking at reproducing the error locally but maybe @dnhatn has some theory what might be happening here and how we can sometimes get these incomplete results after merging the responses. |
This suggests something broke in the StreamInput#readEnum for IndicesOptions enum, but that might be a one-off, I haven't seen these kind of errors in the other test failures so far. So maybe a red hering, but the other cases remain suspect |
|
@cbuescher Thanks for your investigation and the ping. I will look into the failure. |
|
This has happened again: https://gradle-enterprise.elastic.co/s/dn3fz4vt5rxz2 repro line: trace |
|
This seems like a related failure |
|
This happened in a very similar way in another test in the same suite, This is 7.17 doing a mixed cluster test with 7.7. |
|
Another similar failure in
|
|
It's unclear to me what is causing the test failures. To rule out some basic stuff (shards relocating / timing) and provide a bit more details on what's happening, I've opened #82716. Would be interesting to see if we can observe more test failures after merging that. |
|
Closed, see #83405 |
Build scan:
https://gradle-enterprise.elastic.co/s/lgvrzlewzdu3e/tests/:qa:mixed-cluster:v7.7.0%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=field_caps%2F10_basic%2FGet%20object%20and%20nested%20field%20caps%7D
Reproduction line:
./gradlew ':qa:mixed-cluster:v7.7.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=field_caps/10_basic/Get object and nested field caps}" -Dtests.seed=E59B64A1F501FD5 -Dtests.bwc=true -Dtests.locale=es-AR -Dtests.timezone=IST -Druntime.java=8Applicable branches:
7.16
Reproduces locally?:
No
Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Dfield_caps/10_basic/Get%20object%20and%20nested%20field%20caps%7D
Failure excerpt:
The text was updated successfully, but these errors were encountered: