Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] XPackRestIT class failing #111944

Closed
elasticsearchmachine opened this issue Aug 16, 2024 · 10 comments · Fixed by #113021
Closed

[CI] XPackRestIT class failing #111944

elasticsearchmachine opened this issue Aug 16, 2024 · 10 comments · Fixed by #113021
Assignees
@prwhelan
Labels
blocker :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Aug 16, 2024 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:yamlRestCompatTest" --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=rollup/rollup_search/Search error against live index}" -Dtests.seed=1158360447451BB -Dtests.locale=es-419 -Dtests.timezone=US/Central -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [rollup/rollup_search:34]: got unexpected warning header [
	299 Elasticsearch-9.0.0-3d6e9a2850a5d91c7fee0dbe6153903b909c4937 "The rollup functionality will be removed in in Elasticsearch 9.0. See docs for more information."
]

Issue Reasons:

  • [main] 105 failures in class org.elasticsearch.xpack.test.rest.XPackRestIT (10.6% fail rate in 986 executions)
  • [main] 3 failures in step rest-compat (6.5% fail rate in 46 executions)
  • [main] 5 failures in step part-3 (2.6% fail rate in 196 executions)
  • [main] 2 failures in step part3 (4.7% fail rate in 43 executions)
  • [main] 9 failures in step rest-compatibility (4.0% fail rate in 227 executions)
  • [main] 4 failures in step encryption-at-rest (30.8% fail rate in 13 executions)
  • [main] 4 failures in step sles-12_platform-support-unix (33.3% fail rate in 12 executions)
  • [main] 4 failures in step ubuntu-2004_platform-support-unix (50.0% fail rate in 8 executions)
  • [main] 2 failures in step ubuntu-1804_platform-support-unix (18.2% fail rate in 11 executions)
  • [main] 2 failures in step debian-10_platform-support-unix (16.7% fail rate in 12 executions)
  • [main] 4 failures in step rhel-7_platform-support-unix (33.3% fail rate in 12 executions)
  • [main] 2 failures in step centos-7_platform-support-unix (16.7% fail rate in 12 executions)
  • [main] 3 failures in step oraclelinux-8_platform-support-unix (33.3% fail rate in 9 executions)
  • [main] 5 failures in step rhel-9_platform-support-unix (41.7% fail rate in 12 executions)
  • [main] 3 failures in step ubuntu-2204_platform-support-unix (27.3% fail rate in 11 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Search Relevance/Ranking Scoring, rescoring, rank evaluation. >test-failure Triaged test failures from CI labels Aug 16, 2024
elasticsearchmachine added a commit that referenced this issue Aug 16, 2024
@elasticsearchmachine
Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.…
445be15 
…xpack.test.rest.XPackRestIT #111944
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 10 failures in class org.elasticsearch.xpack.test.rest.XPackRestIT (1.1% fail rate in 908 executions)
  • [main] 5 failures in step part-3 (3.9% fail rate in 129 executions)
  • [main] 4 failures in step rest-compatibility (2.9% fail rate in 137 executions)
  • [main] 5 failures in pipeline elasticsearch-pull-request (3.6% fail rate in 138 executions)

Build Scans:

@elasticsearchmachine elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label Aug 16, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-search-relevance (Team:Search Relevance)

@elasticsearchmachine elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Aug 16, 2024
weizijun added a commit to weizijun/elasticsearch that referenced this issue Aug 16, 2024
@weizijun
Merge branch 'upstream/main' into add-alibabacloud-inference
b6b6da5 
* upstream/main: (91 commits)
  Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.xpack.test.rest.XPackRestIT elastic#111944
  Add audit_unenrolled_* attributes to fleet-agents template (elastic#111909)
  Fix windows memory locking (elastic#111866)
  Update OAuth2 OIDC SDK (elastic#108799)
  Adds a warning about manually mounting snapshots managed by ILM (elastic#111883)
  Update geoip fixture files and utility methods (elastic#111913)
  Updated Function Score Query Test with Explain Fixes for 8.15.1 (elastic#111929)
  Mute org.elasticsearch.xpack.sql.qa.security.JdbcCsvSpecIT org.elasticsearch.xpack.sql.qa.security.JdbcCsvSpecIT elastic#111923
  [ESQL] date nanos binary comparisons (elastic#111908)
  [DOCS] Documents output_field behavior after multiple inference runs (elastic#111875)
  Add additional BlobCacheMetrics, expose BlobCacheMetrics via SharedBlobCacheService (elastic#111730)
  Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT elastic#111923
  Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {agg-ordering.testHistogramDateTimeWithCountAndOrder_2} elastic#111919
  Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {date.testDateParseHaving} elastic#111921
  Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {agg-ordering.testHistogramDateTimeWithCountAndOrder_1} elastic#111918
  Mute org.elasticsearch.xpack.sql.qa.multi_cluster_with_security.JdbcCsvSpecIT test {datetime.testDateTimeParseHaving} elastic#111922
  Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT elastic#111923
  Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT test {agg-ordering.testHistogramDateTimeWithCountAndOrder_1} elastic#111918
  Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT test {datetime.testDateTimeParseHaving} elastic#111922
  Mute org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT test {date.testDateParseHaving} elastic#111921
  ...

# Conflicts:
#	server/src/main/java/org/elasticsearch/TransportVersions.java
@kkrik-es kkrik-es mentioned this issue Aug 20, 2024
Merged
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Sep 4, 2024
@elasticsearchmachine @cbuescher
Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.…
f4dbd1f 
…xpack.test.rest.XPackRestIT elastic#111944
davidkyle pushed a commit to davidkyle/elasticsearch that referenced this issue Sep 5, 2024
@elasticsearchmachine @davidkyle
Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.…
26f2fb6 
…xpack.test.rest.XPackRestIT elastic#111944
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.15

Mute Reasons:

  • [8.15] 4 failures in class org.elasticsearch.xpack.test.rest.XPackRestIT (0.5% fail rate in 885 executions)
  • [8.15] 3 failures in pipeline elasticsearch-periodic-platform-support (20.0% fail rate in 15 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Sep 13, 2024
@elasticsearchmachine
Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.…
d72d1e6 
…xpack.test.rest.XPackRestIT #111944
@benwtrent benwtrent added :ml Machine learning and removed :Search Relevance/Ranking Scoring, rescoring, rank evaluation. labels Sep 13, 2024
@elasticsearchmachine elasticsearchmachine added Team:ML Meta label for the ML team and removed Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Sep 13, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 4 consecutive failures in class org.elasticsearch.xpack.test.rest.XPackRestIT
  • [main] 2 consecutive failures in step sles-12_platform-support-unix
  • [main] 3 consecutive failures in step oraclelinux-7_platform-support-unix
  • [main] 3 consecutive failures in step windows-2019_checkpart3_platform-support-windows
  • [main] 2 consecutive failures in step windows-2016_checkpart3_platform-support-windows
  • [main] 3 consecutive failures in step windows-2019_checkrestcompat_platform-support-windows
  • [main] 3 consecutive failures in step windows-2022_checkrestcompat_platform-support-windows
  • [main] 2 consecutive failures in step ubuntu-2004_platform-support-unix
  • [main] 2 consecutive failures in step rhel-9_platform-support-unix
  • [main] 64 failures in class org.elasticsearch.xpack.test.rest.XPackRestIT (6.7% fail rate in 950 executions)
  • [main] 2 failures in step sles-12_platform-support-unix (25.0% fail rate in 8 executions)
  • [main] 2 failures in step rhel-7_platform-support-unix (20.0% fail rate in 10 executions)
  • [main] 3 failures in step oraclelinux-7_platform-support-unix (30.0% fail rate in 10 executions)
  • [main] 2 failures in step windows-2022_checkpart3_platform-support-windows (20.0% fail rate in 10 executions)
  • [main] 2 failures in step release-tests (12.5% fail rate in 16 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Sep 14, 2024
@elasticsearchmachine
Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.…
b712ab5 
…xpack.test.rest.XPackRestIT #111944
@alex-spies alex-spies added blocker and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Sep 16, 2024
@alex-spies
Copy link
Contributor

Hey @elastic/ml-core ! I just noticed that the bot muted the whole of XPackRestIT due to this. I think that's a lot of integration tests, so I marked this as blocker. This is muted since August 16, do you know if the cause of the mute is maybe addressed already?

@prwhelan prwhelan self-assigned this Sep 17, 2024
@prwhelan
Copy link
Member 

XPackRestIT > test {p0=ml/3rd_party_deployment/Test start deployment fails while model download in progress} FAILED

From random sample:

@prwhelan prwhelan mentioned this issue Sep 17, 2024
Closed
@prwhelan
Copy link
Member

Looks like it's all 3rd party model downloads, we'll mute these tests and unmute the whole suite - there's work being done to fix/slow down model download

prwhelan added a commit to prwhelan/elasticsearch that referenced this issue Sep 17, 2024
@prwhelan
[ML] Mute 3p download tests
1bbe18e 
Relates elastic#111944
davidkyle added a commit to davidkyle/elasticsearch that referenced this issue Sep 17, 2024
@davidkyle
Revert "Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasti…
c1d67b1 
…csearch.xpack.test.rest.XPackRestIT elastic#111944"

This reverts commit b712ab5.

# Conflicts:
#	muted-tests.yml
@davidkyle davidkyle mentioned this issue Sep 17, 2024
Merged
davidkyle added a commit that referenced this issue Sep 17, 2024
@davidkyle
Revert "Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasti…
97a724e 
…csearch.xpack.test.rest.XPackRestIT #111944" (#113021)

This reverts commit b712ab5.
davidkyle added a commit to davidkyle/elasticsearch that referenced this issue Sep 17, 2024
@davidkyle
Revert "Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasti…
350058b 
…csearch.xpack.test.rest.XPackRestIT elastic#111944"
@davidkyle davidkyle mentioned this issue Sep 17, 2024
Merged
elasticsearchmachine added a commit that referenced this issue Sep 17, 2024
@elasticsearchmachine
Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.…
be76492 
…xpack.test.rest.XPackRestIT #111944
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 105 failures in class org.elasticsearch.xpack.test.rest.XPackRestIT (10.6% fail rate in 986 executions)
  • [main] 3 failures in step rest-compat (6.5% fail rate in 46 executions)
  • [main] 5 failures in step part-3 (2.6% fail rate in 196 executions)
  • [main] 2 failures in step part3 (4.7% fail rate in 43 executions)
  • [main] 9 failures in step rest-compatibility (4.0% fail rate in 227 executions)
  • [main] 4 failures in step encryption-at-rest (30.8% fail rate in 13 executions)
  • [main] 4 failures in step sles-12_platform-support-unix (33.3% fail rate in 12 executions)
  • [main] 4 failures in step ubuntu-2004_platform-support-unix (50.0% fail rate in 8 executions)
  • [main] 2 failures in step ubuntu-1804_platform-support-unix (18.2% fail rate in 11 executions)
  • [main] 2 failures in step debian-10_platform-support-unix (16.7% fail rate in 12 executions)
  • [main] 4 failures in step rhel-7_platform-support-unix (33.3% fail rate in 12 executions)
  • [main] 2 failures in step centos-7_platform-support-unix (16.7% fail rate in 12 executions)
  • [main] 3 failures in step oraclelinux-8_platform-support-unix (33.3% fail rate in 9 executions)
  • [main] 5 failures in step rhel-9_platform-support-unix (41.7% fail rate in 12 executions)
  • [main] 3 failures in step ubuntu-2204_platform-support-unix (27.3% fail rate in 11 executions)

Build Scans:

elasticsearchmachine pushed a commit that referenced this issue Sep 17, 2024
@davidkyle
Revert "Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasti…
adf33e9 
…csearch.xpack.test.rest.XPackRestIT #111944" (#113037)

…csearch.xpack.test.rest.XPackRestIT #111944"

The tests failed because of
#111684 which has since
been reverted
@martijnvg
Copy link
Member

This is unrelated to the original problem. It was automatically re-opened because of a rollup change that was bad. It has been reverted. I will unmute this test.

javanna pushed a commit to javanna/elasticsearch that referenced this issue Sep 18, 2024
@davidkyle @javanna
Revert "Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasti…
ce3747c 
…csearch.xpack.test.rest.XPackRestIT elastic#111944" (elastic#113021)

This reverts commit b712ab5.
javanna pushed a commit to javanna/elasticsearch that referenced this issue Sep 18, 2024
@elasticsearchmachine @javanna
Mute org.elasticsearch.xpack.test.rest.XPackRestIT org.elasticsearch.…
4f9e009 
…xpack.test.rest.XPackRestIT elastic#111944
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prwhelan prwhelan

Labels
blocker :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
5 participants
@martijnvg @prwhelan @benwtrent @elasticsearchmachine @alex-spies