Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] InferenceCrudIT testSupportedStream failing #113430

Closed
elasticsearchmachine opened this issue Sep 24, 2024 · 10 comments · Fixed by #113456, #115543, #116662 or #118284
Closed

[CI] InferenceCrudIT testSupportedStream failing #113430

elasticsearchmachine opened this issue Sep 24, 2024 · 10 comments · Fixed by #113456, #115543, #116662 or #118284
Assignees
Labels
low-risk An open issue or test failure that is a low risk to future releases :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Sep 24, 2024

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:inference-service-tests:javaRestTest" --tests "org.elasticsearch.xpack.inference.InferenceCrudIT.testSupportedStream" -Dtests.seed=43985B609EF20347 -Dtests.locale=az-AZ -Dtests.timezone=Asia/Qatar -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: "{\"completion\":[{\"delta\":\"GKNETXİTKF\"}]}"
     but: was "{\"completion\":[{\"delta\":\"GKNETXITKF\"}]}"

Issue Reasons:

  • [8.x] 3 failures in test testSupportedStream (2.6% fail rate in 114 executions)
  • [8.x] 2 failures in step part-2 (8.7% fail rate in 23 executions)
  • [8.x] 2 failures in pipeline elasticsearch-pull-request (8.7% fail rate in 23 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

@elasticsearchmachine elasticsearchmachine added :ml Machine learning >test-failure Triaged test failures from CI labels Sep 24, 2024
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in test testSupportedStream (14.3% fail rate in 14 executions)

Build Scans:

@elasticsearchmachine elasticsearchmachine added Team:ML Meta label for the ML team needs:risk Requires assignment of a risk label (low, medium, blocker) labels Sep 24, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/ml-core (Team:ML)

@prwhelan prwhelan self-assigned this Sep 24, 2024
@prwhelan
Copy link
Member

Seems to happen frequently, ~14%, backporting to 8.x until we can figure it out

prwhelan added a commit to prwhelan/elasticsearch that referenced this issue Sep 24, 2024
@prwhelan
Copy link
Member

Actually the fix is faster - #113456
🤦

@elasticsearchmachine
Copy link
Collaborator Author

This issue is being re-opened because the referenced test still exists in muted-tests.yml, so it is still muted.

Please check and unmute the test if it can be unmuted.

In the future, automation will automatically re-open issues whenever they are closed without unmuting the test first.

Reach out in #es-delivery or tag the delivery team if you have an issue.

cc @elasticsearchmachine

prwhelan added a commit to prwhelan/elasticsearch that referenced this issue Oct 24, 2024
prwhelan added a commit that referenced this issue Oct 25, 2024
prwhelan added a commit to prwhelan/elasticsearch that referenced this issue Oct 25, 2024
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in test testSupportedStream (2.5% fail rate in 81 executions)

Build Scans:

@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 8.x

Mute Reasons:

  • [8.x] 3 failures in test testSupportedStream (2.6% fail rate in 114 executions)
  • [8.x] 2 failures in step part-2 (8.7% fail rate in 23 executions)
  • [8.x] 2 failures in pipeline elasticsearch-pull-request (8.7% fail rate in 23 executions)

Build Scans:

@maxhniebergall
Copy link
Contributor

It seems to me that the problem is that the string expected is GKNETXİTKF which contains the latin capital I with dot above character, where as the actual string is GKNETXITKF, which doesn't have the dot above. Possibly this is being converted to lowercase and then uppercase?

@prwhelan
Copy link
Member

It seems to me that the problem is that the string expected is GKNETXİTKF which contains the latin capital I with dot above character, where as the actual string is GKNETXITKF, which doesn't have the dot above. Possibly this is being converted to lowercase and then uppercase?

the random string generator is generating UTF16 strings - the REST controller decodes them as UTF8, which seems to convert İ to i. I couldn't seem to get the string generator to generate UTF8 strings (while excluding control messages, which the REST control is throwing errors on)

prwhelan added a commit to prwhelan/elasticsearch that referenced this issue Nov 12, 2024
randomAlpha is generating UTF16 strings, which return as UTF8 and fail
to match.  Generating random UTF8 strings can generate control
characters, which the REST handler throws an error on.  So we're
generating UUIDs.

Fix elastic#113430
prwhelan added a commit that referenced this issue Nov 20, 2024
randomAlpha is generating UTF16 strings, which return as UTF8 and fail
to match.  Generating random UTF8 strings can generate control
characters, which the REST handler throws an error on.  So we're
generating UUIDs.

Fix #113430
rjernst pushed a commit to rjernst/elasticsearch that referenced this issue Nov 20, 2024
randomAlpha is generating UTF16 strings, which return as UTF8 and fail
to match.  Generating random UTF8 strings can generate control
characters, which the REST handler throws an error on.  So we're
generating UUIDs.

Fix elastic#113430
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
randomAlpha is generating UTF16 strings, which return as UTF8 and fail
to match.  Generating random UTF8 strings can generate control
characters, which the REST handler throws an error on.  So we're
generating UUIDs.

Fix elastic#113430
@javanna
Copy link
Member

javanna commented Dec 9, 2024

Reopening because it's still muted in 8.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
low-risk An open issue or test failure that is a low risk to future releases :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
5 participants