Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] HeapAttackIT testManyConcat failing #100678

Closed
Tracked by #100528
piergm opened this issue Oct 11, 2023 · 5 comments · Fixed by #104107
Closed
Tracked by #100528

[CI] HeapAttackIT testManyConcat failing #100678

piergm opened this issue Oct 11, 2023 · 5 comments · Fixed by #104107
Assignees
@dnhatn
Labels
:Analytics/ES|QL AKA ESQL low-risk An open issue or test failure that is a low risk to future releases Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI

Comments

@piergm
Copy link
Member

piergm commented Oct 11, 2023

Build scan:
https://gradle-enterprise.elastic.co/s/z27lugpake4um/tests/:x-pack:plugin:esql:qa:server:single-node:javaRestTest/org.elasticsearch.xpack.esql.qa.single_node.HeapAttackIT/testManyConcat
Reproduction line:

./gradlew ':x-pack:plugin:esql:qa:server:single-node:javaRestTest' --tests "org.elasticsearch.xpack.esql.qa.single_node.HeapAttackIT.testManyConcat" -Dtests.seed=6986B005E1B4E2DE -Dtests.locale=vi-VN -Dtests.timezone=America/Lima -Druntime.java=17

Applicable branches:
main

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.esql.qa.single_node.HeapAttackIT&tests.test=testManyConcat
Failure excerpt:

java.lang.OutOfMemoryError: Java heap space

  at __randomizedtesting.SeedInfo.seed([6986B005E1B4E2DE:10470D421C8F6C49]:0)
  at org.apache.http.util.CharArrayBuffer.expand(CharArrayBuffer.java:60)
  at org.apache.http.util.CharArrayBuffer.append(CharArrayBuffer.java:90)
  at org.apache.http.util.EntityUtils.toString(EntityUtils.java:228)
  at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
  at org.elasticsearch.xpack.esql.qa.single_node.HeapAttackIT.testManyConcat(HeapAttackIT.java:184)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
@piergm piergm added :Analytics/ES|QL AKA ESQL >test-failure Triaged test failures from CI blocker labels Oct 11, 2023
@elasticsearchmachine elasticsearchmachine added the Team:QL (Deprecated) Meta label for query languages team label Oct 11, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-ql (Team:QL)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

@ChrisHegarty
Copy link
Contributor

This is not a blocker. Removing the blocker label.

@ChrisHegarty ChrisHegarty added low-risk An open issue or test failure that is a low risk to future releases and removed blocker labels Oct 11, 2023
@ChrisHegarty ChrisHegarty mentioned this issue Oct 12, 2023
Closed
23 tasks
@davidkyle davidkyle mentioned this issue Oct 16, 2023
Merged
arteam pushed a commit that referenced this issue Oct 17, 2023
@davidkyle
[CI] Mute Failing tests in HeapAttackIT (#100900)
3a44943 
For #100678 and and #100640
@nik9000 nik9000 mentioned this issue Oct 24, 2023
Closed
21 tasks
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Oct 31, 2023
@davidkyle @cbuescher
[CI] Mute Failing tests in HeapAttackIT (elastic#100900)
51a8b9e 
For elastic#100678 and and elastic#100640
@mark-vieira
Copy link
Contributor

We continue to have several failures in `HeapAttackIT1 every day. It's across a wide range of tests and recently looks to be mostly timeouts presumably due to the test taking out the test cluster.

The nature of this test seems to make it very susceptible to instability. We should probably mute it entirely unless we think there's a simple fix here given how much noise it's creating.

@wchaparro wchaparro added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jan 2, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@elasticsearchmachine elasticsearchmachine removed the Team:QL (Deprecated) Meta label for query languages team label Jan 2, 2024
@dnhatn dnhatn mentioned this issue Jan 10, 2024
Merged
dnhatn added a commit that referenced this issue Jan 10, 2024
@dnhatn
Re-enable HeapAttackIT (#104107)
709c0f5 
This PR enables ESQL heap attack tests. I have run this suite over 500 
iterations with different hardware configurations for the last two days,
and all have been successful with the changes in #104159.

Additionally, this PR adds an action that can trigger OOM to generate a 
heap dump if a test takes more than 5 minutes. I've seen cases
(previously with our CI) where the test didn't result in OOM but was
taking too long. Having the ability to inspect the heap in such cases
would be beneficial.

Closes #103527
Closes #100678
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Analytics/ES|QL AKA ESQL low-risk An open issue or test failure that is a low risk to future releases Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Re-enable HeapAttackIT dnhatn/elasticsearch
6 participants
@mark-vieira @wchaparro @dnhatn @elasticsearchmachine @ChrisHegarty @piergm