Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky tests when asserting memory pools are empty upon QueryRunner termination in integration tests #11275

Closed
findepi opened this issue Mar 2, 2022 · 8 comments · Fixed by #12333
Closed

Flaky tests when asserting memory pools are empty upon QueryRunner termination in integration tests #11275

findepi opened this issue Mar 2, 2022 · 8 comments · Fixed by #12333
Assignees
@arhimondr
Labels
bug Something isn't working

Comments

@findepi
Copy link
Member

findepi commented Mar 2, 2022

#11102, a solution to #11093

is causing build failures, eg in TestDistributedEngineOnlyQueries

https://github.com/trinodb/trino/runs/5380151310?check_suite_focus=true

Error:  io.trino.tests.TestDistributedEngineOnlyQueries.close  Time elapsed: 10.309 s  <<< FAILURE!
org.junit.ComparisonFailure: [memory reservation on server_0(worker)] expected:<[0]L> but was:<[362072]L>
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at io.trino.testing.AbstractTestQueryFramework.lambda$checkQueryMemoryReleased$0(AbstractTestQueryFramework.java:130)
	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:74)
	at io.trino.testing.AbstractTestQueryFramework.checkQueryMemoryReleased(AbstractTestQueryFramework.java:120)
	at io.trino.testing.AbstractTestQueryFramework.close(AbstractTestQueryFramework.java:101)
	at jdk.internal.reflect.GeneratedMethodAccessor4198.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:217)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:115)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

cc @losipiuk @arhimondr
@findepi findepi added the bug Something isn't working label Mar 2, 2022
This was referenced Mar 2, 2022
Merged
Merged
@arhimondr
Copy link
Contributor

Let me take a look

@arhimondr arhimondr self-assigned this Mar 2, 2022
@losipiuk losipiuk mentioned this issue Mar 4, 2022
Merged
@brandboat brandboat mentioned this issue Mar 16, 2022
Merged
@findepi
Copy link
Member Author

findepi commented Mar 21, 2022

https://github.com/trinodb/trino/runs/5601210218?check_suite_focus=true

@findepi findepi mentioned this issue Mar 21, 2022
Merged
@findepi
Copy link
Member Author

findepi commented Mar 25, 2022

https://github.com/trinodb/trino/runs/5645073521?check_suite_focus=true

@findepi findepi mentioned this issue Mar 25, 2022
Merged
@lukasz-stec lukasz-stec mentioned this issue Apr 25, 2022
Merged
This was referenced May 10, 2022
Closed
Merged
@arhimondr arhimondr reopened this May 13, 2022
@findepi
Copy link
Member Author

findepi commented May 24, 2022

https://github.com/trinodb/trino/runs/6572074793?check_suite_focus=true

Error:  Tests run: 773, Failures: 1, Errors: 0, Skipped: 25, Time elapsed: 925.633 s <<< FAILURE! - in TestSuite
Error:  io.trino.plugin.hive.TestHiveFaultTolerantExecutionConnectorTest.close  Time elapsed: 30.832 s  <<< FAILURE!
java.lang.AssertionError: 
Expected memory reservation on server_1(worker)to be 0 but was 79736; detailed memory usage:
 20220524_120822_04156_duwjd:
   SQL: INSERT INTO invalid_partition_value VALUES (4, 'test' || chr(13))
   state: FAILED
   memoryReservation: 79736
   taggedMemoryReservaton: {HashAggregationOperator=79736}

	at org.testng.Assert.fail(Assert.java:94)
	at io.trino.testing.AbstractTestQueryFramework.assertMemoryPoolReleased(AbstractTestQueryFramework.java:148)
	at io.trino.testing.AbstractTestQueryFramework.lambda$checkQueryMemoryReleased$0(AbstractTestQueryFramework.java:133)
	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:77)
	at io.trino.testing.AbstractTestQueryFramework.checkQueryMemoryReleased(AbstractTestQueryFramework.java:126)
	at io.trino.testing.AbstractTestQueryFramework.close(AbstractTestQueryFramework.java:108)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:217)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:115)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

@findepi findepi reopened this May 24, 2022
@findepi findepi mentioned this issue May 24, 2022
Merged
@arhimondr
Copy link
Contributor

Looking more

@arhimondr
Copy link
Contributor

Incredibly difficult to reproduce. Managed to reproduce it locally after more than 20000 of invocations:

2022-08-17T12:51:47.480-0500	INFO	main	io.trino.testng.services.FlakyTestRetryAnalyzer	FlakyTestRetryAnalyzer not enabled: CONTINUOUS_INTEGRATION environment is not detected or system property 'io.trino.testng.services.FlakyTestRetryAnalyzer.enabled' is not set to 'true' (actual: <not set>)

java.lang.AssertionError: Expected memory reservation on server_2(coordinator)to be 0 but was 28152; detailed memory usage:
 20220817_175116_04118_3b2ag:
   SQL: INSERT INTO invalid_partition_value VALUES (4, 'test' || chr(13))
   state: FAILED
   memoryReservation: 28152
   taggedMemoryReservaton: {HashAggregationOperator=28152}


	at org.testng.Assert.fail(Assert.java:94)
	at io.trino.testing.AbstractTestQueryFramework.assertMemoryPoolReleased(AbstractTestQueryFramework.java:148)
	at io.trino.testing.AbstractTestQueryFramework.lambda$checkQueryMemoryReleased$0(AbstractTestQueryFramework.java:133)
	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:77)
	at io.trino.testing.AbstractTestQueryFramework.checkQueryMemoryReleased(AbstractTestQueryFramework.java:126)
	at io.trino.plugin.hive.BaseHiveConnectorTest.testInvalidPartitionValue(BaseHiveConnectorTest.java:5794)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at org.testng.TestRunner.privateRun(TestRunner.java:756)
	at org.testng.TestRunner.run(TestRunner.java:610)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
	at org.testng.TestNG.runSuites(TestNG.java:1133)
	at org.testng.TestNG.run(TestNG.java:1104)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)

@arhimondr arhimondr mentioned this issue Aug 17, 2022
Merged
@arhimondr
Copy link
Contributor

Fix: #13721

@arhimondr
Copy link
Contributor

Closed with #13721

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arhimondr arhimondr

Labels
bug Something isn't working
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix memory accounting leak in aggregation arhimondr/trino
2 participants
@findepi @arhimondr