Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky TestHashAggregationOperator.testMemoryReservationYield, TestMarkDistinctOperator.testMemoryReservationYield on JDK 20 #16933

Closed
ebyhr opened this issue Apr 7, 2023 · 11 comments · Fixed by #17491
Closed

Flaky TestHashAggregationOperator.testMemoryReservationYield, TestMarkDistinctOperator.testMemoryReservationYield on JDK 20 #16933

ebyhr opened this issue Apr 7, 2023 · 11 comments · Fixed by #17491
Labels
bug Something isn't working

Comments

@ebyhr
Copy link
Member

ebyhr commented Apr 7, 2023 

Error:  io.trino.operator.TestHashAggregationOperator.testMemoryReservationYield[bigint](2)  Time elapsed: 0.106 s  <<< FAILURE!
java.lang.IllegalArgumentException: Invalid taskId another_query.0.0
	at io.trino.spi.QueryId.checkArgument(QueryId.java:114)
	at io.trino.spi.QueryId.parseDottedId(QueryId.java:103)
	at io.trino.execution.TaskId.getQueryId(TaskId.java:57)
	at io.trino.memory.MemoryPool.reserve(MemoryPool.java:133)
	at io.trino.operator.GroupByHashYieldAssertion.finishOperatorWithYieldingGroupByHash(GroupByHashYieldAssertion.java:108)
	at io.trino.operator.TestHashAggregationOperator.testMemoryReservationYield(TestHashAggregationOperator.java:414)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

https://github.com/trinodb/trino/actions/runs/4639584809/jobs/8212420239
@ebyhr ebyhr added bug Something isn't working test labels Apr 7, 2023
@findepi
Copy link
Member

findepi commented Apr 14, 2023 

2023-04-13T12:17:32.198-0600	INFO	pool-24-thread-1	io.trino.testng.services.ProgressLoggingListener	[TEST START] io.trino.operator.TestRowNumberOperator.testMemoryReservationYield [varchar]
2023-04-13T12:17:35.447-0600	INFO	pool-24-thread-1	io.trino.testng.services.ProgressLoggingListener	[TEST SUCCESS] io.trino.operator.TestRowNumberOperator.testMemoryReservationYield [varchar]; (took: 3.3 seconds)
2023-04-13T12:17:35.449-0600	INFO	pool-24-thread-1	io.trino.testng.services.ProgressLoggingListener	[TEST START] io.trino.operator.TestRowNumberOperator.testMemoryReservationYield [bigint]
2023-04-13T12:17:35.597-0600	INFO	pool-24-thread-1	io.trino.testng.services.FlakyTestRetryAnalyzer	not retrying; @Flaky annotation not present
2023-04-13T12:17:35.597-0600	ERROR	pool-24-thread-1	io.trino.testng.services.ProgressLoggingListener	[TEST FAILURE] io.trino.operator.TestRowNumberOperator.testMemoryReservationYield [bigint]; (took: 0.1 seconds)
java.lang.IllegalArgumentException: Invalid taskId another_query.0.0
	at io.trino.spi.QueryId.checkArgument(QueryId.java:114)
	at io.trino.spi.QueryId.parseDottedId(QueryId.java:103)
	at io.trino.execution.TaskId.getQueryId(TaskId.java:57)
	at io.trino.memory.MemoryPool.reserve(MemoryPool.java:133)
	at io.trino.operator.GroupByHashYieldAssertion.finishOperatorWithYieldingGroupByHash(GroupByHashYieldAssertion.java:108)
	at io.trino.operator.TestRowNumberOperator.testMemoryReservationYield(TestRowNumberOperator.java:174)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

cc @sopel39 @lukasz-stec

@findepi findepi removed the test label Apr 14, 2023
@sopel39
Copy link
Member

sopel39 commented Apr 14, 2023

@findepi did it start to happen recently?

@findepi
Copy link
Member

findepi commented Apr 14, 2023

@sopel39 i do not know. maybe https://nineinchnick.github.io/trino-cicd/reports/flaky/#flaky-tests-in-last-30-days has some answers

@sopel39
Copy link
Member

sopel39 commented Apr 14, 2023

@findepi it's not on that list

@sopel39
Copy link
Member

sopel39 commented Apr 14, 2023

@arhimondr is it related to 93f9b7d?

@findepi findepi changed the title Flaky TestHashAggregationOperator.testMemoryReservationYield Flaky TestHashAggregationOperator.testMemoryReservationYield, TestMarkDistinctOperator.testMemoryReservationYield Apr 18, 2023
@findepi
Copy link
Member

findepi commented Apr 18, 2023

https://github.com/trinodb/trino/actions/runs/4725414489/jobs/8384161226?pr=17074

Error:  Tests run: 8000, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 823.848 s <<< FAILURE! - in TestSuite
Error:  io.trino.operator.TestMarkDistinctOperator.testMemoryReservationYield[varchar](1)  Time elapsed: 0.494 s  <<< FAILURE!
java.lang.IllegalArgumentException: Invalid taskId another_query.0.0
	at io.trino.spi.QueryId.checkArgument(QueryId.java:114)
	at io.trino.spi.QueryId.parseDottedId(QueryId.java:103)
	at io.trino.execution.TaskId.getQueryId(TaskId.java:57)
	at io.trino.memory.MemoryPool.reserve(MemoryPool.java:133)
	at io.trino.operator.GroupByHashYieldAssertion.finishOperatorWithYieldingGroupByHash(GroupByHashYieldAssertion.java:108)
	at io.trino.operator.TestMarkDistinctOperator.testMemoryReservationYield(TestMarkDistinctOperator.java:179)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

Error:  io.trino.operator.TestMarkDistinctOperator.testMemoryReservationYield[bigint](2)  Time elapsed: 0.114 s  <<< FAILURE!
java.lang.IllegalArgumentException: Invalid taskId another_query.0.0
	at io.trino.spi.QueryId.checkArgument(QueryId.java:114)
	at io.trino.spi.QueryId.parseDottedId(QueryId.java:103)
	at io.trino.execution.TaskId.getQueryId(TaskId.java:57)
	at io.trino.memory.MemoryPool.reserve(MemoryPool.java:133)
	at io.trino.operator.GroupByHashYieldAssertion.finishOperatorWithYieldingGroupByHash(GroupByHashYieldAssertion.java:108)
	at io.trino.operator.TestMarkDistinctOperator.testMemoryReservationYield(TestMarkDistinctOperator.java:179)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

@findepi findepi mentioned this issue Apr 18, 2023
Merged
@radek-kondziolka radek-kondziolka mentioned this issue May 9, 2023
Merged
@sopel39
Copy link
Member

sopel39 commented May 9, 2023

https://github.com/trinodb/trino/actions/runs/4919263242/jobs/8787828461?pr=17265

@findepi findepi mentioned this issue May 10, 2023
Merged
@chenjian2664
Copy link
Contributor

I first saw it on 6 April, and looked the QueryId code, but did not get any useful infos.

@wendigo wendigo mentioned this issue May 11, 2023
Closed
@findepi
Copy link
Member

findepi commented May 12, 2023

@wendigo observed that this typically fails for JDK 20.

@martint martint mentioned this issue May 12, 2023
Merged
@findepi findepi mentioned this issue May 13, 2023
Closed
@findepi
Copy link
Member

findepi commented May 13, 2023

in #17455 it was https://github.com/trinodb/trino/actions/runs/4945027952/jobs/8843538570

@findepi findepi changed the title Flaky TestHashAggregationOperator.testMemoryReservationYield, TestMarkDistinctOperator.testMemoryReservationYield Flaky TestHashAggregationOperator.testMemoryReservationYield, TestMarkDistinctOperator.testMemoryReservationYield on JDK 20 May 13, 2023
@findepi
Copy link
Member

findepi commented May 13, 2023 •
edited
Loading

trino-main is run with JDK 17 and 20, yet so far all failures were observed with JDK 20. changed issue title accordingly.
i think the number of data points is already statistically significant, but it would help if everyone involved was adding a link to the failing job, when they see the failure. Currently, not everyone does that, so there might be some observation-related bias.

stress test on the CI with further evidence: #17490

@findepi findepi mentioned this issue May 13, 2023
Merged
@wendigo wendigo mentioned this issue Jul 13, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Put testing with with JDK 20 on hold findepi/trino
4 participants
@findepi @sopel39 @ebyhr @chenjian2664