Flaky "unable to create native thread: possibly out of memory or process/resource limits reached" in Kudu

[ebyhr]

https://github.com/trinodb/trino/actions/runs/3193106867/jobs/5211602866

Error:  Tests run: 544, Failures: 10, Errors: 0, Skipped: 128, Time elapsed: 1,474.844 s <<< FAILURE! - in TestSuite
Error:  io.trino.plugin.kudu.TestKuduConnectorTest.testAlterTableAddLongColumnName  Time elapsed: 1.62 s  <<< FAILURE!
io.trino.testing.QueryFailedException: unable to create native thread: possibly out of memory or process/resource limits reached
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:123)
	at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:480)
	at io.trino.testing.QueryAssertions.assertUpdate(QueryAssertions.java:71)
	at io.trino.testing.AbstractTestQueryFramework.assertUpdate(AbstractTestQueryFramework.java:373)
	at io.trino.testing.AbstractTestQueryFramework.assertUpdate(AbstractTestQueryFramework.java:368)
	at io.trino.testing.BaseConnectorTest.testAlterTableAddLongColumnName(BaseConnectorTest.java:2303)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: java.lang.Exception: SQL: CREATE TABLE test_long_columngyc4pwj9rp AS SELECT 123 x
		at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:483)
		... 17 more
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:802)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1353)
	at io.trino.execution.StateMachine.safeExecute(StateMachine.java:320)
	at io.trino.execution.StateMachine.addStateChangeListener(StateMachine.java:289)
	at io.trino.execution.StageStateMachine.addStateChangeListener(StageStateMachine.java:134)
	at io.trino.execution.SqlStage.initialize(SqlStage.java:131)
	at io.trino.execution.SqlStage.createSqlStage(SqlStage.java:103)
	at io.trino.execution.scheduler.StageManager.create(StageManager.java:91)
	at io.trino.execution.scheduler.PipelinedQueryScheduler.<init>(PipelinedQueryScheduler.java:222)
	at io.trino.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:519)
	at io.trino.execution.SqlQueryExecution.start(SqlQueryExecution.java:395)
	at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:249)
	at io.trino.dispatcher.LocalDispatchQuery.lambda$startExecution$7(LocalDispatchQuery.java:143)
	at io.trino.$gen.Trino_testversion____20221005_232147_71.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

...

Error:  Failures: 
Error:    TestKuduConnectorTest>AbstractTestQueryFramework.close:115->AbstractTestQueryFramework.checkQueryInfosFinal:184->AbstractTestQueryFramework.lambda$checkQueryInfosFinal$5:184->AbstractTestQueryFramework.lambda$checkQueryInfosFinal$4:197 QueryInfo is expected to be final: SELECT count(regionkey) FROM nation
Error:    TestKuduConnectorTest>BaseConnectorTest.testAggregation:376->AbstractTestQueryFramework.assertQuery:292->AbstractTestQueryFramework.assertQuery:297 Execution of 'actual' query failed: SELECT count(regionkey) FROM nation
Error:    TestKuduConnectorTest>BaseConnectorTest.testAlterTableAddLongColumnName:2303->AbstractTestQueryFramework.assertUpdate:368->AbstractTestQueryFramework.assertUpdate:373 » QueryFailed
Error:    TestKuduConnectorTest>BaseConnectorTest.testAlterTableRenameColumnToLongName:2325->AbstractTestQueryFramework.assertQuery:302 Execution of 'actual' query failed: SELECT colzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz FROM test_long_column12d2onck9y
Error:    TestKuduConnectorTest>BaseConnectorTest.testCaseSensitiveDataMapping:4178->BaseConnectorTest.testDataMapping:4101->AbstractTestQueryFramework.assertQuery:302 Execution of 'actual' query failed: SELECT row_id FROM test_data_mapping_smoke_char_1__39yfaz25i9 WHERE rand() = 42 OR value IS NULL
Error:    TestKuduConnectorTest>BaseConnectorTest.testCaseSensitiveDataMapping:4178->BaseConnectorTest.testDataMapping:4102->AbstractTestQueryFramework.assertQuery:302 Execution of 'actual' query failed: SELECT row_id FROM test_data_mapping_smoke_char_1__1xjxd067lj WHERE rand() = 42 OR value IS NOT NULL
Error:    TestKuduConnectorTest>BaseConnectorTest.testCaseSensitiveDataMapping:4178->BaseConnectorTest.testDataMapping:4121->AbstractTestQueryFramework.assertQuery:302 Execution of 'actual' query failed: SELECT row_id FROM test_data_mapping_smoke_varchar_1__m6ypxda0jc WHERE value = 'A' OR another_column = 'A'
Error:    TestKuduConnectorTest>BaseConnectorTest.testColumnName:3803->testColumnName:309->AbstractTestQueryFramework.assertQuery:302 Execution of 'actual' query failed: SELECT * FROM tcn_uppercase1lm55mnnun
Error:    TestKuduConnectorTest>BaseConnectorTest.testQueryLoggingCount:3693->AbstractTestQueryFramework.executeExclusively:536->BaseConnectorTest.lambda$testQueryLoggingCount$57:3694->BaseConnectorTest.lambda$testQueryLoggingCount$54:3696 lists don't have the same size expected [0] but found [1]
Error:    TestKuduConnectorTest>BaseConnectorTest.testRenameColumn:2048->AbstractTestQueryFramework.assertQuery:302 Execution of 'actual' query failed: SELECT z FROM test_rename_column_11hknh9tgs

[findepi]

@ksobolew you linked to this issue in #14933 (comment) and #15051 (comment)

did you intent to add a comment here with a link to failing builds?

@nineinchnick will #15181 produce a heap dump in "unable to create native thread: possibly out of memory or process/resource limits reached" case too?

[nineinchnick]

@findepi if there's any failure in the job, it'll try to grab any *.hprof file in any subdir and upload it as an artifact. I think you might be asking how Maven and/or the Surefire plugin runs Java, and I don't think I can answer this.

[ksobolew]

@ksobolew you linked to this issue in #14933 (comment) and #15051 (comment)

did you intent to add a comment here with a link to failing builds?

Yeah, sorry, will post the link in the future.