TestDistributedFaultTolerantEngineOnlyQueries takes longer time to execute

[Praveen2112]

It ends up in crossing the time limit for trino-test module (60 min) in GHA.

2022-08-04T09:40:03.8940316Z 	io.trino.tests.TestDistributedFaultTolerantEngineOnlyQueries running for 27.43m
2022-08-04T09:40:03.8940904Z 	io.trino.tests.TestDistributedFaultTolerantEngineOnlyQueries.testScalarSubquery running for 25.44m.

[Praveen2112]

https://github.com/trinodb/trino/runs/7668645829?check_suite_focus=true

[ebyhr]

TestHiveFaultTolerantExecutionJoinQueries also took longer time and caused timed out.

2022-08-05T08:45:04.3200406Z 	io.trino.faulttolerant.hive.TestHiveFaultTolerantExecutionJoinQueries.testOutputDuplicatesInsensitiveJoin running for 17.34m
2022-08-05T08:45:04.3201201Z 	io.trino.faulttolerant.hive.TestHiveFaultTolerantExecutionJoinQueries running for 18.98m.

https://github.com/trinodb/trino/runs/7687591861

[ebyhr]

TestHiveFaultTolerantExecutionJoinQueries.testOutputDuplicatesInsensitiveJoin
https://github.com/trinodb/trino/runs/7689410625

[Praveen2112]

TestDistributedFaultTolerantEngineOnlyQueries.testScalarSubquery
https://github.com/trinodb/trino/runs/7725553971

[raunaqmorarka]

cc: @arhimondr

[arhimondr]

Looking

[raunaqmorarka]

https://github.com/trinodb/trino/runs/7763457547?check_suite_focus=true

2022-08-10T06:07:37.073-0500	WARN	TestHangMonitor	io.trino.testng.services.LogTestDurationListener	No test started or completed in 8.00m. Running tests:
	io.trino.faulttolerant.hive.TestHiveFaultTolerantExecutionJoinQueries.testOutputDuplicatesInsensitiveJoin running for 18.19m
	io.trino.faulttolerant.hive.TestHiveFaultTolerantExecutionJoinQueries running for 20.00m.

[findepi]

It seems TestDistributedFaultTolerantEngineOnlyQueries.testScalarSubquery can take over 45 minutes

2022-08-11T10:28:35.1938505Z 2022-08-11T05:28:35.179-0500	WARN	TestHangMonitor	io.trino.testng.services.LogTestDurationListener	No test started or completed in 8.00m. Running tests:
2022-08-11T10:28:35.1939236Z 	io.trino.tests.TestDistributedFaultTolerantEngineOnlyQueries running for 49.29m
2022-08-11T10:28:35.1940018Z 	io.trino.tests.TestDistributedFaultTolerantEngineOnlyQueries.testScalarSubquery running for 45.59m.

https://github.com/trinodb/trino/runs/7783779560?check_suite_focus=true

[findepi]

Locally the test finishes in one minute on my laptop, so 45 minutes on the CI looks suspicious.

[sopel39]

I've seen another occurrence.

[arhimondr]

I've run both faulty tests for thousands of times over night and haven't been able to reproduce. Still working on it.

[arhimondr]

Fix: #13695

[raunaqmorarka]

Fixed by #13695