Check all queries returned memory on query runner teardown #11102

losipiuk · 2022-02-18T13:57:28Z

Description

Check all queries returned memory on query runner teardown

Is this change a fix, improvement, new feature, refactoring, or other?

testing improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

tests

Related issues, pull requests, and links

fixes: #11093

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

findepi · 2022-02-18T15:36:23Z

testing/trino-testing/src/main/java/io/trino/testing/DistributedQueryRunner.java

+    private void closeCloserSuppressing(Throwable exception)
+    {
+        try {
+            closer.close();
+        }
+        catch (IOException e) {
+            exception.addSuppressed(e);
+        }
+    }


io.airlift.testing.Closeables#closeAllSuppress ?

findepi · 2022-02-18T15:39:08Z

testing/trino-testing/src/main/java/io/trino/testing/DistributedQueryRunner.java

@@ -583,6 +586,15 @@ public void loadExchangeManager(String name, Map<String, String> properties)
    public final void close()
    {
        cancelAllQueries();
+
+        try {
+            checkQueryMemoryReleased();


close() it not an awesome place to assert.

we could perhaps have a @AfterMethod check in TestDistributedEngineOnlyQueries, using queryRunner.getExclusiveLock()?

@AfterMethod might not work great as we run tests in multiple threads. How about putting it in the AbstractTestQueryFramework#close annotated with @AfterClass? (or maybe have a dedicated @AfterClass method that does just that?)

Yeah - I agree place is not great.
TestDistributedEngineOnlyQueries is not the only place we want this. Specifically, we want to test in Test*QueryFailureRecoveryTest.

@AfterClass feels less invasive. Having @AfterMethod with lock may make test execution significantly longer, due to threads waiting one for another.
Not sure we can have two @AfterClass methods (would would be the ordering). But putting that in AbstractTestQueryFramework.close() looks good to me.
There will still be some places where query runner is constructed directly (not via AbstractTestQueryFramework) which will not be subject to validation - but we can live with that I guess.

Having @AfterMethod with lock may make test execution significantly longer, due to threads waiting one for another.

i thought about that, but was more optimistic.
is it hard to measure?

note that pools should be empty immediately after query finishes, so the "assert eventually" should end real quick

But putting that in AbstractTestQueryFramework.close() looks good to me.

Looks good to me too, let's do this

arhimondr

LGTM % comments

arhimondr · 2022-02-18T17:03:16Z

testing/trino-testing/src/main/java/io/trino/testing/DistributedQueryRunner.java

@@ -583,6 +586,15 @@ public void loadExchangeManager(String name, Map<String, String> properties)
    public final void close()
    {
        cancelAllQueries();
+
+        try {
+            checkQueryMemoryReleased();


@AfterMethod might not work great as we run tests in multiple threads. How about putting it in the AbstractTestQueryFramework#close annotated with @AfterClass? (or maybe have a dedicated @AfterClass method that does just that?)

arhimondr · 2022-02-18T17:03:55Z

testing/trino-testing/src/main/java/io/trino/testing/DistributedQueryRunner.java

+            closer.close();
+        }
+        catch (IOException e) {
+            exception.addSuppressed(e);


if(exception != e) as self suppression is not allowed

Not needed as exception cannot be IOException (IOException is checked).

losipiuk · 2022-02-21T08:41:14Z

I temporarily cherry-picked #11097 here. Should fix the CI failures.

arhimondr · 2022-02-22T15:31:24Z

testing/trino-testing/src/main/java/io/trino/testing/AbstractTestQueryFramework.java

+                throw e;
+            }
+
+            afterClassCloser.close();


Should this go to finally?

no. It would mess up exception in case closer would throw. I want to get exception from checkQueryMemoryReleased if there is one.

It's still feels like it is worth calling close. If close is not called the in memory cluster will remain active while the testing framework is running other test cases. Maybe put it in finally and wrap it in a try-catch?

i agree. afterClassCloser must be invoked even if checkQueryMemoryReleased fals.

But it is. In catch clause

right, missed that.
you could also use try() for same effect

true. Wil change

You meant like that?

I totally missed that the close was called. The current state looks good to me.

findepi · 2022-02-22T20:35:35Z

testing/trino-testing/src/main/java/io/trino/testing/AbstractTestQueryFramework.java

+                throw e;
+            }
+
+            afterClassCloser.close();


i agree. afterClassCloser must be invoked even if checkQueryMemoryReleased fals.

findepi · 2022-02-23T09:35:10Z

testing/trino-testing/src/main/java/io/trino/testing/AbstractTestQueryFramework.java

-        h2QueryRunner = null;
-        sqlParser = null;
-        queryAssertions = null;
+        try (AutoCloseable ignored = afterClassCloser) {


try (afterClassCloser)

Wow. 🤯 . Was that always possible?

No, not always. Initially wasn't.

findepi · 2022-03-02T09:51:09Z

Follow-up issue: #11275

cla-bot bot added the cla-signed label Feb 18, 2022

losipiuk requested review from findepi and arhimondr February 18, 2022 13:57

losipiuk force-pushed the lo/dist-que-run-check-pools branch from ae531ba to ee82260 Compare February 18, 2022 13:59

findepi reviewed Feb 18, 2022

View reviewed changes

arhimondr approved these changes Feb 18, 2022

View reviewed changes

losipiuk force-pushed the lo/dist-que-run-check-pools branch from ee82260 to 7e81ac7 Compare February 19, 2022 15:50

losipiuk requested a review from findepi February 21, 2022 08:38

losipiuk force-pushed the lo/dist-que-run-check-pools branch from 7e81ac7 to 7bc06f9 Compare February 21, 2022 08:40

arhimondr approved these changes Feb 22, 2022

View reviewed changes

losipiuk force-pushed the lo/dist-que-run-check-pools branch from 7bc06f9 to 7793b00 Compare February 22, 2022 16:23

findepi approved these changes Feb 22, 2022

View reviewed changes

losipiuk force-pushed the lo/dist-que-run-check-pools branch from 7793b00 to 1724f93 Compare February 22, 2022 20:43

findepi reviewed Feb 23, 2022

View reviewed changes

Check all queries returned memory on query runner teardown

a971cac

losipiuk force-pushed the lo/dist-que-run-check-pools branch from 1724f93 to a971cac Compare February 23, 2022 14:25

losipiuk merged commit 173402e into trinodb:master Feb 23, 2022

mosabua mentioned this pull request Feb 23, 2022

Add Trino 372 release notes #11094

Merged

github-actions bot added this to the 372 milestone Feb 23, 2022

findepi mentioned this pull request Mar 2, 2022

Flaky tests when asserting memory pools are empty upon QueryRunner termination in integration tests #11275

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check all queries returned memory on query runner teardown #11102

Check all queries returned memory on query runner teardown #11102

losipiuk commented Feb 18, 2022

findepi Feb 18, 2022

findepi Feb 18, 2022

arhimondr Feb 18, 2022

losipiuk Feb 18, 2022

losipiuk Feb 18, 2022

findepi Feb 21, 2022

findepi Feb 21, 2022

arhimondr left a comment

arhimondr Feb 18, 2022

arhimondr Feb 18, 2022

losipiuk Feb 18, 2022

losipiuk commented Feb 21, 2022

arhimondr Feb 22, 2022

losipiuk Feb 22, 2022

arhimondr Feb 22, 2022

findepi Feb 22, 2022

losipiuk Feb 22, 2022

findepi Feb 22, 2022

losipiuk Feb 22, 2022

losipiuk Feb 22, 2022

arhimondr Feb 22, 2022

findepi Feb 22, 2022

findepi Feb 23, 2022

losipiuk Feb 23, 2022

findepi Mar 2, 2022

findepi commented Mar 2, 2022

Check all queries returned memory on query runner teardown #11102

Check all queries returned memory on query runner teardown #11102

Conversation

losipiuk commented Feb 18, 2022

Description

Related issues, pull requests, and links

Documentation

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arhimondr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

losipiuk commented Feb 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Mar 2, 2022