Track Glue API calls that were untracked #11059

homar · 2022-02-16T08:12:34Z

Description

One call to Glue was not being tracked.

Rename GlueMetastoreStats.renameTable to GlueMetastoreStats.updateTable.

General information

Is this change a fix, improvement, new feature, refactoring, or other?

a fix

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

a connector

How would you describe this change to a non-technical end user or system administrator?

It will allow to more precisely track calls to external api - Glue

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`5678`)

findepi

are you positive this is the only untracked one?

findepi · 2022-02-16T11:33:07Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

@@ -370,10 +370,10 @@ public void updateTableStatistics(String databaseName, String tableName, AcidTra
            final Map<String, String> statisticsParameters = updateStatisticsParameters(table.getParameters(), updatedStatistics.getBasicStatistics());
            tableInput.setParameters(statisticsParameters);
            table = Table.builder(table).setParameters(statisticsParameters).build();
-            glueClient.updateTable(new UpdateTableRequest()
+            stats.getReplaceTable().call(() -> glueClient.updateTable(new UpdateTableRequest()


the stats should be called "updateTable"

findepi · 2022-02-16T11:33:28Z

cc @losipiuk @alexjo2144

homar · 2022-02-16T11:35:59Z

are you positive this is the only untracked one?

I looked through other usages of glueClient and didn't spot anything else.

findepi · 2022-02-16T11:38:43Z

What about getAllDatabases and other places where we call getPaginatedResults?
or

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

Lines 429 to 433 in 262c6ec

    
           partitionUpdateRequestsFutures.add(glueClient.batchUpdatePartitionAsync(new BatchUpdatePartitionRequest() 
        
                   .withCatalogId(catalogId) 
        
                   .withDatabaseName(table.getDatabaseName()) 
        
                   .withTableName(table.getTableName()) 
        
                   .withEntries(partitionUpdateRequestsPartition)));

?

homar · 2022-02-16T11:54:36Z

What about getAllDatabases and other places where we call getPaginatedResults? or

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

Lines 429 to 433 in 262c6ec

partitionUpdateRequestsFutures.add(glueClient.batchUpdatePartitionAsync(new BatchUpdatePartitionRequest()

.withCatalogId(catalogId)

.withDatabaseName(table.getDatabaseName())

.withTableName(table.getTableName())

.withEntries(partitionUpdateRequestsPartition)));

?

getPaginatedResults is covered everywhere, just look 1 or 2 lines above

Regarding the one you pointed out, maybe I am wrong but I thought there is no good way to measure this as this is an async call so we pretty much will measure only the time of the invocation of async which is not very informative

losipiuk · 2022-02-17T13:23:21Z

that was -> that were

losipiuk

LGTM - commit message may be a bit more accurate as you are also changing counter name

losipiuk · 2022-02-17T13:40:32Z

What about getAllDatabases and other places where we call getPaginatedResults? or

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

Lines 429 to 433 in 262c6ec

partitionUpdateRequestsFutures.add(glueClient.batchUpdatePartitionAsync(new BatchUpdatePartitionRequest()

.withCatalogId(catalogId)

.withDatabaseName(table.getDatabaseName())

.withTableName(table.getTableName())

.withEntries(partitionUpdateRequestsPartition)));

?

getPaginatedResults is covered everywhere, just look 1 or 2 lines above

Regarding the one you pointed out, maybe I am wrong but I thought there is no good way to measure this as this is an async call so we pretty much will measure only the time of the invocation of async which is not very informative

You may want to add a callback on the returned future to bump statistic.
May be a follwoup.

alexjo2144 · 2022-02-17T15:50:15Z

Not strictly necessary but a possible improvement from what we have. Would it be better if getPaginatedResults took a GlueMetastoreApiStats as a parameter and tracked individual API calls rather than wrapping the whole method with one?

Seems like that might be a more helpful way of tracking those stats to me.

losipiuk · 2022-02-17T17:03:59Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

-                    .withDatabaseName(table.getDatabaseName())
-                    .withTableName(table.getTableName())
-                    .withEntries(partitionUpdateRequestsPartition)));
+                            .withCatalogId(catalogId)


there is more *async calls than just this one. We should address all together.

losipiuk · 2022-02-17T17:07:02Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueMetastoreApiStats.java

+    public void updateFailures()
+    {
+        totalFailures.update(1);
+    }
+
+    public void updateTime(long elapsedMilliseconds)
+    {
+        time.add(elapsedMilliseconds, MILLISECONDS);
+    }


replace with single method

void recordCall(executionTime, boolean failure)

losipiuk · 2022-02-17T17:09:18Z

Not strictly necessary but a possible improvement from what we have. Would it be better if getPaginatedResults took a GlueMetastoreApiStats as a parameter and tracked individual API calls rather than wrapping the whole method with one?

Seems like that might be a more helpful way of tracking those stats to me.

Agreed. Currently lots of calls is still untracked.

findepi · 2022-02-18T13:26:22Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

-                        .flatMap(List::stream)
-                        .map(com.amazonaws.services.glue.model.Database::getName)
-                        .collect(toImmutableList());
-                return databaseNames;


If you retained the variable, the diff would be readable.

(btw the IDE asks me to inline such variables, but i acutally find it useful during debugging)

findepi · 2022-02-18T13:27:16Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

+
+        public StatsRecordingAsyncHandler(GlueMetastoreStats stats, long startTimeInMillis)
+        {
+            this.stats = stats;


requireNonNull

findepi · 2022-02-18T13:27:44Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

-                        .filter(tableFilter)
-                        .map(com.amazonaws.services.glue.model.Table::getName)
-                        .collect(toImmutableList());
-                return tableNames;


same, no idea what changed

losipiuk · 2022-02-18T13:29:09Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

+        @Override
+        public void onError(Exception e)
+        {
+            stats.getBatchUpdatePartition().recordCall(System.currentTimeMillis() - startTimeInMillis, true);


You should use different GlueMetastoreApiStats instance for different places when you use StatsRecordingAsyncHandler. Not stats.getBatchUpdatePartition() all the time.
Change parametrization of StatsRecordingAsyncHandler to use GlueMetastoreApiStats instead of GlueMetastoreStats

yes definitely, I forgot to change that, thanks for catching this

losipiuk · 2022-02-21T11:50:05Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

-                        .collect(toImmutableList());
-                return databaseNames;
-            });
+            ImmutableList<String> databases = getPaginatedResults(


use List

And keep variable name

Also add a followup commit which just gets rid of variables.

losipiuk · 2022-02-21T11:53:56Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

+                            .withDatabaseName(databaseName),
+                    GetTablesRequest::setNextToken,
+                    GetTablesResult::getNextToken,
+                    stats.getGetAllTables())


I think we should merge stats couneters for getting views and tables.

Can you also rename dropDatabase stats counter tdeleteDatabase.

(you can put all the renames in single commit)

Same for renameDatabase -> updateDatabase

And dropTable -> deleteTable

batchGetPartitionAsync is not covered with stats from what I see.

dropPartition -> deletePartition

findepi

skimmed, lgtm

findepi · 2022-02-21T16:00:58Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueMetastoreStats.java

@@ -93,13 +92,6 @@ public GlueMetastoreApiStats getGetTable()
        return getTable;
    }

-    @Managed
-    @Nested
-    public GlueMetastoreApiStats getGetAllViews()


This belongs to previous commit, where the last usage was removed.

alexjo2144 · 2022-02-22T16:54:55Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

@@ -1126,4 +1130,29 @@ public void revokeTablePrivileges(String databaseName, String tableName, String
    {
        return ImmutableSet.of();
    }
+
+    static class StatsRecordingAsyncHandler<Request extends AmazonWebServiceRequest, Result>


Seems like this belongs in GlueMetastoreApiStats

Let's change it as a followup :)

cla-bot bot added the cla-signed label Feb 16, 2022

github-actions bot added the tests:hive label Feb 16, 2022

findepi reviewed Feb 16, 2022

View reviewed changes

homar closed this Feb 16, 2022

homar reopened this Feb 16, 2022

homar force-pushed the homar/track_glue_api_calls branch from ff65249 to 72d5f58 Compare February 16, 2022 12:04

losipiuk approved these changes Feb 17, 2022

View reviewed changes

homar force-pushed the homar/track_glue_api_calls branch 2 times, most recently from c2cf1ca to e739f6b Compare February 17, 2022 13:44

homar changed the title ~~Track Glue API calls that was untracked~~ Track Glue API calls that were untracked Feb 17, 2022

homar force-pushed the homar/track_glue_api_calls branch from e739f6b to a1aa458 Compare February 17, 2022 15:05

losipiuk reviewed Feb 17, 2022

View reviewed changes

homar force-pushed the homar/track_glue_api_calls branch from a1aa458 to 1fc15e3 Compare February 18, 2022 13:12

findepi reviewed Feb 18, 2022

View reviewed changes

losipiuk reviewed Feb 18, 2022

View reviewed changes

homar force-pushed the homar/track_glue_api_calls branch 2 times, most recently from edafa22 to 61df66b Compare February 21, 2022 11:02

losipiuk reviewed Feb 21, 2022

View reviewed changes

homar force-pushed the homar/track_glue_api_calls branch from 61df66b to fabf601 Compare February 21, 2022 12:43

Rename Glue API calls stats

05831d3

homar force-pushed the homar/track_glue_api_calls branch from fabf601 to 442a800 Compare February 21, 2022 14:24

findepi reviewed Feb 21, 2022

View reviewed changes

homar added 2 commits February 22, 2022 09:44

Track Glue API calls that were untracked

2aa4938

Merge getViews and getTables GlueApiStats

017d982

homar force-pushed the homar/track_glue_api_calls branch from 442a800 to 017d982 Compare February 22, 2022 08:47

alexjo2144 reviewed Feb 22, 2022

View reviewed changes

losipiuk merged commit 6293d52 into trinodb:master Feb 22, 2022

github-actions bot added this to the 372 milestone Feb 22, 2022

mosabua mentioned this pull request Feb 23, 2022

Add Trino 372 release notes #11094

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track Glue API calls that were untracked #11059

Track Glue API calls that were untracked #11059

homar commented Feb 16, 2022 •

edited by losipiuk

Loading

findepi left a comment

findepi Feb 16, 2022

findepi commented Feb 16, 2022

homar commented Feb 16, 2022

findepi commented Feb 16, 2022

homar commented Feb 16, 2022

losipiuk commented Feb 17, 2022

losipiuk left a comment

losipiuk commented Feb 17, 2022

alexjo2144 commented Feb 17, 2022

losipiuk Feb 17, 2022

losipiuk Feb 17, 2022

losipiuk commented Feb 17, 2022

findepi Feb 18, 2022

findepi Feb 18, 2022

findepi Feb 18, 2022

losipiuk Feb 18, 2022

homar Feb 18, 2022 •

edited

Loading

losipiuk Feb 21, 2022

losipiuk Feb 21, 2022

losipiuk Feb 21, 2022

losipiuk Feb 21, 2022

losipiuk Feb 21, 2022

losipiuk Feb 21, 2022

losipiuk Feb 21, 2022

losipiuk Feb 21, 2022

findepi left a comment

findepi Feb 21, 2022

alexjo2144 Feb 22, 2022

losipiuk Feb 22, 2022

Track Glue API calls that were untracked #11059

Track Glue API calls that were untracked #11059

Conversation

homar commented Feb 16, 2022 • edited by losipiuk Loading

Description

General information

Related issues, pull requests, and links

Documentation

Release notes

findepi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Feb 16, 2022

homar commented Feb 16, 2022

findepi commented Feb 16, 2022

homar commented Feb 16, 2022

losipiuk commented Feb 17, 2022

losipiuk left a comment

Choose a reason for hiding this comment

losipiuk commented Feb 17, 2022

alexjo2144 commented Feb 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

losipiuk commented Feb 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

homar Feb 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

homar commented Feb 16, 2022 •

edited by losipiuk

Loading

homar Feb 18, 2022 •

edited

Loading