Add support for the custom hive catalog to support Hive3 #21502

osscm · 2024-04-11T00:12:41Z

Description

Currently Trino is constrained to utilize the default hive catalog (referred to as "hive") when interfacing with the hive metastore. Consequently, users are confined to employing only a two-level hierarchy (schema.table) within a Trino catalog/connector. In contrast, Hive thrift allows for custom hive catalog names as the parent of the schema, enabling the utilization of a three-level hierarchy such as hive-catalog.schema.table.

Hive Thrift API supports custom catalog since Hive3
Therefore, at high level, this PR has the following changes

Customize the hive catalog name within Trino's catalog settings and transmit it via the factories to the Thrift client --> StaticMetastoreConfig , StaticTokenAwareHttpMetastoreClientFactory , StaticTokenAwareMetastoreClientFactory , ThriftMetastoreClientFactory, 'DefaultThriftMetastoreClientFactory', HttpThriftMetastoreClientFactory
Thrift client to support custom catalog: ThriftHiveMetastoreClient
There will be 1:1 mapping between Trino catalog and hive catalog.
"hive" will be the default catalog name.

Thanks to @dain @electrum @hashhar @findinpath @anusudarsan @samssh @mosabua

Additional context and related issues

Above approach was discussed at a few places, so adopted in the PR as well.
one
two

@samssh please let us know, if anything is missing or need to take care of.
Fixes #10287

co-author : @samssh, he has worked on the similar change.

Release notes

(x) Release notes are required, with the following suggested text:

# Hive
* Add support for specifying catalog name in Thrift metastore. (`#10287`)

osscm · 2024-04-17T16:27:04Z

cc @samssh

anusudarsan

skimmed and first pass

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastoreClient.java

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHttpMetastoreConfig.java

findinpath · 2024-04-18T11:24:47Z

@osscm now that you have a "ready to review" PR could you please update the description to contain the business case & concise notes including the implementation strategy contained in the code of this PR?

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastoreClient.java

anusudarsan

@osscm TestThriftHttpMetastoreClient is failing.

You need to fix https://github.com/trinodb/trino/blob/master/plugin/trino-hive/src/test/java/io/trino/plugin/hive/metastore/thrift/TestThriftHttpMetastoreClient.java#L60 as something like

 if (databaseName.equals("@hive#testDbName")) {
                    return new Database("testDbName", "testOwner", "testLocation", Map.of("key", "value"));
                }

and also TestingThriftHttpMetastoreServer to have the new method called

            case "getAllDatabases", "getDatabases" -> delegate.getAllDatabases();

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastoreClient.java

anusudarsan · 2024-04-18T21:40:03Z

@electrum can you enable test with secrets in this PR?

osscm · 2024-04-19T02:16:16Z

@osscm now that you have a "ready to review" PR could you please update the description to contain the business case & concise notes including the implementation strategy contained in the code of this PR?

thanks @findinpath
added details, please see if this make sense.

cc @anusudarsan

ebyhr · 2024-04-19T03:57:12Z

/test-with-secrets sha=e7556cb2cb2db99b03236409fcc1d1186badf430

github-actions · 2024-04-19T03:58:16Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/8748163326

anusudarsan · 2024-04-19T17:42:55Z

@osscm re: tests can we extend TestHiveMetastoreMetadataQueriesAccessOperations and have the new test set a custom catalog name ? To create the custom catalogs/namespaces in hive, it doesnt look like Hive yet supports creating catalogs via beeline CLI or SQL. We could update the metastore directly

hiveHadoop.runOnMetastore(INSERT INTO CTLGS VALUES (2, 'test_custom_catalog', 'test catalog', 'hdfs://hadoop-master:9000/user/hive/warehouse/test_custom_catalog')");

I guess you will also need to create a default schema in test_catalog using either CREATE DATABASE if hive supports it for custom catalog or directly modifying the metastore db like

hiveHadoop.runOnMetastore("INSERT INTO DBS VALUES (4, 'test default schema', 'hdfs://hadoop-master:9000/user/hive/warehouse/test_custom_catalog/default.db', 'default', 'hive', 'USER', 'test_custom_catalog')");

let me know if you hit any issues with this or need help.

@findinpath @electrum is extending TestHiveMetastoreMetadataQueriesAccessOperations good enough for this functionality, or do we need BCT as well?

osscm · 2024-04-19T19:49:55Z

fyi, will work on the pending comments today.

findinpath

LGTM % open comments

ebyhr · 2024-05-06T23:20:08Z

/test-with-secrets sha=cf153475e20161039b1e99815c414b2bf2bfafb5

github-actions · 2024-05-06T23:21:16Z

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/8977088500

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastoreClient.java

plugin/trino-hive/src/test/java/io/trino/plugin/hive/metastore/TestHiveMetastoreCatalogs.java

...-hive/src/test/java/io/trino/plugin/hive/metastore/thrift/TestThriftHttpMetastoreClient.java

docs/src/main/sphinx/object-storage/metastores.md

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastoreClient.java

...ive/src/test/java/io/trino/plugin/hive/TestHiveConnectorCustomCatalogConnectorSmokeTest.java

osscm · 2024-05-13T08:46:10Z

sorry, got engaged in other work, will try to take care of remaining comments in next a few days.

ebyhr · 2024-05-13T09:21:59Z

@osscm I pushed some changes instead of you.

osscm · 2024-05-14T05:52:08Z

@osscm I pushed some changes instead of you.

Thanks a lot @ebyhr!

ebyhr · 2024-05-15T02:52:28Z

@osscm Why did you revert my change?

osscm · 2024-05-15T04:51:25Z

@osscm Why did you revert my change?

did I revert?
Apologies! maybe by mistake while squashing in local.

ebyhr · 2024-05-16T04:02:51Z

@osscm I restored to the status before you reverted my change. Please add a fixup commit for additional changes.

osscm · 2024-05-16T16:01:57Z

@osscm I restored to the status you reverted my change. Please add a fixup commit for additional changes.

Thanks @ebyhr!

I think, already it has all the changes in! so dont need to do any changes.

two test cases are failing, checking why

test-jdbc failed --> seems to be transient?

com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"manifest for trinodb/trino:448 not found: manifest unknown: manifest unknown"}

pt (default, suite-7-non-generic, )

this one I am not sure, was not able to run in local, we can try to run build again?

2024-05-16 10:09:50 INFO: FAILURE     /    io.trino.tests.product.cli.TestTrinoCli.shouldPrintExplainAnalyzePlan (Groups: cli) took 3.7 seconds
2024-05-16T04:24:50.1411172Z tests               | 2024-05-16 10:09:50 SEVERE: Failure cause:
2024-05-16T04:24:50.1412046Z tests               | java.lang.AssertionError: 
2024-05-16T04:24:50.1412723Z tests               | Expecting ArrayList:
2024-05-16T04:24:50.1413290Z tests               |   ["",
2024-05-16T04:24:50.1414102Z tests               |     "Query 20240516_042448_00042_raqe3 [RUNNING] i[0 0B 0B] o[0 0B 0B] splits[0/0/0]",
2024-05-16T04:24:50.1415007Z tests               |     "",
2024-05-16T04:24:50.1415937Z tests               |     "Query 20240516_042448_00042_raqe3 [FAILED] i[35 3.31K 2.05K] o[35 3.31K 2.05K] splits[4/8/4]",
2024-05-16T04:24:50.1417109Z tests               |     "Query 20240516_042448_00042_raqe3, FAILED, 1 node",
2024-05-16T04:24:50.1417922Z tests               |     "Splits: 16 total, 4 done (25.00%)",
2024-05-16T04:24:50.1418669Z tests               |     "1.58 [35 rows, 3.31KB] [22 rows/s, 2.09KB/s]",
2024-05-16T04:24:50.1419308Z tests               |     "",
2024-05-16T04:24:50.1420095Z tests               |     "Query 20240516_042448_00042_raqe3 failed: Error committing write parquet to Hive"]
2024-05-16T04:24:50.1421074Z tests               | to contain:
2024-05-16T04:24:50.1421690Z tests               |   ["CREATE TABLE", "Query Plan"]
2024-05-16T04:24:50.1422472Z tests               | but could not find the following element(s):
2024-05-16T04:24:50.1423281Z tests               |   ["CREATE TABLE", "Query Plan"]
2024-05-16T04:24:50.1423913Z tests               | 
2024-05-16T04:24:50.1425259Z tests               | 	at io.trino.tests.product.cli.TestTrinoCli.shouldPrintExplainAnalyzePlan(TestTrinoCli.java:416)
2024-05-16T04:24:50.1427234Z tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-05-16T04:24:50.1428777Z tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-05-16T04:24:50.1430220Z tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
2024-05-16T04:24:50.1432046Z tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
2024-05-16T04:24:50.1433710Z tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
2024-05-16T04:24:50.1435238Z tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
2024-05-16T04:24:50.1479952Z tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
2024-05-16T04:24:50.1481452Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2024-05-16T04:24:50.1483137Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)

anusudarsan · 2024-05-17T15:18:38Z

@osscm the CI looks good now

osscm · 2024-05-17T15:41:43Z

@osscm the CI looks good now

@anusudarsan, so good to merge now? :)

@ebyhr or @dain if you can please help to merge, thanks!

cla-bot bot added the cla-signed label Apr 11, 2024

osscm marked this pull request as draft April 11, 2024 00:12

github-actions bot added tests:hive hive Hive connector labels Apr 11, 2024

osscm force-pushed the hive3thrift branch 3 times, most recently from 4dd8066 to 141b9e7 Compare April 17, 2024 07:09

osscm changed the title ~~WIP/Temp: Hive3 compatible ThriftHiveMetastore client~~ Add support for the custom hive catalog Apr 17, 2024

osscm assigned osscm and unassigned osscm Apr 17, 2024

osscm marked this pull request as ready for review April 17, 2024 16:00

This was referenced Apr 17, 2024

Add support non-default (hive) catalog for Hive Metastore 3.x or newer #10287

Closed

[WIP] add support catalog name for ThriftHiveMetastoreClient in thrift calls and fix tests #21458

Draft

anusudarsan reviewed Apr 17, 2024

View reviewed changes

osscm force-pushed the hive3thrift branch from 999ab45 to 3920283 Compare April 18, 2024 00:26

findinpath reviewed Apr 18, 2024

View reviewed changes

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastoreClient.java Outdated Show resolved Hide resolved

findinpath reviewed Apr 18, 2024

View reviewed changes

...rino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastoreClient.java Outdated Show resolved Hide resolved

osscm force-pushed the hive3thrift branch from 1d08be1 to f7e5b56 Compare April 18, 2024 17:08

findepi removed the tests:hive label Apr 18, 2024

anusudarsan reviewed Apr 18, 2024

View reviewed changes

anusudarsan requested a review from electrum April 18, 2024 21:39

osscm force-pushed the hive3thrift branch from 6ec0565 to e7556cb Compare April 18, 2024 22:29

findinpath approved these changes May 3, 2024

View reviewed changes

ebyhr reviewed May 7, 2024

View reviewed changes

anusudarsan reviewed May 8, 2024

View reviewed changes

ebyhr force-pushed the hive3thrift branch 4 times, most recently from c62f95e to f0b0a7f Compare May 13, 2024 09:20

anusudarsan approved these changes May 13, 2024

View reviewed changes

osscm force-pushed the hive3thrift branch from f0b0a7f to bcb3a2d Compare May 14, 2024 05:42

github-actions bot added iceberg Iceberg connector bigquery BigQuery connector labels May 15, 2024

osscm force-pushed the hive3thrift branch from 59e7502 to a265611 Compare May 15, 2024 07:09

Add support for the custom hive catalog

e2decbb

ebyhr force-pushed the hive3thrift branch from a265611 to e2decbb Compare May 16, 2024 04:02

dain approved these changes May 17, 2024

View reviewed changes

dain merged commit fe9b6ef into trinodb:master May 17, 2024
60 checks passed

ebyhr added this to the 449 milestone May 17, 2024

colebow mentioned this pull request May 29, 2024

Add Trino 449 release notes #22195

Merged

dain mentioned this pull request Aug 17, 2024

Trino Thrift Metastore client not support multiple catalogs #7486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for the custom hive catalog to support Hive3 #21502

Add support for the custom hive catalog to support Hive3 #21502

osscm commented Apr 11, 2024 •

edited

Loading

osscm commented Apr 17, 2024

anusudarsan left a comment

findinpath commented Apr 18, 2024

anusudarsan left a comment •

edited

Loading

anusudarsan commented Apr 18, 2024

osscm commented Apr 19, 2024

ebyhr commented Apr 19, 2024

github-actions bot commented Apr 19, 2024 •

edited

Loading

anusudarsan commented Apr 19, 2024

osscm commented Apr 19, 2024

findinpath left a comment

ebyhr commented May 6, 2024

github-actions bot commented May 6, 2024

osscm commented May 13, 2024

ebyhr commented May 13, 2024

osscm commented May 14, 2024

ebyhr commented May 15, 2024

osscm commented May 15, 2024

ebyhr commented May 16, 2024 •

edited

Loading

osscm commented May 16, 2024

anusudarsan commented May 17, 2024

osscm commented May 17, 2024

Add support for the custom hive catalog to support Hive3 #21502

Add support for the custom hive catalog to support Hive3 #21502

Conversation

osscm commented Apr 11, 2024 • edited Loading

Description

Additional context and related issues

Release notes

osscm commented Apr 17, 2024

anusudarsan left a comment

Choose a reason for hiding this comment

findinpath commented Apr 18, 2024

anusudarsan left a comment • edited Loading

Choose a reason for hiding this comment

anusudarsan commented Apr 18, 2024

osscm commented Apr 19, 2024

ebyhr commented Apr 19, 2024

github-actions bot commented Apr 19, 2024 • edited Loading

anusudarsan commented Apr 19, 2024

osscm commented Apr 19, 2024

findinpath left a comment

Choose a reason for hiding this comment

ebyhr commented May 6, 2024

github-actions bot commented May 6, 2024

osscm commented May 13, 2024

ebyhr commented May 13, 2024

osscm commented May 14, 2024

ebyhr commented May 15, 2024

osscm commented May 15, 2024

ebyhr commented May 16, 2024 • edited Loading

osscm commented May 16, 2024

anusudarsan commented May 17, 2024

osscm commented May 17, 2024

osscm commented Apr 11, 2024 •

edited

Loading

anusudarsan left a comment •

edited

Loading

github-actions bot commented Apr 19, 2024 •

edited

Loading

ebyhr commented May 16, 2024 •

edited

Loading