-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Glue catalog in Iceberg connector #10845
Support Glue catalog in Iceberg connector #10845
Conversation
@alexjo2144 can you please squash commits before re-review? it's been a while, so i need to start reading anew anyway. |
250a301
to
ae22437
Compare
Squashed. I left a copy of the history on a separate branch in case it's useful to see what I changed from Jack's initial PR https://github.com/alexjo2144/trino/tree/alexjo/glue-iceberg-catalog-full-history |
@findepi I took a stab at adapted Jack's TestIcebergGlueConnectorSmokeTest to work on CI. PTAL |
8487560
to
1ca1f73
Compare
695d596
to
06ff7e6
Compare
Pushed another commit with some more Catalog tests, and one which should fix #10902 |
I am still working on the inverse problem, where reading the Iceberg Glue information schema fails if Hive tables are present. I'll try to have that fix and a test up later today. |
188868b
to
9196e0f
Compare
Just pushed an update for that in the last commit along with some tests for that case. The fixups were getting a little out of hand so I did squash them. |
One other thing I noticed while I was writing these tests though. |
...o-hive/src/main/java/io/trino/plugin/hive/metastore/glue/converter/GlueToTrinoConverter.java
Outdated
Show resolved
Hide resolved
5af448e
to
3d88427
Compare
@findepi I pushed an update with a fix for table redirections, and some tests with a Glue metastore that has Hive and Iceberg tables in the same namespace. |
plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java
Outdated
Show resolved
Hide resolved
...o-hive/src/main/java/io/trino/plugin/hive/metastore/glue/converter/GlueToTrinoConverter.java
Outdated
Show resolved
Hide resolved
...o-hive/src/main/java/io/trino/plugin/hive/metastore/glue/converter/GlueToTrinoConverter.java
Outdated
Show resolved
Hide resolved
...o-hive/src/main/java/io/trino/plugin/hive/metastore/glue/converter/GlueToTrinoConverter.java
Outdated
Show resolved
Hide resolved
...o-hive/src/main/java/io/trino/plugin/hive/metastore/glue/converter/GlueToTrinoConverter.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestTrinoHiveCatalog.java
Outdated
Show resolved
Hide resolved
} | ||
|
||
@Test | ||
public void testDefaultLocation() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this be in BaseTestTrinoCatalog
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This tests the hive.metastore.glue.default-warehouse-dir
config, which is Glue specific
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestTrinoHiveCatalog.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/TrinoCatalog.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java
Outdated
Show resolved
Hide resolved
@alexjo2144 can you please extract Handle missing StorageDescriptor in Hive Glue tables to a separate PR? doing otherwise would cause breakage for the master CI runs, right? |
@alexjo2144 Thanks, we are waiting to see this feature released. Can you please help with when is this going to be available? |
Tried to use this patch with the latest 371 version. But failed with the following error, the same can be seen in mvn tests as well.
Any workaround? |
3d88427
to
682f3c7
Compare
Pushed a fixup with a small test fix related to @electrum 's changes |
d78ab73
to
3a12e01
Compare
(squashed) |
From a users perspective (and as mentioned in the linked documentation review) the choice of "catalog type" is not really logical and also inconsistent with the Hive connector (and maybe others). That configuration only sets the metastore and it therefore should be called as such. iceberg.metastore .. and NOT catalog.type. See https://trino.io/docs/current/connector/hive.html#metastore-configuration-properties And then we can also namespace the specific properties nicely.. iceberg.metastore.glue.* and maybe iceberg.metastore.thrift (or hms or hive) And btw.. sorry for chiming in late .. I just saw the doc PR now.. |
3a12e01
to
cf36012
Compare
Co-authored-by: Alex <[email protected]>
cf36012
to
30ab6ac
Compare
Hm .. very interesting @findepi .. thanks for the link. Its a bit unfortunate that catalog type is overloaded as a term in Iceberg .. but I guess that is out of our control... |
Authored by @jackye1995, based off of #10151
I still need to address the testing and figure out what is possible to run on the Trino automation.