Disallow querying iceberg tables in hive #10441

findinpath · 2021-12-31T14:09:30Z

Hive connector cannot read from Iceberg tables reason
why querying such tables shouldn't be permitted within
the hive connector.
In case of trying to query an Iceberg table from the
hive connector provide a meaningful message to the user
about not supporting such an operation.

In case that the hive users don't want to see at all
the Iceberg tables, the property hive.hide-iceberg-tables
can be set for the hive connector to true.

Fixes #8693

docs/src/main/sphinx/connector/hive.rst

findinpath · 2021-12-31T14:11:23Z

plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/TestHive.java

+                        " <\\[\\1]>\n" +
+                        "but found.*");
+
+        throw new SkipException("not supported");


I found the same model also in the testHideDeltaLakeTables() and I am wondering whether it makes sense to do the skipping at the end of the method.
Can anyone explain the reasoning behind this ?

the assertion is so we do not forget to "uncomment" the test when we fix the tested code.
But actually why is this test supposed to fail?

seems testDisallowQueryingOfIcebergTables is expected to pass, so why the override?

findinpath · 2021-12-31T14:12:41Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/MetastoreConfig.java

    @Config("hive.hide-delta-lake-tables")
    @ConfigDescription("Hide Delta Lake tables in table listings")
    public MetastoreConfig setHideDeltaLakeTables(boolean hideDeltaLakeTables)
    {
        this.hideDeltaLakeTables = hideDeltaLakeTables;
        return this;
    }
+
+    @Config("hive.hide-iceberg-tables")


I'm not sure whether the strategy of using hive.hide-xxx-tables properties is that good.
Would it make sense to rather offer a generic property to hide any non-hive tables?

losipiuk · 2021-12-31T14:30:06Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHive.java

@@ -2870,6 +2870,7 @@ public void testHideDeltaLakeTables()
        ConnectorSession session = newSession();
        HiveIdentity identity = new HiveIdentity(session);
        SchemaTableName tableName = temporaryTable("trino_delta_lake_table");
+        SchemaTableName propertiesSchemaTableName = new SchemaTableName(tableName.getSchemaName(), format("%s$properties", tableName.getTableName()));


nit: propertiesTableName. Or jus tinline.

docs/src/main/sphinx/connector/hive.rst

losipiuk

First and last commit LGTM (can you extract separate PR with those?)

losipiuk · 2021-12-31T14:39:27Z

First and last commit LGTM (can you extract separate PR with those?)

Or two separate PRs?

sajjoseph · 2022-01-02T01:57:30Z

While I understand the need for this change, shouldn't this PR - #10173 - be a better solution for production use?

findinpath · 2022-01-03T05:48:49Z

While I understand the need for this change, shouldn't this PR - #10173 - be a better solution for production use?

@sajjoseph I'd say that the redirect usage is a different use case than what this PR is trying to solve.

findinpath · 2022-01-03T06:20:15Z

First and last commit LGTM (can you extract separate PR with those?)

Or two separate PRs?

I've created a separate PR for disallowing the query of $properties delta lake table in hive here:

#10447

plugin/trino-hive/src/main/java/io/trino/plugin/hive/PropertiesSystemTableProvider.java

docs/src/main/sphinx/connector/hive.rst

plugin/trino-hive/src/main/java/io/trino/plugin/hive/PropertiesSystemTableProvider.java

findinpath · 2022-01-05T05:52:50Z

As @sajjoseph (#10441 (comment)) and @findepi (#10441 (comment)) pointed out, the iceberg redirects are a better option to deal with iceberg tables from shared hive metastores.

findepi · 2022-01-05T08:37:33Z

@findinpath i think there is a misunderstanding about my comment #10441 (comment) .
I asked you to drop the hive.hide-iceberg-tables property, not to drop the PR.

We still want to properly reject Iceberg tables when redirects are not enabled, #8693
Same as we do for Delta tables (to be improved in #10447).

findepi · 2022-01-10T16:12:20Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java

@@ -436,6 +437,9 @@ public HiveTableHandle getTableHandle(ConnectorSession session, SchemaTableName
        if (isDeltaLakeTable(table)) {
            throw new TrinoException(HIVE_UNSUPPORTED_FORMAT, format("Cannot query Delta Lake table '%s'", tableName));
        }
+        if (isIcebergTable(table)) {
+            return null;


Throw "Cannot query Iceberg table" instead of making the system throw "table not found".

Unfortunately this doesn't work well with the concept of redirection recently implemented in hive.
If we throw an exception while trying to get the table handle in Hive, the workflow used for in the analyzer for getting the table handle will stop abruptly before checking for redirections.

See https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java#L1517 for reference

The current logic first checks whether we're dealing with a materialized view, then with a view and only at the end with a table (that may contain redirections).

If we throw an exception while trying to get the table handle in Hive, the workflow used for in the analyzer for getting the table handle will stop abruptly before checking for redirections.

the redirections should be consulted first.

The current logic first checks whether we're dealing with a materialized view, then with a view and only at the end with a table (that may contain redirections).

An iceberg table is not a materialized view, nor a view, so get[Materialized]View should return Optional.empty for these calls.

At the moment, when checking whether we're dealing with a materialized view or view there is subsequently made a call towards HiveMetadata#getTableHandle which would end up in throwing an exception instead of returning Optional.empty in case that an exception would be thrown (in a similar fashion as for delta lake tables).

Changing the logic of the StatementAnalyzer is something I'd avoid to do in this case for retrieving first the table redirections.

For this reason, I have opted to partly replicate the logic of HiveMetadata#getTableHandle method in the PartitionsSystemTableProvider which offers me the possibility to do a fine grained handling in case of dealing with an iceberg/delta lake table.

One inconvenient of this approach is that we have duplicated logic in HiveMetadata and in PartitionsSystemTableProvider.

Changing the logic of the StatementAnalyzer is something I'd avoid to do in this case for retrieving first the table redirections.

Agreed, this is out of question.

findepi · 2022-01-10T16:13:19Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveInMemoryMetastore.java

+    @Override
+    public void testDisallowQueryingOfIcebergTables()
+    {
+        throw new SkipException("not supported");


why disabled?

add assertThatThrownBy(super:: ...

While trying to run the test I receive

2022-01-11T03:30:02.708-0600 INFO Copying resource dir 'spark_bucketed_nation' to /var/folders/1q/y42hmc4s3yl38kp0t0q142_c0000gn/T/TestHiveInMemoryMetastore7495715074983273908 java.lang.IllegalArgumentException: Table directory does not exist at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) at io.trino.plugin.hive.metastore.thrift.InMemoryThriftMetastore.createTable(InMemoryThriftMetastore.java:190) at io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore.createTable(BridgingHiveMetastore.java:205) at io.trino.plugin.hive.AbstractTestHive.testDisallowQueryingOfIcebergTables(AbstractTestHive.java:2978) at io.trino.plugin.hive.TestHiveInMemoryMetastore.testDisallowQueryingOfIcebergTables(TestHiveInMemoryMetastore.java:68)

Which directory?
This sounds like a test setup problem. Does it mean TestHiveInMemoryMetastore cannot run a test that uses createTable?

I updated the comment to contain the missing directory path.

Note that the similar test testHideDeltaLakeTables is also containing a not supported skip exception.

https://github.com/trinodb/trino/blob/master/plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveInMemoryMetastore.java#L59-L63

plugin/trino-hive/src/main/java/io/trino/plugin/hive/PartitionsSystemTableProvider.java

core/trino-main/src/main/java/io/trino/execution/AddColumnTask.java

findinpath · 2022-01-12T09:50:54Z

...ct-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergHiveViewsCompatibility.java

@@ -75,7 +76,6 @@ public void testIcebergHiveViewsCompatibility()
                            .add(row("hive_view_qualified_hive"))


@losipiuk

.add(row("iceberg_table")) // TODO: should this be filtered out?

filtering out the entry from show tables would involve querying the properties metadata of each table.

Can I remove this TODO ?

findinpath · 2022-01-12T15:41:06Z

I've added a separate PR #10577 for extending the redirection awareness on the table tasks because the TestHiveRedirectionToIceberg tests were failing with the modification of HiveMetadata#getTableHandle() method when a Iceberg table is being queried.

The current PR has the tests from TestHiveRedirectionToIceberg currently failing. Please advise what would be the sane way to go further with this PR.

findepi · 2022-01-14T11:21:17Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/PartitionsSystemTableProvider.java

+        Table sourceTable = metadata.getMetastore()
+                .getTable(new HiveIdentity(session), sourceTableName.getSchemaName(), sourceTableName.getTableName())
+                .orElse(null);
+        if (sourceTable == null || isDeltaLakeTable(sourceTable)) {


Disallow querying Delta Lake system tables

This is not the right commit title, since it was disallowed before.

I thought to name it Disallow instead of Deny.
I considered that before it was denied - by throwing the exception and now it is disallowed. Any wording suggestions in this case?

findepi · 2022-01-14T11:22:11Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/PartitionsSystemTableProvider.java

@@ -82,7 +83,7 @@ public PartitionsSystemTableProvider(HivePartitionManager partitionManager, Type
        Table sourceTable = metadata.getMetastore()
                .getTable(new HiveIdentity(session), sourceTableName.getSchemaName(), sourceTableName.getTableName())
                .orElse(null);
-        if (sourceTable == null || isDeltaLakeTable(sourceTable)) {
+        if (sourceTable == null || isDeltaLakeTable(sourceTable) || isIcebergTable(sourceTable)) {


for follow-up, this returns empty when table not found, while PropertiesSystemTableProvider throws in such case

Actually PropertiesSystemTableProvider returns Optional.empty() as well.
This is covered by the previous test 3fb545e#diff-92c37834248d69878a49876bfa440c9f6afbee04b1363d402a04cf7fddc7bb62R2905-R2913

findepi · 2022-01-14T11:22:54Z

...no-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveRedirectionToIceberg.java

-        onTrino().executeQuery("DROP TABLE " + hiveTableName);
-        assertQueryFailure(() -> onTrino().executeQuery("TABLE " + icebergTableName))
-                .hasMessageMatching("\\QQuery failed (#\\E\\S+\\Q): line 1:1: Table '" + icebergTableName + "' does not exist");
+        //TODO restore test assertions after adding redirection awareness to the DropTableTask


do we have an issue to link to?

(nit: add space after //)

findinpath · 2022-01-14T14:00:36Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/PartitionsSystemTableProvider.java

            return Optional.empty();
        }
+        verifyOnline(sourceTableName, Optional.empty(), getProtectMode(sourceTable), sourceTable.getParameters());
+        HiveTableHandle sourceTableHandle = new HiveTableHandle(


NOTE that compared to HiveMetadata#getTableHandle I didn't include here anymore the check

// we must not allow system tables due to how permissions are checked in SystemTableAwareAccessControl if (getSourceTableNameFromSystemTable(systemTableProviders, tableName).isPresent()) { throw new TrinoException(HIVE_INVALID_METADATA, "Unexpected table present in Hive metastore: " + tableName); }

Hive connector cannot read from Delta Lake tables reason why querying system tables for such tables shouldn't be permitted within the hive connector. In case of trying to query on the hive connector the special hive tables: - $properties - $partitions on a Delta Lake table, the user will receive a table not found exception.

Hive connector cannot read from Iceberg tables reason why querying such tables shouldn't be permitted within the hive connector. In case of trying to query an Iceberg table from the hive connector (without iceberg redirection enabled) the user will receive a hive unsupported format exception. It is no longer possible to create a view in hive which selects from Iceberg. In case of trying to query on the hive connector the special hive tables: - $properties - $partitions on an Iceberg table, the user will receive a table not found exception.

findepi · 2022-01-18T14:26:28Z

The force push diff (https://github.com/trinodb/trino/compare/3fb545ed8d4c40042966ec147923987ff79208f3..22c899a87226b5636191acc3e25cfc00be641f1b) is large.
What were the actual changes here?

findinpath · 2022-01-18T14:31:25Z

@findepi I changed the assertions in the test class testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveRedirectionToIceberg.java so that the tests are green.

cla-bot bot added the cla-signed label Dec 31, 2021

findinpath requested review from findepi, losipiuk and mosabua December 31, 2021 14:09

findinpath commented Dec 31, 2021

View reviewed changes

docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

findinpath commented Dec 31, 2021

View reviewed changes

findinpath force-pushed the disallow-querying-iceberg-tables-in-hive branch from 8b30733 to d679601 Compare December 31, 2021 14:13

losipiuk reviewed Dec 31, 2021

View reviewed changes

docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

losipiuk reviewed Dec 31, 2021

View reviewed changes

findinpath mentioned this pull request Jan 3, 2022

Disallow querying delta lake properties table #10447

Merged

findepi requested changes Jan 4, 2022

View reviewed changes

findinpath closed this Jan 5, 2022

findinpath reopened this Jan 5, 2022

findinpath force-pushed the disallow-querying-iceberg-tables-in-hive branch from d679601 to b87a079 Compare January 10, 2022 16:10

findinpath requested review from findepi and losipiuk and removed request for mosabua January 10, 2022 16:10

findepi reviewed Jan 10, 2022

View reviewed changes

findinpath force-pushed the disallow-querying-iceberg-tables-in-hive branch 3 times, most recently from 1ec8505 to 20ba90a Compare January 12, 2022 05:57

findinpath requested a review from findepi January 12, 2022 05:57

findinpath commented Jan 12, 2022

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/PartitionsSystemTableProvider.java Outdated Show resolved Hide resolved

findinpath force-pushed the disallow-querying-iceberg-tables-in-hive branch 2 times, most recently from 96bb5b3 to 44f670f Compare January 12, 2022 09:46

findinpath commented Jan 12, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/execution/AddColumnTask.java Outdated Show resolved Hide resolved

findinpath commented Jan 12, 2022

View reviewed changes

findinpath force-pushed the disallow-querying-iceberg-tables-in-hive branch 2 times, most recently from bdafcba to 96b1e51 Compare January 14, 2022 09:24

github-actions bot added the tests:hive label Jan 14, 2022

findinpath force-pushed the disallow-querying-iceberg-tables-in-hive branch 2 times, most recently from bccfb68 to 3fb545e Compare January 14, 2022 11:13

findepi approved these changes Jan 14, 2022

View reviewed changes

findinpath commented Jan 14, 2022

View reviewed changes

findinpath added 2 commits January 18, 2022 15:08

findinpath force-pushed the disallow-querying-iceberg-tables-in-hive branch from 3fb545e to 22c899a Compare January 18, 2022 14:08

findepi merged commit 12a6b5f into trinodb:master Jan 19, 2022

github-actions bot added this to the 369 milestone Jan 19, 2022

This was referenced Jan 19, 2022

Add Trino 369 release notes #10553

Merged

Release notes for 369 #10552

Closed

findinpath mentioned this pull request Mar 9, 2022

Add support to redirect table operations from Iceberg to Hive #11356

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disallow querying iceberg tables in hive #10441

Disallow querying iceberg tables in hive #10441

findinpath commented Dec 31, 2021

findinpath Dec 31, 2021

losipiuk Dec 31, 2021

findepi Jan 10, 2022

findinpath Dec 31, 2021

losipiuk Dec 31, 2021

losipiuk left a comment

losipiuk commented Dec 31, 2021

sajjoseph commented Jan 2, 2022

findinpath commented Jan 3, 2022

findinpath commented Jan 3, 2022

findinpath commented Jan 5, 2022

findepi commented Jan 5, 2022

findepi Jan 10, 2022

findinpath Jan 10, 2022 •

edited

Loading

findepi Jan 11, 2022

findepi Jan 11, 2022

findinpath Jan 11, 2022

findinpath Jan 11, 2022

findepi Jan 14, 2022

findepi Jan 10, 2022

findinpath Jan 10, 2022 •

edited

Loading

findepi Jan 11, 2022

findinpath Jan 11, 2022

findinpath Jan 12, 2022

findinpath commented Jan 12, 2022

findepi Jan 14, 2022

findinpath Jan 14, 2022 •

edited

Loading

findepi Jan 14, 2022

findinpath Jan 14, 2022

findepi Jan 14, 2022

findinpath Jan 14, 2022

findepi commented Jan 18, 2022

findinpath commented Jan 18, 2022

		@@ -75,7 +76,6 @@ public void testIcebergHiveViewsCompatibility()
		.add(row("hive_view_qualified_hive"))

Disallow querying iceberg tables in hive #10441

Disallow querying iceberg tables in hive #10441

Conversation

findinpath commented Dec 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

losipiuk left a comment

Choose a reason for hiding this comment

losipiuk commented Dec 31, 2021

sajjoseph commented Jan 2, 2022

findinpath commented Jan 3, 2022

findinpath commented Jan 3, 2022

findinpath commented Jan 5, 2022

findepi commented Jan 5, 2022

Choose a reason for hiding this comment

findinpath Jan 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath Jan 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath commented Jan 12, 2022

Choose a reason for hiding this comment

findinpath Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Jan 18, 2022

findinpath commented Jan 18, 2022

findinpath Jan 10, 2022 •

edited

Loading

findinpath Jan 10, 2022 •

edited

Loading

findinpath Jan 14, 2022 •

edited

Loading