Add support to redirect table reads from Hive to Iceberg #10173

findepi · 2021-12-03T16:50:15Z

Hive Connector redirects Iceberg table access to the configured
Iceberg catalog.

This change adds implementation of HiveTableRedirectionsProvider,
plugging in Hive->Iceberg redirects and leveraging existing framework.

This is based on work from several authors. In particular, this is based
on #8340 with conflicts resolved, and retrofit to current APIs in Hive connector.

As tests show, not all the statements support redirects properly and
this needs to be followed up upon.

findepi · 2021-12-03T19:25:23Z

per @ssheikin's #8340 (comment), added @MiguelWeezardo as co-author.

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveConfig.java

losipiuk · 2021-12-06T07:52:14Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveWriterFactory.java

+                .map(propertyMetadata -> immutableEntry(
+                        propertyMetadata.getName(),
+                        session.getProperty(propertyMetadata.getName(), propertyMetadata.getJavaType())))
+                // The session properties collected here are used for events only. Filter out nulls to avoid problems with downstream consumers


Is there a reason why we do not care about properties which are set to null in events?

I am not aware of any consumer of these events. This must be some extension point for someone.
The only usage i found looks like a dummy consumer, at least this is what the commit message says.

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveEventClient.java

Lines 31 to 38 in cea5842

log.debug("File created: query: %s, schema: %s, table: %s, partition: '%s', format: %s, size: %s, path: %s",

writeCompletedEvent.getQueryId(),

writeCompletedEvent.getSchemaName(),

writeCompletedEvent.getTableName(),

writeCompletedEvent.getPartitionName(),

writeCompletedEvent.getStorageFormat(),

writeCompletedEvent.getBytes(),

writeCompletedEvent.getPath());

Including null values could be a breaking change to downstream even consumers (e.g. if the [Immutable]Map.copyOf(event.getSessionProperties()). Converting values to Optional would be a breaking change to. In the absence of more information, i deemed that omitting null values is the least breaking of all options.

losipiuk · 2021-12-06T07:53:24Z

...a/io/trino/tests/product/launcher/env/environment/EnvSinglenodeHiveRedirectionToIceberg.java

+import static org.testcontainers.utility.MountableFile.forHostPath;
+
+@TestsEnvironment
+public class EnvSinglenodeHiveRedirectionToIceberg


This probably does not matter here but I heard many times in the past that we should refrain from using singlenode envs as they provide worse test coverage than multinode ones.

Thanks for reminding me. We have 27 or something singlenode environments. And yes, here it doesn't matter, as tested functionality is really coordinator-only. Let me keep it as is.

...no-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveRedirectionToIceberg.java

losipiuk · 2021-12-06T08:09:31Z

...no-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveRedirectionToIceberg.java

+
+        onTrino().executeQuery("ALTER TABLE " + hiveTableName + " ADD COLUMN some_new_column double");
+
+        // TODO: ALTER TABLE succeeded, but new column was not added


can you create a github issue which lists missing functionalities?
And tag relevant TODOs with an issue

I don't think it's necessary in this case. #8340 is going to fix this problems (once #8340 (comment) is resolved).

hashhar

Do we want to test SHOW STATS redirection too?

What about a non-default hive.timestamp-precision? Redirects will return different results.
Same for the use_column_name configs in tables which have different schema across partitions.

findepi · 2021-12-06T10:33:58Z

What about a non-default hive.timestamp-precision? Redirects will return different results.

hive.timestamp-precision doesn't matter, since there is no Hive table. Hive doesn't know the table schema, so it doesn't know whether there are any timestamps.

Same for the use_column_name configs in tables which have different schema across partitions.

same

Hive Connector redirects Iceberg table access to the configured Iceberg catalog. This change adds implementation of `HiveTableRedirectionsProvider`, plugging in Hive->Iceberg redirects and leveraging existing framework. This is based on work from several authors, mentioned at the end of commit message. As tests show, not all the statements support redirects properly and this needs to be followed up upon. Co-authored-by: Xingyuan Lin <[email protected]> Co-authored-by: Pratham Desai <[email protected]> Co-authored-by: Sasha Sheikin <[email protected]> Co-authored-by: Michał Ślizak <[email protected]> Co-authored-by: Łukasz Osipiuk <[email protected]>

findepi · 2021-12-06T11:17:46Z

Do we want to test SHOW STATS redirection too?

added

Also, split the test class into two (#10173 (comment)).

aierate · 2022-10-27T12:44:05Z

Do I have to use like
USE a-catalog.myschema;
SELECT * FROM mytable;
or
SELECT * FROM a-catalog.myschema.mytable;
?
Instead of using SELECT * FROM myschema.mytable without using a-catalog or USE a-catalog?

findepi added enhancement New feature or request tests:hive labels Dec 3, 2021

findepi requested review from losipiuk, phd3, alexjo2144 and hashhar December 3, 2021 16:50

cla-bot bot added the cla-signed label Dec 3, 2021

findepi force-pushed the findepi/redirect-hive-iceberg/there-and-back-again branch from f4a3f99 to 708cdf5 Compare December 3, 2021 16:52

findepi mentioned this pull request Dec 3, 2021

Add support to redirect table reads from Hive to Iceberg #8340

Closed

findepi force-pushed the findepi/redirect-hive-iceberg/there-and-back-again branch from 708cdf5 to d12502a Compare December 3, 2021 19:24

findepi force-pushed the findepi/redirect-hive-iceberg/there-and-back-again branch from ceae60d to 7b9d7ba Compare December 4, 2021 11:24