Add `register_table` procedure support for iceberg table #14375

krvikash · 2022-09-29T12:56:29Z

Description

Adding register_table procedure for iceberg connector to register table using existing metadata.

Iceberg table will be created using provided table location
Look for the latest metadata file ending with .metadata.json inside the metadata folder and use it for creating the table
If provided location is invalid/does not exist then throw an exception
If no metadata file exists inside provided table location then throw an exception
If more than one metadata file exists with the latest sequence number then throw an exception
schema_name, table_name, and table_location should be not-null and valid, Otherwise, the exception will be thrown
User can optionally provide the valid metadata_file_location (See the usage below)

Valid usages:

register_table(schema_name => ..., table_name => ..., table_location => ...)
register_table(schema_name => ..., table_name => ..., table_location => ..., metadata_file_location => ...)

Sample Queries:
CALL iceberg.system.register_table('default', 'src_22', 'hdfs://hadoop-master:9000/user/hive/warehouse/orders_5-581fad8517934af6be1857a903559d44');

CALL iceberg.system.register_table('default', 'src_22', 'hdfs://hadoop-master:9000/user/hive/warehouse/orders_5-581fad8517934af6be1857a903559d44', null);

CALL iceberg.system.register_table('default', 'src_22', 'hdfs://hadoop-master:9000/user/hive/warehouse/orders_5-581fad8517934af6be1857a903559d44', '00003-409702ba-4735-4645-8f14-09537cc0b2c8.metadata.json');

Test cases are added for success and failure scenarios in the following classes.

Flat File -> TestIcebergRegisterTableProcedure
AWS Glue, Minio, GCP -> BaseIcebergConnectorSmokeTest
HDFS (Spark) -> TestIcebergSparkCompatibility

Configuration:
By default register_table procedure is disabled. Enable procedure by setting iceberg.register-table-procedure.enabled to true in config.

Non-technical explanation

NA

Release notes

( ) This is not user-visible or docs only and no release notes are required.
(X) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/hms/TrinoHiveCatalog.java

alexjo2144 · 2022-09-29T15:44:27Z

The approach generally looks good to me. Thanks for putting this together

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergErrorCode.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

...no-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractIcebergTableOperations.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/hms/TrinoHiveCatalog.java

...roduct-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergSparkCompatibility.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

...no-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractIcebergTableOperations.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/hms/TrinoHiveCatalog.java

...n/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergRegisterTableProcedure.java

findinpath · 2022-10-03T07:50:17Z

...n/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergRegisterTableProcedure.java

+import static java.lang.String.format;
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class TestIcebergRegisterTableProcedure


Should we maybe integrate the tests from this class into BaseIcebergConnectorTest ?

...n/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergRegisterTableProcedure.java

...roduct-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergSparkCompatibility.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/TrinoCatalog.java

...o-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/GlueIcebergTableOperations.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java

...in/trino-iceberg/src/main/java/io/trino/plugin/iceberg/procedure/RegisterTableProcedure.java

testing/trino-testing-containers/src/main/java/io/trino/testing/minio/MinioClient.java

...roduct-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergSparkCompatibility.java

.../trino-product-tests/src/main/java/io/trino/tests/product/iceberg/util/IcebergTestUtils.java

alexjo2144

Just some documentation wording nit picks 👍

docs/src/main/sphinx/connector/iceberg.rst

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java

mosabua · 2022-11-10T02:23:10Z

Congratulations @krvikash .. well done on persisting through and working with all the reviewers towards a successful merge. Great collaboration everyone!

krvikash · 2022-11-10T07:26:49Z

Congratulations @krvikash .. well done on persisting through and working with all the reviewers towards a successful merge. Great collaboration everyone!

Thanks @mosabua :)

cla-bot bot added the cla-signed label Sep 29, 2022

krvikash requested review from findinpath, findepi and alexjo2144 September 29, 2022 12:56

krvikash changed the title ~~Iceberg support register table procedure~~ Iceberg support registertable procedure Sep 29, 2022

krvikash changed the title ~~Iceberg support registertable procedure~~ Add register_table procedure support for iceberg table Sep 29, 2022

github-actions bot added the tests:hive label Sep 29, 2022

krvikash force-pushed the iceberg-support-register-table-procedure branch from 6e09f17 to e464f17 Compare September 29, 2022 14:34

krvikash self-assigned this Sep 29, 2022

alexjo2144 reviewed Sep 29, 2022

View reviewed changes

krvikash force-pushed the iceberg-support-register-table-procedure branch 2 times, most recently from f9fca42 to 01679f5 Compare September 29, 2022 18:26

martint force-pushed the master branch from 8558015 to 6912e92 Compare September 29, 2022 19:09

krvikash force-pushed the iceberg-support-register-table-procedure branch from 01679f5 to 1203740 Compare September 29, 2022 22:06

This was referenced Sep 30, 2022

Add register_table procedure support for iceberg using existing metadata #14325

Closed

Add support of creating iceberg table with existing data #14273

Closed

findinpath reviewed Sep 30, 2022

View reviewed changes

krvikash marked this pull request as ready for review September 30, 2022 08:50

findepi reviewed Sep 30, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java Outdated Show resolved Hide resolved

krvikash force-pushed the iceberg-support-register-table-procedure branch 6 times, most recently from 57a7092 to 9ddb33d Compare October 1, 2022 15:23

findinpath reviewed Oct 3, 2022

View reviewed changes

findepi reviewed Oct 3, 2022

View reviewed changes

krvikash force-pushed the iceberg-support-register-table-procedure branch 2 times, most recently from 0308ec8 to 034538f Compare October 4, 2022 08:09

krvikash force-pushed the iceberg-support-register-table-procedure branch from 70a8a52 to 0f59c9f Compare November 7, 2022 11:59

findepi reviewed Nov 7, 2022

View reviewed changes

krvikash force-pushed the iceberg-support-register-table-procedure branch from 0f59c9f to a278ee1 Compare November 7, 2022 13:22

findepi approved these changes Nov 7, 2022

View reviewed changes

krvikash mentioned this pull request Nov 7, 2022

Remove unnecessary test cases from BaseIcebergMinioConnectorSmokeTest of register_table procedure #14934

Closed

krvikash force-pushed the iceberg-support-register-table-procedure branch 6 times, most recently from 8a18f3f to f4affd3 Compare November 8, 2022 06:41

findinpath reviewed Nov 8, 2022

View reviewed changes

testing/trino-testing-containers/src/main/java/io/trino/testing/minio/MinioClient.java Outdated Show resolved Hide resolved

findinpath reviewed Nov 8, 2022

View reviewed changes

...roduct-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergSparkCompatibility.java Show resolved Hide resolved

findinpath reviewed Nov 8, 2022

View reviewed changes

.../trino-product-tests/src/main/java/io/trino/tests/product/iceberg/util/IcebergTestUtils.java Outdated Show resolved Hide resolved

Add register_table procedure support for iceberg table

e20ce1d

krvikash force-pushed the iceberg-support-register-table-procedure branch 2 times, most recently from 55d8d5c to 8058841 Compare November 8, 2022 11:27

findinpath approved these changes Nov 8, 2022

View reviewed changes

alexjo2144 approved these changes Nov 8, 2022

View reviewed changes

docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java Show resolved Hide resolved

Document register_table procedure support for iceberg table

cfcfbd0

krvikash force-pushed the iceberg-support-register-table-procedure branch from 8058841 to cfcfbd0 Compare November 8, 2022 15:41

krvikash mentioned this pull request Nov 9, 2022

Let GlueIcebergUtil.getTableInput populate the table type property #14967

Closed

findepi merged commit ce68ae9 into trinodb:master Nov 9, 2022

findepi mentioned this pull request Nov 9, 2022

Release notes for 403 #14878

Closed

github-actions bot added this to the 403 milestone Nov 9, 2022

colebow mentioned this pull request Nov 9, 2022

Add Trino 403 release notes #14977

Merged

krvikash mentioned this pull request Dec 22, 2022

Add register_table and unregister_table procedures in Iceberg REST catalog #15512

Closed

krvikash deleted the iceberg-support-register-table-procedure branch April 30, 2023 08:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `register_table` procedure support for iceberg table #14375

Add `register_table` procedure support for iceberg table #14375

krvikash commented Sep 29, 2022 •

edited

Loading

alexjo2144 commented Sep 29, 2022

findinpath Oct 3, 2022

alexjo2144 left a comment

mosabua commented Nov 10, 2022

krvikash commented Nov 10, 2022

Add register_table procedure support for iceberg table #14375

Add register_table procedure support for iceberg table #14375

Conversation

krvikash commented Sep 29, 2022 • edited Loading

Description

Non-technical explanation

Release notes

alexjo2144 commented Sep 29, 2022

findinpath Oct 3, 2022

Choose a reason for hiding this comment

alexjo2144 left a comment

Choose a reason for hiding this comment

mosabua commented Nov 10, 2022

krvikash commented Nov 10, 2022

Add `register_table` procedure support for iceberg table #14375

Add `register_table` procedure support for iceberg table #14375

krvikash commented Sep 29, 2022 •

edited

Loading