ALTER TABLE EXECUTE + Hive OPTIMIZE (explicit TableExecuteNode) #9665

losipiuk · 2021-10-16T19:53:45Z

PR adds SPI and execution support for new table procedures.

New syntax extends ALTER TABLE family allowing for proper semantic
analysis of target table (unlike what we have with CALL when table
schema and table name are passed as string literals via table
arguments).

New syntax example:

ALTER TABLE <table> EXECUTE procedure
ALTER TABLE <table> EXECUTE procedure(value1, value2, ...)
ALTER TABLE <table> EXECUTE procedure(param1 => value1, param2 => value2, ...) WHERE ...

New table procedures allow for rewriting table data which makes them
feasible for implementing data cleansing routines like:

compacting small files into larger ones for HIVE table
changing files sorting or bucketing scheme for a table
Iceberg OPTIMIZE

Currently exectuion flow which does not rewrite table data is not
implemented. It will be implemented as a followup. Then current
procedures available via CALL will be migrated to new mechanism.

Procedures are exposed via connectors to engine via a set of new SPI
methods and classes:

io.trino.spi.connector.TableProcedureMetadata
io.trino.spi.connector.TableProcedureExecutionMode
io.trino.spi.connector.ConnectorTableExecuteHandle
io.trino.spi.connector.ConnectorMetadata#getTableHandleForExecute
io.trino.spi.connector.ConnectorMetadata#getLayoutForTableExecute
io.trino.spi.connector.ConnectorMetadata#beginTableExecute
io.trino.spi.connector.ConnectorMetadata#finishTableExecute

findepi

lots of editorials, nothing material. good job

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/BeginTableWrite.java

findepi · 2021-10-19T09:07:25Z

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/BeginTableWrite.java

-                PlanNode source = rewriteModifyTableScan(((SemiJoinNode) node).getSource(), handle);
-                return replaceChildren(node, ImmutableList.of(source, ((SemiJoinNode) node).getFilteringSource()));
-            }
-            if (node instanceof JoinNode) {
-                PlanNode source = rewriteModifyTableScan(((JoinNode) node).getLeft(), handle);
-                return replaceChildren(node, ImmutableList.of(source, ((JoinNode) node).getRight()));


Previously we required that updated TS is on the left side of Join / SemiJoin.
I think this is an important property. I don't imagine how we could get a wrong SemiJoin, but getting flipped Join is certainly possible. cc @kasiafi
Is it still validated somewhere? or are we going to hit

trino/core/trino-main/src/main/java/io/trino/operator/AbstractRowChangeOperator.java

Line 161 in 7c1a277

checkState(source.isPresent(), "UpdatablePageSource not set");

?

Actually that is still enforced by logic in findTableScanHandleForDeleteOrUpdate

core/trino-spi/src/main/java/io/trino/spi/connector/BeginTableExecuteResult.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveTableExecuteHandle.java

plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveConnectorTest.java

core/trino-main/src/main/java/io/trino/sql/planner/iterative/rule/RemoveEmptyTableExecute.java

martint · 2021-10-19T17:21:22Z

core/trino-main/src/main/java/io/trino/metadata/TableExecuteHandle.java

+
+import static java.util.Objects.requireNonNull;
+
+public final class TableExecuteHandle


What does this handle identify, exactly? It's not clear from the name.

I updated javadoc a bit. Hope it is better now.

core/trino-main/src/main/java/io/trino/metadata/AbstractCatalogPropertyManager.java

core/trino-parser/src/main/antlr4/io/trino/sql/parser/SqlBase.g4

martint · 2021-10-19T17:28:01Z

core/trino-main/src/main/java/io/trino/operator/TableFinishOperator.java

@@ -21,6 +21,7 @@
 import io.trino.operator.OperationTimer.OperationTiming;


This commit should be folded into whichever commit introduce execution for ALTER TABLE EXECUTE. Otherwise, that other commit would be "broken".

In fact, I would squash all the commits related to adding the new feature into a single one before it gets merged. You can leave the unrelated changes separate (as long as they have value and stand on their own)

In fact, I would squash all the commits related to adding the new feature into a single one before it gets merged

Yeah - that is the plan.

jhlodin · 2021-10-19T18:08:09Z

Docs PR here, can make additional updates as needed but otherwise waiting until this code is merged. #9682

losipiuk · 2021-10-21T10:23:00Z

AC. I will squash commits that are not self-contained later today or tomorrow. @martint please let me know if you want to dig deeper into this one.

losipiuk · 2021-10-22T14:31:29Z

Squashed implementation commmits and rebased on top of current master

Allow using different key than catalog name for storing properites metadata. This is preparatory work for introducing table procedures, where each table procedure, even from single catalog, may allow different set of properties.

Commit adds SPI and execution support for new table procedures. New syntax extends ALTER TABLE family allowing for proper semantic analysis of target table (unlike what we have with CALL when table schema and table name are passed as string literals via table arguments). New syntax example: * ALTER TABLE <table> EXECUTE * ALTER TABLE <table> EXECUTE(value1, value2, ...) * ALTER TABLE <table> EXECUTE(param1 => value1, param2 => value2, ...) WHERE ... New table procedures allow for rewriting table data which makes them feasible for implementing data cleansing routines like: * compacting small files into larger ones for HIVE table * changing files sorting or bucketing scheme for a table * Iceberg OPTIMIZE Currently exectuion flow which _does not_ rewrite table data is not implemented. It will be implemented as a followup. Then current procedures available via `CALL` will be migrated to new mechanism. Procedures are exposed via connectors to engine via a set of new SPI methods and classes: * io.trino.spi.connector.TableProcedureMetadata * io.trino.spi.connector.TableProcedureExecutionMode * io.trino.spi.connector.ConnectorTableExecuteHandle * io.trino.spi.connector.ConnectorMetadata#getTableHandleForExecute * io.trino.spi.connector.ConnectorMetadata#getLayoutForTableExecute * io.trino.spi.connector.ConnectorMetadata#beginTableExecute * io.trino.spi.connector.ConnectorMetadata#finishTableExecute

Add support for compacting small files for non-transactional, non-bucketed Hive tables. ALTER TABLE xxxxx EXECUTE OPTIMIZE WITH(file_size_threshold = ...)

cla-bot bot added the cla-signed label Oct 16, 2021

losipiuk force-pushed the lo/distributed-dml-3 branch 4 times, most recently from d2629fe to 2851668 Compare October 17, 2021 08:58

losipiuk changed the title ~~Lo/distributed dml 3~~ Draft: ALTER TABLE ANALYZE + Hive OPTIMIZE (explicit TableExecuteNode) Oct 17, 2021

losipiuk force-pushed the lo/distributed-dml-3 branch from 2851668 to 39a35ed Compare October 17, 2021 09:37

losipiuk marked this pull request as ready for review October 19, 2021 08:01

losipiuk changed the title ~~Draft: ALTER TABLE ANALYZE + Hive OPTIMIZE (explicit TableExecuteNode)~~ ALTER TABLE EXECUTE + Hive OPTIMIZE (explicit TableExecuteNode) Oct 19, 2021

losipiuk mentioned this pull request Oct 19, 2021

Add support for compacting small files for Hive tables #9398

Closed

findepi approved these changes Oct 19, 2021

View reviewed changes

martint reviewed Oct 19, 2021

View reviewed changes

losipiuk force-pushed the lo/distributed-dml-3 branch 3 times, most recently from f42d8f4 to fadee63 Compare October 21, 2021 10:21

losipiuk force-pushed the lo/distributed-dml-3 branch 2 times, most recently from de2bbc2 to aed3887 Compare October 22, 2021 14:30

losipiuk force-pushed the lo/distributed-dml-3 branch from aed3887 to 8469bd4 Compare October 22, 2021 14:32

losipiuk added 8 commits October 25, 2021 10:54

Simplfy table scan handle update in BeginTableWrite

7fdfff4

Rename method

bceb7bc

Use descriptive argument name

176773a

Generalize AbstractPropertyManager

5ed2ede

Allow using different key than catalog name for storing properites metadata. This is preparatory work for introducing table procedures, where each table procedure, even from single catalog, may allow different set of properties.

Prefer toImmutableList in HiveMetadata

fb1c883

Short-circuit empty table execute plan

2b9cdfe

Add Hive OPTIMIZE table procedure

57e48d2

Add support for compacting small files for non-transactional, non-bucketed Hive tables. ALTER TABLE xxxxx EXECUTE OPTIMIZE WITH(file_size_threshold = ...)

losipiuk force-pushed the lo/distributed-dml-3 branch from 8469bd4 to 57e48d2 Compare October 25, 2021 08:56

losipiuk merged commit 2fe7084 into trinodb:master Oct 25, 2021

github-actions bot added this to the 364 milestone Oct 25, 2021

losipiuk mentioned this pull request Oct 26, 2021

Release notes for 364 #9534

Closed

12 tasks

findepi mentioned this pull request Jun 7, 2022

Support distributed table-scoped connector-provided procedures #9466

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ALTER TABLE EXECUTE + Hive OPTIMIZE (explicit TableExecuteNode) #9665

ALTER TABLE EXECUTE + Hive OPTIMIZE (explicit TableExecuteNode) #9665

losipiuk commented Oct 16, 2021 •

edited

Loading

findepi left a comment

findepi Oct 19, 2021

losipiuk Oct 21, 2021

martint Oct 19, 2021

losipiuk Oct 20, 2021

martint Oct 19, 2021

losipiuk Oct 19, 2021

jhlodin commented Oct 19, 2021

losipiuk commented Oct 21, 2021

losipiuk commented Oct 22, 2021


		import static java.util.Objects.requireNonNull;

		public final class TableExecuteHandle

		@@ -21,6 +21,7 @@
		import io.trino.operator.OperationTimer.OperationTiming;

ALTER TABLE EXECUTE + Hive OPTIMIZE (explicit TableExecuteNode) #9665

ALTER TABLE EXECUTE + Hive OPTIMIZE (explicit TableExecuteNode) #9665

Conversation

losipiuk commented Oct 16, 2021 • edited Loading

findepi left a comment

Choose a reason for hiding this comment

findepi Oct 19, 2021

Choose a reason for hiding this comment

losipiuk Oct 21, 2021

Choose a reason for hiding this comment

martint Oct 19, 2021

Choose a reason for hiding this comment

losipiuk Oct 20, 2021

Choose a reason for hiding this comment

martint Oct 19, 2021

Choose a reason for hiding this comment

losipiuk Oct 19, 2021

Choose a reason for hiding this comment

jhlodin commented Oct 19, 2021

losipiuk commented Oct 21, 2021

losipiuk commented Oct 22, 2021

losipiuk commented Oct 16, 2021 •

edited

Loading