Support table functions with pushdown to connector #11336

kasiafi · 2022-03-04T22:25:30Z

This PR introduces partial support for Table Functions (including Polymorphic Table Functions).

Temporarily, only constant scalar arguments are supported (no Table or Descriptor arguments).
The supported execution path for a Table Function is to be pushed down into connector, and replaced with a TableHandle.

Later, Descriptor and Table arguments will be supported, and operator-based execution will be enabled for Table Functions.

kasiafi · 2022-03-14T14:42:49Z

Sadly, not all jdbc-based connectors want to cooperate. Testing all to see what they need.

martint · 2022-03-15T18:37:26Z

core/trino-parser/src/main/java/io/trino/sql/SqlFormatter.java

+                    append(indent, "");
+                }
+                Node value = argument.getValue();
+                if (value instanceof TableArgument) {


Ideally, the formatting of each argument should be done by a recursive call to format each element, instead of switching on the specific concrete classes here.

Done for all except Expression arguments. I order to be processed recursively, they would require wrapping in another AST node, which I considered too much overhead.

martint · 2022-03-15T18:43:59Z

core/trino-parser/src/main/java/io/trino/sql/parser/AstBuilder.java

+        if (context.qualifiedName() != null) {
+            table = new Table(getLocation(context.TABLE()), getQualifiedName(context.qualifiedName()));
+        }
+        else {
+            table = new TableSubquery(getLocation(context.TABLE()), (Query) visit(context.query()));


It'd be better to model these as sub-rules so that the construction occurs naturally by the recursive tree traversal instead of switching on whether qualifiedName is present or not

core/trino-spi/src/main/java/io/trino/spi/ptf/Descriptor.java

martint · 2022-04-25T21:32:04Z

core/trino-main/src/main/java/io/trino/metadata/TableFunctionRegistry.java

+        catalogTableFunctions.remove(catalogName);
+    }
+
+    public TableFunction resolve(List<CatalogSchemaRoutineName> names)


Why is the argument a list of names?

Refactored so that the list is implementation detail in table function resolution.

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

core/trino-main/src/main/java/io/trino/metadata/TableFunctionRegistry.java

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

martint · 2022-04-25T21:46:50Z

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

+                for (Argument argument : arguments) {
+                    ArgumentDeclaration declaration = declarationsByName.remove(argument.getName().get().getCanonicalValue());
+                    if (declaration == null) {
+                        throw semanticException(INVALID_FUNCTION_ARGUMENT, argument, "Unexpected argument name: ", argument.getName().get().getCanonicalValue());


There should be explicit validation to make sure the arguments provided to the function are not duplicated. This will catch that condition, but with an unclear error message.

martint · 2022-04-26T00:46:17Z

...-main/src/main/java/io/trino/sql/planner/iterative/rule/RewriteTableFunctionToTableScan.java

+    @Override
+    public Result apply(TableFunctionNode tableFunctionNode, Captures captures, Context context)
+    {
+        Optional<TableHandle> result = plannerContext.getMetadata().applyTableFunction(context.getSession(), tableFunctionNode.getName(), tableFunctionNode.getHandle());


applyFunction should return a TableHandle and a list of ColumnHandle for that TH corresponding to the columns coming out of the original table function. This optimizer should not have to derive that from the table metadata, which may not actually exist for a "virtual" table.

martint · 2022-04-26T00:47:58Z

core/trino-main/src/main/java/io/trino/sql/planner/plan/TableFunctionNode.java

+public class TableFunctionNode
+        extends PlanNode
+{
+    private final QualifiedObjectName name;


Why does this hold a name?

Refactored. Now the TableFunctionHandle holds the catalog name.

kasiafi · 2022-05-01T08:11:59Z

@martint Applied comments

core/trino-spi/src/main/java/io/trino/spi/Plugin.java

core/trino-main/src/main/java/io/trino/metadata/TableFunctionRegistry.java

core/trino-main/src/main/java/io/trino/metadata/TableFunctionHandle.java

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

core/trino-spi/src/main/java/io/trino/spi/ptf/ConnectorTableFunction.java

Praveen2112 · 2022-05-02T12:05:54Z

core/trino-spi/src/main/java/io/trino/spi/ptf/ConnectorTableFunction.java

+        }
+    }
+
+    static String checkNotNullOrEmpty(String value, String name)


As a follow up - can we extract it to a common util method ?

core/trino-spi/src/main/java/io/trino/spi/ptf/Descriptor.java

core/trino-spi/src/main/java/io/trino/spi/ptf/TableArgument.java

martint

Great job. Let's merge everything in except for the functions added to the jdbc and elasticsearch connectors. We need to come up with a good name for them and make sure they have reasonable semantics -- I haven't had a chance to review them yet.

core/trino-main/src/main/java/io/trino/metadata/TableFunctionHandle.java

Preparatory commit for supporting table functions in JDBC connectors

wendigo · 2022-05-10T13:54:23Z

Woah, nice job @kasiafi!

mosabua · 2022-05-11T03:48:20Z

I am going to assume this needs to release notes entry or docs at this stage. Going forward can you maybe use the PR template to indicate that @kasiafi .. thanks

kasiafi · 2022-05-11T06:12:10Z

I am going to assume this needs to release notes entry or docs at this stage.

@mosabua this does need a release notes entry and docs. I'll add the proposed release notes entry in the relnotes issue. I'm currently working on the documentation -- it's likely we'll have to wait for it until the next release.

kasiafi force-pushed the 308PTF branch from 9d12358 to be3b049 Compare March 14, 2022 14:40

cla-bot bot added the cla-signed label Mar 14, 2022

kasiafi force-pushed the 308PTF branch from be3b049 to 5fbf8b6 Compare March 14, 2022 14:45

kasiafi changed the title ~~Support table functions in grammar and AST~~ Support table functions with pushdown to connector Mar 14, 2022

kasiafi force-pushed the 308PTF branch from 5fbf8b6 to 5b3ce86 Compare March 15, 2022 14:37

martint reviewed Mar 15, 2022

View reviewed changes

kasiafi force-pushed the 308PTF branch from 5b3ce86 to e32befd Compare March 16, 2022 09:58

kasiafi requested a review from martint March 16, 2022 10:02

kasiafi force-pushed the 308PTF branch 3 times, most recently from 8c3cdc9 to a05ddd6 Compare March 18, 2022 11:15

kasiafi force-pushed the 308PTF branch 3 times, most recently from 149971a to b8aa8f0 Compare April 12, 2022 12:40

kasiafi mentioned this pull request Apr 13, 2022

Table Functions support for Trino #11937

Closed

54 tasks

kasiafi force-pushed the 308PTF branch 2 times, most recently from 34b3d71 to 907158e Compare April 26, 2022 12:33

martint reviewed Apr 26, 2022

View reviewed changes

kasiafi force-pushed the 308PTF branch 2 times, most recently from 3325323 to 0acafc1 Compare May 1, 2022 07:58

kasiafi requested a review from martint May 1, 2022 08:12

Praveen2112 reviewed May 2, 2022

View reviewed changes

kasiafi force-pushed the 308PTF branch 3 times, most recently from 7ddb7ae to 45cb337 Compare May 4, 2022 12:48

kasiafi mentioned this pull request May 4, 2022

Add access control for functions based on full qualified name #12241

Merged

kasiafi force-pushed the 308PTF branch from 45cb337 to b3026ce Compare May 5, 2022 10:29

github-actions bot added the tests:hive label May 5, 2022

kasiafi force-pushed the 308PTF branch 2 times, most recently from eec1b22 to a2139f7 Compare May 6, 2022 07:05

martint approved these changes May 9, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/metadata/TableFunctionHandle.java Outdated Show resolved Hide resolved

kasiafi added 9 commits May 10, 2022 09:19

Support table function invocation in grammar and AST

2e00c8e

Add SPI support for declaring and analyzing TableFunctions

395fd91

Pass TransactionManager to StatementAnalyzer

e69a052

Register table functions

493f639

Analyze table function invocation

9e8d51a

Plan table function invocation as TableFunctionNode

d4c7338

Add methods for table function pushdown into connector

5657009

Add rule to push table function invocation into connector

ec8b9fd

Extract transaction manager from JdbcConnector

b4d4b51

Preparatory commit for supporting table functions in JDBC connectors

kasiafi force-pushed the 308PTF branch from a2139f7 to b4d4b51 Compare May 10, 2022 07:27

kasiafi merged commit 287e869 into trinodb:master May 10, 2022

findepi mentioned this pull request May 10, 2022

Polymorphic table functions #1839

Closed

59 tasks

github-actions bot added this to the 381 milestone May 10, 2022

mosabua mentioned this pull request May 10, 2022

Add Trino 381 release notes #12277

Merged

This was referenced May 11, 2022

Release notes for 381 #12278

Closed

Document table function support #12338

Merged

kasiafi mentioned this pull request Jun 14, 2022

Support file based access control for table functions #12833

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support table functions with pushdown to connector #11336

Support table functions with pushdown to connector #11336

kasiafi commented Mar 4, 2022 •

edited

Loading

kasiafi commented Mar 14, 2022

martint Mar 15, 2022

kasiafi Mar 16, 2022

martint Mar 15, 2022

kasiafi Mar 16, 2022

martint Apr 25, 2022

kasiafi Apr 27, 2022

martint Apr 25, 2022

martint Apr 26, 2022

martint Apr 26, 2022

kasiafi Apr 30, 2022

kasiafi commented May 1, 2022

Praveen2112 May 2, 2022

martint left a comment

wendigo commented May 10, 2022

mosabua commented May 11, 2022

kasiafi commented May 11, 2022

Support table functions with pushdown to connector #11336

Support table functions with pushdown to connector #11336

Conversation

kasiafi commented Mar 4, 2022 • edited Loading

kasiafi commented Mar 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kasiafi commented May 1, 2022

Choose a reason for hiding this comment

martint left a comment

Choose a reason for hiding this comment

wendigo commented May 10, 2022

mosabua commented May 11, 2022

kasiafi commented May 11, 2022

kasiafi commented Mar 4, 2022 •

edited

Loading