Implement complex join pushdown in JDBC connectors #19996
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
findepi
force-pushed
the
findepi/apply-join
branch
from
31de58e to
bbdc659
Compare
findepi
force-pushed
the
findepi/apply-join
branch
from
bbdc659 to
ae7aaa0
Compare
|
Will unnest operation be covered in this PR? |
sajjoseph
reviewed
Dec 3, 2023
plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/DefaultJdbcMetadata.java
Show resolved
Hide resolved
sajjoseph
reviewed
Dec 3, 2023
plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/DefaultJdbcMetadata.java
Show resolved
Hide resolved
no, no plan for this. |
findepi
force-pushed
the
findepi/apply-join
branch
from
da7a572 to
bfbc2b2
Compare
|
(just rebased after #19998 merged. no other changes yet) |
findepi
force-pushed
the
findepi/apply-join
branch
from
bfbc2b2 to
75df2c9
Compare
|
(just rebased) |
findepi
force-pushed
the
findepi/apply-join
branch
5 times, most recently
from
7b863c2 to
75c7b08
Compare
|
PTAL |
wendigo
approved these changes
Jan 3, 2024
...-base-jdbc/src/main/java/io/trino/plugin/jdbc/expression/RewriteCaseSensitiveComparison.java
Show resolved
Hide resolved
...-base-jdbc/src/main/java/io/trino/plugin/jdbc/expression/RewriteCaseSensitiveComparison.java
Show resolved
Hide resolved
...-base-jdbc/src/main/java/io/trino/plugin/jdbc/expression/RewriteCaseSensitiveComparison.java
Show resolved
Hide resolved
...-base-jdbc/src/main/java/io/trino/plugin/jdbc/expression/RewriteCaseSensitiveComparison.java
Show resolved
Hide resolved
plugin/trino-oracle/src/main/java/io/trino/plugin/oracle/RewriteStringComparison.java
Show resolved
Hide resolved
plugin/trino-oracle/src/main/java/io/trino/plugin/oracle/RewriteStringComparison.java
Show resolved
Hide resolved
plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java
Show resolved
Hide resolved
findepi
force-pushed
the
findepi/apply-join
branch
from
f0b0a45 to
5683eee
Compare
|
(just rebased, no other changes) |
Implement non-deprecated `ConnectorMetadata.applyJoin` overload in `DefaultJdbcMetadata`. Thew old implementation is retained as a safety valve. The new implementation is not limited to the `List<JdbcJoinCondition>` model, so allows pushdown of joins involving more complex expressions, such as arithmetics. The `BaseJdbcClient.implementJoin` and `QueryBuilder.prepareJoinQuery` methods logically changed, but the old implementation is left as the fallback. These methods were extension points, so the old implementations are renamed to ensure implementors are updated. For example, if an implementation was overriding `BaseJdbcClient.implementJoin` it most likely wants to override the new `implementJoin` method as well, and this is reminded about by rename of the old method.
Restore older JDBC join pushdown implementation not based on `ConnectorExpression` as a fallback. This comes as a separate commit so that the introduction of `ConnectorExpression`-based join pushdown can be seen (e.g. reviewed) as a _change_, not as an _addition_.
findepi
force-pushed
the
findepi/apply-join
branch
from
5683eee to
4450dde
Compare
wendigo
approved these changes
Jan 10, 2024
|
CI #20324 |
martint
reviewed
Jan 10, 2024
| @@ -1196,6 +1199,8 @@ public void testJoinPushdown() | |||
| "nation_lowercase", | |||
| "AS SELECT nationkey, lower(name) name, regionkey FROM nation")) { | |||
| for (JoinOperator joinOperator : JoinOperator.values()) { | |||
| log.info("Testing joinOperator=%s", joinOperator); | |||
There was a problem hiding this comment.
This pollutes the logs and makes it much harder to spot where the actual errors are happening. We're actually trying to reduce logging in tests. Please revert this.
There was a problem hiding this comment.
if there is a failure on CI, how do we know what failed?
There was a problem hiding this comment.
You can add a .describedAs(...) to the assertion containing the details of which join operator is being evaluated. That will be included in the failure message.
|
Does this need a release note? |
findepi
added
the
no-release-notes
|
Thanks @findepi for this PR. I see tremendous improvements with JOIN pushdown in our custom connector (Trino-to-Trino, similar to Stargate from Starburst). We had multiple custom rules in our connector that are now working in tandem and many complex joins are now getting pushed down into the remote cluster. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement non-deprecated
ConnectorMetadata.applyJoinoverload inDefaultJdbcMetadata. Thew old implementation is retained as a safetyvalve. The new implementation is not limited to the
List<JdbcJoinCondition>model, so allows pushdown of joins involvingmore complex expressions, such as arithmetics.