SQL MERGE Fixes #14650
Merged
SQL MERGE Fixes #14650
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
djsstarburst
force-pushed
the
david.stryker/sql-merge-fixes
branch
2 times, most recently
from
fea5a9e to
f6f7981
Compare
ksobolew
reviewed
Oct 17, 2022
core/trino-main/src/main/java/io/trino/operator/ChangeOnlyUpdatedColumnsMergeProcessor.java
Outdated
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMergeSink.java
Outdated
Show resolved
Hide resolved
djsstarburst
force-pushed
the
david.stryker/sql-merge-fixes
branch
from
f6f7981 to
eabdc67
Compare
We need to look at them one by one. With only one exception, we haven't seen these problems with SQL MERGE itself. Most of these problems showed up in tests in the PR that implements UPDATE and DELETE using MERGE plumbing, but migrating many of those tests would be difficult. I'll take a look, and consult @electrum, and see what new tests it makes sense to add. Thanks for looking @ksobolew! |
electrum
approved these changes
Oct 18, 2022
testing/trino-testing/src/main/java/io/trino/testing/BaseConnectorTest.java
Outdated
Show resolved
Hide resolved
...-singlenode-hdp3-config-default-41fc9f4ea7a64eeba44f57fca29cf339/hadoop-master/container.log
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/MergeProcessorOperator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/QueryPlanner.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/PlanOptimizers.java
Outdated
Show resolved
Hide resolved
| * - Values (0) | ||
| * </pre> | ||
| */ | ||
| public final class RemoveEmptyMergeWriterRuleSet |
There was a problem hiding this comment.
Can you add a test for this? Maybe TestRemoveEmptyDeleteRuleSet is a good example
There was a problem hiding this comment.
This turned out to be somewhat challenging. I spent an hour on it and didn't get the test working. I'll add the test in a second pass.
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMergeSink.java
Outdated
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMergeSink.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMergeSink.java
Outdated
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMergeSink.java
Outdated
Show resolved
Hide resolved
The bug showed up as an exception in BeginTableWrite.rewriteModifyTableScan. However, that method doesn't need to be called for SQL MERGE. It is only needed if the connector operation requires an UpdatablePageSource, and MERGE does not, so the bug is fixed by this one-liner.
Preprocess pages passed as inputs to MergeProcessorOperator and MergeWriterOperator using the canonical pagePreprocessor.
This commit adds RemoveEmptyMergeWriterRuleSet, which turns MergeWriter over an empty TableScan into an empty values node, analogous to RemoveEmptyDeleteRuleSet. Rename Pattern.merge() to Pattern.mergeWriter(), since there are several kinds of merge nodes.
Null values aren't allowed in ImmutableLists, so use an ArrayList for partition values, and update the test to succeed when changing partition columns.
This commit keep track of the number of rows written in DeltaLakeMergeSink.rewriteParquetFile, and only commits the fileWriter if at least one row was written. Otherwise it calls fileWriter.rollback(). This fixes the Delte Lake tests that failed because of unexpected filex. I think this fix also applies to SQL MERGE.
In BeginTableWrite, when returning the MergeTarget, get the table handle from the TableScanNode.
This commit adds HiveMetadata.getUpdateLayout and classes HiveUpdateHandle and HiveUpdateBucketFunction.
KuduTableHandle had a boolean field isDeleteHandle. If the field was not set, rowIds would not be generated. This commit renames the field to more appropriate requiresRowId, and sets it in KuduMetadata.beginMerge. This fixes bugs in delete and update using merge components.
djsstarburst
force-pushed
the
david.stryker/sql-merge-fixes
branch
from
eabdc67 to
6f64339
Compare
This was referenced Oct 18, 2022
|
Is there a release note that we should include for this? |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR contains the accumulation of commits in PR #13926 that fix or improve the infrastructure developed to support SQL MERGE.
Non-technical explanation
Release notes
( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: