-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add explicit check for modifying data in delta table with CDF enabled #14470
Add explicit check for modifying data in delta table with CDF enabled #14470
Conversation
a9c893c
to
677d2ca
Compare
0e0e7e2
to
5f87b1f
Compare
Sent #14508 within the repository. |
...elta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/DeltaLakeSchemaSupport.java
Outdated
Show resolved
Hide resolved
5f87b1f
to
e51007e
Compare
public static boolean changeDataFeedEnabled(MetadataEntry metadataEntry) | ||
{ | ||
String enableChangeDataFeed = metadataEntry.getConfiguration().getOrDefault("delta.enableChangeDataFeed", "false"); | ||
return enableChangeDataFeed.equalsIgnoreCase("true"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is java.lang.Boolean#parseBoolean
what Delta is using?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is Delta implementation:
val CHANGE_DATA_FEED = buildConfig[Boolean](
"enableChangeDataFeed",
"false",
_.toBoolean,
_ => true,
"needs to be a boolean.",
alternateConfs = Seq(CHANGE_DATA_FEED_LEGACY))
def toBoolean: Boolean = parseBoolean(toString)
private def parseBoolean(s: String): Boolean =
if (s != null) s.toLowerCase match {
case "true" => true
case "false" => false
case _ => throw new IllegalArgumentException("For input string: \""+s+"\"")
}
else
throw new IllegalArgumentException("For input string: \"null\"")
"TBLPROPERTIES (delta.enableChangeDataFeed = true)"); | ||
|
||
assertQueryFailure(() -> onTrino().executeQuery("INSERT INTO delta.default." + tableName + " VALUES (1, 2)")) | ||
.hasMessageMatching(".* Table .* requires Delta Lake writer version 4 which is not supported"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So it was blocked before the change as well and the change just introduces an explicit check with an explicit message.
Let's have a different commit and PR title then
e51007e
to
c28b844
Compare
maven checks were killed because they were super slow, doesn't look like anything actionable |
Could you rebase on upstream to resolve conflicts? |
e7d2288
to
875a541
Compare
Could you clarify what the user impact of this is for the release note? It seems to me like this would always fail (due to that writer version check), but now it's failing with better error messaging? |
Correct. There's no need to mention in a release note. |
Description
Add explicit check for modifying data in delta table with CDF enabled.
Inserts could be allowed as they don't have to do anything special to support CDF but they are blocked by writer version check.
Non-technical explanation
Release notes
(x) This is not user-visible or docs only and no release notes are required.