Delta 0.8.0 under Spark 3.1.1 throws NoSuchMethodError when invoking updateExpr method on deltaTable

[unical1988]

Hello,

The error is the following :
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.Alias.<init>(Lorg/apache/spark/sql/catalyst/expressions/Expression;Ljava/lang/String;Lorg/apache/spark/sql/catalyst/expressions/ExprId;Lscala/collection/Seq;Lscala/Option;)V

Spark version :

<groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>3.1.1</version>

Delta Version :

<groupId>io.delta</groupId>
            <artifactId>delta-core_2.12</artifactId>
            <version>0.8.0</version>

[tdas]

delta 0.8.0 is not compatible with spark 3.1 yet. we are in the process of migrating our master branch to be 3.1 compatible. next release will be compatible with 3.1.

[unical1988]

Even 0.7.0 version of delta is not ! what's compatible with Spark 3.1 ? i want to use the following method under Spark 3.1.1 how to do ? :

 deltaTable.updateExpr(            
                "col1= 'FZA'",
                new HashMap<String, String>() {{
                    put("col1", "'FZZR'");
                }}
        );
Otherwise how to do the update based on a constraint over the column of a DataFrame in Spark ? 

[jaceklaskowski]

Even 0.7.0 version of delta is not

At the time of 0.7.0 Spark 3.1.1 was not even available so I consider it expected 😏

[jaceklaskowski]

Duplicate of #594

[michaelli916]

I am using insert sql to insert into a delta table, the statement is like "insert into delta_test_table1 values (current_date(),'eventid1','eventtype1','data1');",
and i see the same error when using delta-core_2.12-0.8.0 against spark-3.1.1-bin-hadoop2.7, which is officically released by spark, while i don't see the issue when using delta-core_2.12-0.8.0 against spark-3.0.2-bin-hadoop2.7-hive1.2.

So i guess, yes, delta 0.8.0 is not compatible with spark-3.1.1.
Look forward to the patch release of delta 0.8.0, which should be compatible with spark 3.1.1 on this sql insert issue.

[dishkakrauch]

Also faced this problem using spark 3.1.1 and delta 0.8.0.
When will be delta patch released?

[zsxwing]

Closing this. Please check #594 for the latest status.