You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When a UDF is passed to explode() function like explode(myUdf), the UDF compiler will not be used to try to compile this myUdf.
Root cause
The udf plugin didn't try to replace a udf is this udf is a child plan of Project
Steps/Code to reproduce bug
val myudf: (String) => Array[String] = a => {
a.split(",")
}
val u = makeUdf(myudf)
val dataset = List("first,second").toDF("x").repartition(1)
var result = dataset.withColumn("new", explode(u(col("x"))))
result.explain(true)
Expected behavior
I should be able to see the compiled Catalyst expressions like "split(x....) AS new" after "Analyzed Logical Plan".
Environment details (please complete the following information)
Environment location: Local in an IDEA debug environment
Spark configuration settings related to the issue :
With the fix from @abellina , the project child of explode can be compiled now.
diff --git a/udf-compiler/src/main/scala/com/nvidia/spark/udf/Plugin.scala b/udf-compiler/src/main/scala/com/nvidia/spark/udf/Plugin.scala
index 679d126b9..98164e70e 100644
--- a/udf-compiler/src/main/scala/com/nvidia/spark/udf/Plugin.scala
+++ b/udf-compiler/src/main/scala/com/nvidia/spark/udf/Plugin.scala
@@ -82,7 +82,7 @@ case class LogicalPlanRules() extends Rule[LogicalPlan] with Logging {
plan match {
case project: Project =>
Project(project.projectList.map(e => attemptToReplaceExpression(plan, e))
- .asInstanceOf[Seq[NamedExpression]], project.child)
+ .asInstanceOf[Seq[NamedExpression]], apply(project.child))
case x => {
x.transformExpressions(replacePartialFunc(plan))
}
The real problem and correct description for this issue turns to be "child of Project cannot be replaced". @abellina Do you think we can close this issue with your fix and create another issue for GpuExplode ? It's for the sql plugin side, not a UDF problem.
Describe the bug
When a UDF is passed to explode() function like
explode(myUdf)
, the UDF compiler will not be used to try to compile thismyUdf
.Root cause
The udf plugin didn't try to replace a udf is this udf is a child plan of
Project
Steps/Code to reproduce bug
Expected behavior
I should be able to see the compiled Catalyst expressions like "split(x....) AS new" after "Analyzed Logical Plan".
Environment details (please complete the following information)
The text was updated successfully, but these errors were encountered: