[FEA] UDF-Compiler: Translation of simple predicate UDF should allow predicate pushdown

[ywang529]

What is your question?
Hi, It is a great open-source project, thanks for sharing. We have been evaluating the udf-compiler, it works great as advertised, but we run into a problem when we try to test a filter pushdown. We wrote a simple UDF as below, but the PushedFilters is empty with udf-compiler:

    spark.udf.register("isOlderThan20", (i: Int) => {i>20})
    val udfResult = spark.sql("SELECT * FROM people_with_schema WHERE isOlderThan20(age)")
    
   +- FileScan json [name#0,age#1] Batched: false, DataFilters: [if ((age#1 > 20)) true else false], ..... PushedFilters: [], 

What we expected is to have a physical plan like this:

    val udfResult = spark.sql("SELECT * FROM people_with_schema WHERE age > 20")
   
    !+- Filter (isnotnull(age#1) AND (age#1 > 20))                                                                                                                                                                                                                                                     
    +- FileScan ...PushedFilters: [IsNotNull(age), GreaterThan(age,20)], 

By turning on trace, it looks like that the udf-compiler translated this udf as "if-else", instead of just "age#1 >20":
Filter if ((age#1 > 20)) true else false

Wonder if this is by design? If not, wonder if you can suggest some ideas to us on how to fix this?

Thank you very much!
-Yong Wang

[jlowe]

Thanks for the report! I verified with a vanilla Spark session that specifying an expression as an IF statement is sufficient to defeat the predicate pushdown into the load.

Migrating this into a feature request to add support for predicate pushdown for Catalyst operations resulting from the UDF compiler.

[ywang529]

Thank you very much for the help!