UDF/UDAF plugin

[EricJoy2048]

Now we cannot use UDF and UDAF in ballista because ballista cannot know how to serialize and deserialize UDF / UDAF.
We are using Trino. Referring to the practice of Trino, we can realize the plug-in of UDF through the way of rust dynamic library. In this way, ballista and datafusion only need to know the plug-in interface of UDF, and they can work without knowing the specific implementation of UDF.

[Igosuki]

Hi, really cool stuff. I use dtolnay/inventory on my project, but it has a known issue where one cannot guarantee that symbols won't get mangled from statically compiled code by llvm. Stuff got stirred and the core team reacted rust-lang/rust#47384 but it's not solved yet.
Can we guarantee here that statically compiled plugins won't end up forgotten in binaries ?

Secondly, I think it'd be cool to implement your interface in datafusion python so that people can use a python function as a UDAF like it's done in pyspark https://spark.apache.org/docs/2.4.0/sql-pyspark-pandas-with-arrow.html#pandas-udfs-aka-vectorized-udfs

[EricJoy2048]

In the PR: #1881 I refer to https://adventures.michaelfbryan.com/posts/plugins-in-rust/ and https://michael-f-bryan.github.io/rust-ffi-guide/dynamic_loading.html These two articles are used to design UDF plugin. The idea requires that the crate type of the plug-in must be cdylib .According to the preliminary test, there is no problem of statically compiled plugins won't end up forgotten in binaries.
Sorry, I'm not very familiar with Python. If you like, you can help implement the code related to datafusion python.
thank you!

[alamb]

@gaojun2048 is this issue still tracking anything actionable? I think this has been done

[EricJoy2048]

Now we cannot use UDF and UDAF in ballista because ballista cannot know how to serialize and deserialize UDF / UDAF.
We are using Trino. Referring to the practice of Trino, we can realize the plug-in of UDF through the way of rust dynamic library. In this way, ballista and datafusion only need to know the plug-in interface of UDF, and they can work without knowing the specific implementation of UDF.

Yes. Let me close this issue.