-
Notifications
You must be signed in to change notification settings - Fork 240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEA] Let Scalar Pandas UDF
support array of struct type.
#1912
Labels
feature request
New feature or request
Comments
firestarman
added
feature request
New feature or request
? - Needs Triage
Need team to review and classify
labels
Mar 11, 2021
Now the cuDF complains the error as below when transfering data to Python. Looks like cudf does not handle struct data well.
|
firestarman
changed the title
[FEA] Let
[FEA] Let Mar 11, 2021
Scalar Pandas UDF
support array type.Scalar Pandas UDF
support array of struct type.
Filed an issue rapidsai/cudf#7570 for the exception above. |
firestarman
added a commit
that referenced
this issue
Mar 24, 2021
This PR is to support running scalar pandas UDF with array type. Add array type signature for related expressions and plans. Flatten the names of nested struct columns from schema, which is also required by the cudf Arrow IPC writer. This PR depends on rapidsai/cudf#7598 closes #1912 Signed-off-by: Firestarman <[email protected]>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this issue
Jun 9, 2021
This PR is to support running scalar pandas UDF with array type. Add array type signature for related expressions and plans. Flatten the names of nested struct columns from schema, which is also required by the cudf Arrow IPC writer. This PR depends on rapidsai/cudf#7598 closes NVIDIA#1912 Signed-off-by: Firestarman <[email protected]>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this issue
Jun 9, 2021
This PR is to support running scalar pandas UDF with array type. Add array type signature for related expressions and plans. Flatten the names of nested struct columns from schema, which is also required by the cudf Arrow IPC writer. This PR depends on rapidsai/cudf#7598 closes NVIDIA#1912 Signed-off-by: Firestarman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
I wish the RAPIDS Accelerator for Apache Spark would support running Scalar Pandas UDF with array type as input, then it can work with
collect_list
, such as the app code as below.The text was updated successfully, but these errors were encountered: