-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEA] Spark 3.1 orc nested predicate pushdown support #576
Comments
this got backported to 3.0.1 so we need to fix for 0.2 release |
it looks like changes in 3.0.1 and 3.1.0 are also different so we would need a shim layer. Temporarily I'm going to put in a change to just copy the function that got moved to private:
And then we can look at this in more detail to support it fully. |
ok so leaving a P2 because we fixed the function and it works, this is only needed if we want to support nested predicate pushdown |
Depends on rapidsai/cudf#7640 and rapidsai/cudf#7830 |
our orc filter code was updated with #1982 and it looks like this should just work, so perhaps we just need to test it. |
Closing as a duplicate of #1481 |
[auto-merge] bot-auto-merge-branch-22.10 to branch-22.12 [skip ci] [bot]
Describe the bug
Spark 3.1 added nested predicate pushdown support in ORC - apache/spark@7b6e1d5
This removed a function we were using:
E Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.orc.OrcFiltersBase.isSearchableType$(Lorg/apache/spark/sql/execution/datasources/orc/OrcFiltersBase;Lorg/apache/spark/sql/types/DataType;)Z
We should go through and add the same support or fix it to use the new function.
Steps/Code to reproduce bug
Run spark 3.1.0 integration tests:
FAILED integration_tests/src/main/python/orc_test.py::test_input_meta - py4j....
Expected behavior
tests pass
Environment details (please complete the following information)
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: