Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Create non-shim specific version of ParquetCachedBatchSerializer #3314

Closed
tgravescs opened this issue Aug 26, 2021 · 0 comments · Fixed by #3473
Closed

[FEA] Create non-shim specific version of ParquetCachedBatchSerializer #3314

tgravescs opened this issue Aug 26, 2021 · 0 comments · Fixed by #3473
Assignees
Labels
P0 Must have for release task Work required that improves the product but is not user facing

Comments

@tgravescs
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Today for user to specify ParquetCachedBatchSerializer, they have to specify the shim version:
--conf spark.sql.cache.serializer=com.nvidia.spark.rapids.shims.spark311.ParquetCachedBatchSerializer

Ideally we don't want user to have to say spark version and they just specify something like com.nvidia.spark.rapids.ParquetCachedBatchSerializer and we load the proper version.

Note with the 3.2 build work going on, we have created versions for each shim temporarily as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0 Must have for release task Work required that improves the product but is not user facing
Projects
None yet
3 participants