Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Support Parquet Read of Decimal FIXED_LENGTH_BYTE_ARRAY #1324

Closed
revans2 opened this issue Dec 8, 2020 · 4 comments
Closed

[FEA] Support Parquet Read of Decimal FIXED_LENGTH_BYTE_ARRAY #1324

revans2 opened this issue Dec 8, 2020 · 4 comments
Assignees
@sperlingxx
Labels
feature request New feature or request P0 Must have for release

Comments

@revans2
Copy link
Collaborator

revans2 commented Dec 8, 2020

Is your feature request related to a problem? Please describe.

rapidsai/cudf#6909 was filed to support reading FIXED_LENGTH_BYTE_ARRAY decimal values. Older versions of Spark store decimal this way so we do want to support this before we can merge decimal reading support.

#1294 is a PR that covers most of reading decimal, but not everything.
@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Dec 8, 2020
@sameerz sameerz added P0 Must have for release and removed ? - Needs Triage Need team to review and classify labels Dec 8, 2020
@revans2 revans2 mentioned this issue Dec 8, 2020
Closed
27 tasks
@sperlingxx
Copy link
Collaborator

sperlingxx commented Dec 9, 2020 •
edited
Loading

I am willing to take the cuDF side task to support reading FIXED_LENGTH_BYTE_ARRAY decimal values if necessary.

@sperlingxx
Copy link
Collaborator

sperlingxx commented Dec 16, 2020 •
edited
Loading

@sameerz I think we can close this issue because corresponding cuDF side task has been merged. And I have manually verified this feature with spark-rapids.

@sameerz
Copy link
Collaborator

sameerz commented Dec 18, 2020

@sperlingxx do we have automated integration test cases to read FIXED_LENGTH_BYTE_ARRAY decimal values?

@revans2
Copy link
Collaborator Author

revans2 commented Jan 5, 2021

This appears to be done

@revans2 revans2 closed this as completed Jan 5, 2021
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to edb25a84aaafa0d65d36ebb94113393d4b6474fb (NV…
55d4d4e 
…IDIA#1324)

Signed-off-by: spark-rapids automation <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sperlingxx sperlingxx

Labels
feature request New feature or request P0 Must have for release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@revans2 @sperlingxx @sameerz