Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Do not convert decimal32/64 cols to decimal128 in to_arrow API and PQ writer when arrow schema is in use #17080

Open
mhaseeb123 opened this issue Oct 14, 2024 · 1 comment
Labels
0 - Blocked Cannot progress due to external reasons feature request New feature or request

Comments

@mhaseeb123
Copy link
Member

mhaseeb123 commented Oct 14, 2024

Is your feature request related to a problem? Please describe.
We currently convert decimal32 and decimal64 columns to decimal128 using #16236 whenever converting to arrow table via to_arrow or to_parquet with store_schema=True. apache/arrow#43956 (when completes) will the add the support for decimal32 and decimal64 types in arrow as well making it unnecessary to to the conversion at libcudf side and hence should be removed.

Describe the solution you'd like
Remove the conversion from d32 and d64 cols to d128 and directly write parquet or convert to_arrow

Describe alternatives you've considered
Keep converting to d128 but not needed soon.

Additional context
Blocked on the completion of apache/arrow#43956

@mhaseeb123 mhaseeb123 added feature request New feature or request 0 - Blocked Cannot progress due to external reasons labels Oct 14, 2024
@zeroshade
Copy link
Contributor

With the release of Arrow v18, i'll add implementations of Decimal32/Decimal64 to nanoarrow which will then allow avoiding the conversions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0 - Blocked Cannot progress due to external reasons feature request New feature or request
Projects
Status: No status
Development

No branches or pull requests

2 participants