Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Parquet V2 page headers cannot currently be used with zStandard compression #14781

Closed
etseidl opened this issue Jan 18, 2024 · 0 comments · Fixed by #14841
Closed

[BUG] Parquet V2 page headers cannot currently be used with zStandard compression #14781

etseidl opened this issue Jan 18, 2024 · 0 comments · Fixed by #14841
Labels
bug Something isn't working

Comments

@etseidl
Copy link
Contributor

etseidl commented Jan 18, 2024

Describe the bug
When using V2 page headers, the repetition and definition level data is not compressed. The Parquet writer currently increments the data pointer past the level data before sending to the compression function. The zStandard compression, however, requires input to be aligned on a 4 byte boundary, so simply incrementing the pointer past the level data often results in misaligned access errors.

Additional context
Current workaround PR #14772
@etseidl etseidl added Needs Triage Need team to review and classify bug Something isn't working labels Jan 18, 2024
@etseidl etseidl mentioned this issue Jan 23, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Jan 30, 2024
@etseidl
Fix V2 Parquet page alignment for use with zStandard compression (#14841
6ed75ff 
)

Fixes #14781 

This PR makes changes to the Parquet writer to ensure that data to be compressed is properly aligned. Changes have also been made to the `EncPage` struct to make it easier to keep fields in that struct aligned, and also to reduce confusing re-use of fields. In particular, the `max_data_size` field can be any of a) the maximum possible size for the page data, b) the actual size of page data after encoding, c) the actual size of compressed page data. The latter two now have their own fields, `data_size` and `comp_data_size`.

Authors:
  - Ed Seidl (https://github.com/etseidl)
  - Mike Wilson (https://github.com/hyperbolic2346)

Approvers:
  - Mike Wilson (https://github.com/hyperbolic2346)
  - Vukasin Milovanovic (https://github.com/vuule)

URL: #14841
@bdice bdice removed the Needs Triage Need team to review and classify label Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix V2 Parquet page alignment for use with zStandard compression etseidl/cudf
2 participants
@bdice @etseidl