Variable fragment sizes for Parquet writer #12685
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pull requests from external contributors require approval from a |
|
This PR retains the performance gains seen in #12627 for LIST benchmarks. 23.04: This PR: |
|
For parquet_write_io_compression This PR: |
etseidl
force-pushed
the
feature/frag_sizev4
branch
from
fa06424 to
b8bb1dd
Compare
etseidl
commented
Feb 3, 2023
etseidl
commented
Feb 3, 2023
etseidl
commented
Feb 3, 2023
|
LIST write benchmark got even better with the 4 fragments/page change, but this is due to allowing more parallelism in page encoding and compression due to simply adding more pages of smaller size. Kind of a cheat 😅 |
etseidl
force-pushed
the
feature/frag_sizev4
branch
from
c674c23 to
138d5f4
Compare
Co-authored-by: Vukasin Milovanovic <[email protected]>
Co-authored-by: Vukasin Milovanovic <[email protected]>
|
/ok to test |
vuule
approved these changes
Feb 15, 2023
|
Thanks for all the help! Going to release it into the wild now :D |
etseidl
changed the title
hyperbolic2346
requested changes
Feb 16, 2023
hyperbolic2346
approved these changes
Feb 16, 2023
|
/ok to test |
|
/ok to test |
|
/merge |
3 tasks
rapids-bot bot
pushed a commit
that referenced
this pull request
Mar 1, 2023
Fixes #12867. Bug introduced in #12685. A calculation of total bytes in a column was returned in a 32-bit `size_type` rather than 64-bit `size_t` leading to overflow for tables with many millions of rows. Authors: - Ed Seidl (https://github.com/etseidl) - Vukasin Milovanovic (https://github.com/vuule) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Karthikeyan (https://github.com/karthikeyann) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: #12870
3 tasks
rapids-bot bot
pushed a commit
that referenced
this pull request
Aug 3, 2023
…ragment size (#13806) #12685 introduced a bug in page calculation. If the `max_page_size_rows` parameter is set smaller than the page fragment size, the writer will produce a spurious empty page. This PR fixes this by only checking the fragment size if there are already rows in the page, and then returns the old check for number of rows exceeding the page limit. Interestingly, libcudf can read these files with empty pages just fine, but parquet-mr cannot. Authors: - Ed Seidl (https://github.com/etseidl) Approvers: - Nghia Truong (https://github.com/ttnghia) - Vukasin Milovanovic (https://github.com/vuule) URL: #13806
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes #12613
This PR adds the ability for columns to have different fragment sizes. This allows a large fragment size for narrow columns, but allows for finer grained fragments for very wide columns. This change should make wide columns fit (approximately) within page size constraints, and should help with compressors that rely on pages being under a certain threshold (i.e. Zstandard).
Checklist