Address poor performance of Parquet string decoding #15304
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidwendt
added
2 - In Progress
|
/ok to test |
|
String decode current 24.04 top, this PR (using avg length per batch of 32 strings) bottom. Note that there are only 4 pages being decoded in this file.
|
|
Thank you @etseidl for looking into this performance case! |
|
@etseidl would you please let me know if there are more changes you would like to make - or is this ready for review? |
@GregoryKimball If a quick fix is wanted for 24.04, then I think this is ready. It will take longer to evaluate more complicated solutions. My first attempt at going block wide didn't pan out :( |
etseidl
changed the title
|
/ok to test |
ttnghia
approved these changes
Mar 19, 2024
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
See #15297. The Parquet string decoder can become a bottleneck in the presence of strings of widely varying sizes. This PR is an attempt to address this, at least as a stop gap solution. A more complete solution may be to rework the string decoder to work in a block-wide fashion, such as the new micro-kernels added in #15159.
Checklist