Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the int32 overflow when computing page fragment sizes for large string columns #16028

Merged
merged 2 commits into from
Jun 14, 2024
Merged

Fix the int32 overflow when computing page fragment sizes for large string columns #16028

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
acfcd8a
Fix int32 overflow for large string columns when computing fragment s…
mhaseeb123 Jun 13, 2024
85299e4
Merge branch 'rapidsai:branch-24.08' into pq-large-strings-zstd-comp
mhaseeb123 Jun 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cpp/src/io/parquet/writer_impl.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1763,10 +1763,10 @@ auto convert_table_to_parquet_data(table_input_metadata& table_meta,
// for multiple fragments per page to smooth things out. using 2 was too
// unbalanced in final page sizes, so using 4 which seems to be a good
// compromise at smoothing things out without getting fragment sizes too small.
auto frag_size_fn = [&](auto const& col, size_type col_size) {
auto frag_size_fn = [&](auto const& col, size_t col_size) {
mhaseeb123 marked this conversation as resolved.
Show resolved Hide resolved
int const target_frags_per_page = is_col_fixed_width(col) ? 1 : 4;
auto const avg_len =
target_frags_per_page * util::div_rounding_up_safe<size_type>(col_size, input.num_rows());
target_frags_per_page * util::div_rounding_up_safe<size_t>(col_size, input.num_rows());
ttnghia marked this conversation as resolved.
Show resolved Hide resolved
if (avg_len > 0) {
auto const frag_size = util::div_rounding_up_safe<size_type>(max_page_size_bytes, avg_len);
return std::min<size_type>(max_page_fragment_size, frag_size);
Expand Down
Loading