Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change the default dictionary policy in Parquet writer from ALWAYS to ADAPTIVE #15570

Merged
merged 16 commits into from
May 11, 2024
Merged

Change the default dictionary policy in Parquet writer from ALWAYS to ADAPTIVE #15570

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
7ef83a6
Change the default dict policy from ALWAYS to ADAPTIVE
mhaseeb123 Apr 19, 2024
d897d94
updating default dict policy for python to ADAPTIVE as well
mhaseeb123 Apr 19, 2024
96b35f8
minor comment update.
mhaseeb123 Apr 22, 2024
8ba2258
Merge branch 'branch-24.06' into pq-dictpolicy-and-nvcomp-with-zstd
mhaseeb123 May 6, 2024
144dedf
updates for dict_policy to accept boolean as well as strings
mhaseeb123 May 7, 2024
1e6e982
Revert "updates for dict_policy to accept boolean as well as strings"
mhaseeb123 May 7, 2024
ed9a324
add max_dictionary_size argument to control the adaptive dictionary b…
mhaseeb123 May 7, 2024
992ec9f
add docstring for max_dictionary_size
mhaseeb123 May 8, 2024
bf296d4
minor trim whitespaces hook changes
mhaseeb123 May 8, 2024
187729f
Apply suggestions from code review
mhaseeb123 May 8, 2024
3d56930
dictionary updates for the chunked writer
mhaseeb123 May 8, 2024
7162c0e
add dictionary options to ParquetWriter class as well
mhaseeb123 May 8, 2024
233b3f8
minor error fix
mhaseeb123 May 8, 2024
729a609
add pytests for dictionary settings in ParquetWriter
mhaseeb123 May 9, 2024
3900120
Update comment for consistency
vuule May 10, 2024
b718fd6
Merge branch 'branch-24.06' into pq-dictpolicy-and-nvcomp-with-zstd
vuule May 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cpp/include/cudf/io/parquet.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -555,7 +555,7 @@ class parquet_writer_options {
// Maximum size of min or max values in column index
int32_t _column_index_truncate_length = default_column_index_truncate_length;
// When to use dictionary encoding for data
dictionary_policy _dictionary_policy = dictionary_policy::ALWAYS;
mhaseeb123 marked this conversation as resolved.
Show resolved Hide resolved
dictionary_policy _dictionary_policy = dictionary_policy::ADAPTIVE;
mhaseeb123 marked this conversation as resolved.
Show resolved Hide resolved
// Maximum size of column chunk dictionary (in bytes)
size_t _max_dictionary_size = default_max_dictionary_size;
// Maximum number of rows in a page fragment
Expand Down
Loading