Measure the number of Parquet row groups filtered by predicate pushdown #17594
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
mhaseeb123
added
2 - In Progress
mhaseeb123
changed the title
mhaseeb123
commented
Jan 16, 2025
| // if filter is not empty, then gather row groups to read after predicate pushdown | ||
| if (filter.has_value()) { | ||
| filtered_row_group_indices = filter_row_groups( | ||
| sources, row_group_indices, output_dtypes, output_column_schemas, filter.value(), stream); | ||
| // Span of input row group indices for predicate pushdown |
There was a problem hiding this comment.
Moved this piece of code out of filter_row_groups() to make its signature consistent with apply_bloom_filters()
mhaseeb123
commented
Jan 16, 2025
| host_span<data_type const> output_dtypes, | ||
| host_span<int const> output_column_schemas, | ||
| std::reference_wrapper<ast::expression const> filter, | ||
| rmm::cuda_stream_view stream) const | ||
| { | ||
| auto mr = cudf::get_current_device_resource_ref(); | ||
| // Create row group indices. |
There was a problem hiding this comment.
Moved this piece outside in select_row_groups() to take in input_row_group_indices as input param
mhaseeb123
commented
Jan 16, 2025
| std::map<std::string, std::string> user_data; //!< Format-dependent metadata of the first input | ||
| //!< file as key-values pairs (deprecated) | ||
| std::vector<std::unordered_map<std::string, std::string>> | ||
| per_file_user_data; //!< Per file format-dependent metadata as key-values pairs | ||
|
|
||
| // The following variables are currently only computed for Parquet reader | ||
| size_type num_input_row_groups; //!< Number of input row groups across all data sources |
There was a problem hiding this comment.
Keeping all variables tracking number of row groups as size_type for consistency across the entire Parquet stack. Happy to change it to size_t all across in a separate PR if needed. @vuule @nvdbaranec
mhaseeb123
commented
Jan 16, 2025
| bloom_filter_spans, parquet_types, total_row_groups, equality_col_schemas.size()}; | ||
| bloom_filter_caster const bloom_filter_col{bloom_filter_spans, | ||
| parquet_types, | ||
| static_cast<size_t>(total_row_groups), |
There was a problem hiding this comment.
static_cast simply reformatted this line
mhaseeb123
added
3 - Ready for Review
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Related to #17164
This PR adds a method to measure the number of row groups filtered using stats and bloom filters during predicate pushdown.
Checklist