parquet: Make page_index/pushdown metrics consistent with row_group metrics #12545
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…etrics
1. Rename `{pushdown,page_index}_filtered` to `{pushdown,page_index}_pruned`
2. Add `{pushdown,page_index}_matched`
The latter makes it clearer in EXPLAIN ANALYZE when the Page Index is
not checked because their row groups were already eliminated
(with a Bloom Filter or row group statistics).
github-actions
bot
added
documentation
alamb
approved these changes
Sep 20, 2024
datafusion/core/src/datasource/physical_plan/parquet/page_filter.rs
Outdated
Show resolved
Hide resolved
| @@ -276,6 +281,14 @@ fn rows_skipped(selection: &RowSelection) -> usize { | |||
| .fold(0, |acc, x| if x.skip { acc + x.row_count } else { acc }) | |||
| } | |||
|
|
|||
| /// returns the number of rows not skipped in the selection | |||
| /// TODO should this be upstreamed to RowSelection? | |||
There was a problem hiding this comment.
This looks the same as https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.RowSelection.html#method.row_count
It would be great to upstream this and rows_skipped to parquet -- any chance you are willing to file a ticket to do so?
There was a problem hiding this comment.
sent to apache/arrow-rs#6429
| @@ -223,6 +223,21 @@ Again, reading from bottom up: | |||
| - `SortPreservingMergeExec` | |||
| - `output_rows=5`, `elapsed_compute=2.375µs`: Produced the final 5 rows in 2.375µs (microseconds) | |||
|
|
|||
| When predicate pushdown is enabled, `ParquetExec` gains the following metrics: | |||
Co-authored-by: Andrew Lamb <[email protected]>
This was referenced Sep 20, 2024
|
Thanks agian @progval |
bgjackma
pushed a commit
to bgjackma/datafusion
that referenced
this pull request
Sep 25, 2024
…etrics (apache#12545) * parquet: Make page_index/pushdown metrics consistent with row_group metrics 1. Rename `{pushdown,page_index}_filtered` to `{pushdown,page_index}_pruned` 2. Add `{pushdown,page_index}_matched` The latter makes it clearer in EXPLAIN ANALYZE when the Page Index is not checked because their row groups were already eliminated (with a Bloom Filter or row group statistics). * Add missing metric definitions in the docs Co-authored-by: Andrew Lamb <[email protected]> * s/pass/select/ --------- Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #12543.
Closes #12544.
What changes are included in this PR?
{pushdown,page_index}_filteredto{pushdown,page_index}_pruned{pushdown,page_index}_matchedRationale for this change
The latter makes it clearer in EXPLAIN ANALYZE when the Page Index is not checked because their row groups were already eliminated (with a Bloom Filter or row group statistics).
Are these changes tested?
yes
Are there any user-facing changes?
New metrics in
EXPLAIN ANALYZE, documented in docs/source/user-guide/explain-usage.md