exploit iceberg row-count metadata #74
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
October 9, 2024 14:09
Examine the row-counts in the manifest and count how many rows are in the (existing or added) data an deletion files. Use these two counts to pass the new named_parameter "explicit_cardinality" into the generated parquet_scans. Note that because iceberg is passing in an explicit schema into the parquet_scan, it does not open any file during bind. That means the parquet_scans it generates did not have any cardinality information during query optimization. That can cause rather bad query plans. This PR (pb/explicit-iceberg-cardinality) fixes that. Note that this PR needs the DuckDB PR pb/exicit-parquet-cardinality that adds the "explicit_cardinality" named_parameter to parquet_scan.
|
This CI is expected to fail (unknown named_parameter 'explicit_cardinality') until duckdb/duckdb#14292 would have been merged |
added 3 commits
October 14, 2024 14:30
- add test that checks that there are cardinalities in the generated parquet_scans
mike-luabase
pushed a commit
to definite-app/duckdb_iceberg
that referenced
this pull request
Oct 27, 2024
…rdinality exploit iceberg row-count metadata
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Examine the row-counts in the manifest and count how many rows are in the (existing or added) data resp. deletion files.
Use these two counts to pass the new named_parameter "explicit_cardinality" into the generated parquet_scans.
Note that because iceberg is passing in an explicit schema into the parquet_scan, it does not open any file during bind. That means the parquet_scans it generates did not have any cardinality information during query optimization.
That can cause rather bad query plans. This PR (pb/explicit-iceberg-cardinality) fixes that. Note that this PR needs the DuckDB PR pb/explicit-parquet-cardinality that adds the "explicit_cardinality" named_parameter to parquet_scan.