[fix](nereids)Solve the problem of pruning wrong partitions in multi-column partition pruning #43332
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
1 similar comment
|
run buildall |
morrySnow
previously approved these changes
Nov 7, 2024
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
|
run buildall |
github-actions
bot
removed
the
approved
|
run p0 |
1 similar comment
|
run p0 |
feiniaofeiafei
force-pushed
the
fix_multi_column_partition_prune_with_func
branch
from
85f09cd to
cf9dd4c
Compare
|
run buildall |
morrySnow
approved these changes
Nov 11, 2024
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
924060929
approved these changes
Nov 12, 2024
github-actions bot
pushed a commit
that referenced
this pull request
Nov 12, 2024
…column partition pruning (#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
github-actions bot
pushed a commit
that referenced
this pull request
Nov 12, 2024
…column partition pruning (#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 12, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 12, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
zzzxl1993
pushed a commit
to zzzxl1993/doris
that referenced
this pull request
Nov 12, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
yiguolei
pushed a commit
that referenced
this pull request
Nov 12, 2024
…ns in multi-column partition pruning (#43658) Cherry-picked from #43332 Co-authored-by: feiniaofeiafei <[email protected]>
py023
pushed a commit
to py023/doris
that referenced
this pull request
Nov 13, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to:
a = 0, dt in ['2024-01-01 00:00:00', +∞)
a = 1, dt in (-∞, +∞)
a = 2, dt in (-∞, +∞)
...
a = 10, dt in (-∞, '2024-01-10 00:00:00')
Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned.
During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞)) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day').
Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition.
The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
Issue Number: close #xxx
Related PR: introduced by #38849
Problem Summary:
Check List (For Committer)
Test
Behavior changed:
Does this need documentation?
Release note
None
Check List (For Reviewer who merge this PR)