HIVE-28675: Maximize the removal of redundant columns from GROUP BY clauses #5586
Conversation
|
| if (aggregate.getGroupSet().contains(key)) { | ||
| groupByUniqueKey = key; | ||
| break; | ||
| ImmutableBitSet removableCols = originalGroupSet.except(key).except(fieldsUsed); |
There was a problem hiding this comment.
Is it possible to compute except(fieldsUsed) outside the loop? I believe changing the order of excepts will still yield the same resulting set.
There was a problem hiding this comment.
Yes we can do this, but just want to be careful that if there is no uniquekey match then we should not remove any columns at all(should skip the except in the return statement below in that case)
There was a problem hiding this comment.
Just realized that we only update columnsToRemove when there is atleast one match. So I think yes we can move the except(fieldsUsed) outside the for loop.
Also is it efficient to loop the fieldsUsed(and then check if there is a field that is a unique key and part of aggregate keys to retain only fieldsUsed) vs looping uniquekeys? Mostly depends on size of fieldsUsed vs uniqueKeys. I will probably leave it upto you to decide on that.
What changes were proposed in this pull request?
Enhance
HiveRelFieldTrimmerto remove the maximum number of redundant columns from theGROUP BYclause.Why are the changes needed?
For more see HIVE-28675.
Does this PR introduce any user-facing change?
More efficient query plans.
Is the change a dependency upgrade?
No
How was this patch tested?