You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I am seeing a Repartition being added incorrectly in some cases in our IOx plans (which then causes resorts to happen, which is a huge deal for us)
Expected behavior
If the data is sorted it should not be resorted
To Reproduce
My suspicion is that #4714 / 899c86a is the specific code that is causing this change (as the relies_on_input_order, which I added explicitly for this case, is now ignored -- see #4856).
I think the fix is to update DataFusion to be smarter about knowing how UnionExec is sorted in
In fact, looking at the tests I wrote and were changed in #4714
Describe the bug
I am seeing a Repartition being added incorrectly in some cases in our IOx plans (which then causes resorts to happen, which is a huge deal for us)
Expected behavior
If the data is sorted it should not be resorted
Additional context
The repartition is being added by the https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/physical_optimizer/repartition.rs physical optimizer pass
I previously fixed a similar issue like this in #1776 but DataFusion got more sophisticated recently (in a good way). I believe that somehow it doesn't realize that the output of the UnionExec is sorted and thus should not be repartitioned
To Reproduce
My suspicion is that #4714 / 899c86a is the specific code that is causing this change (as the relies_on_input_order, which I added explicitly for this case, is now ignored -- see #4856).
I think the fix is to update DataFusion to be smarter about knowing how UnionExec is sorted in
In fact, looking at the tests I wrote and were changed in #4714
https://github.com/apache/arrow-datafusion/blob/556282a8b6da6cb7d41d8c311211ae49b7ed82a7/datafusion/core/src/physical_optimizer/repartition.rs#L574-L586
You can see exactly that the Repartition and sort have been added
Found while updating DataFusion in IOx: https://github.com/influxdata/influxdb_iox/pull/6483#discussion_r1065433368
The text was updated successfully, but these errors were encountered: