[BUG] DataFrame.join(Series) does not maintain order of Series values in mode.pandas_compatible #14001
Comments
mroeschke
added
bug
|
We should only expect this is a bug when Otherwise these docs about no result ordering guarantee applies, from what I can tell. |
|
Ah totally forgot about the ordering guarantee. Yes, this is probably a bug only when |
mroeschke
changed the title
|
This will likely require some nontrivial changes, like adding a sequential index column before joining, then sorting after the join? Or perhaps some intermediate sorting of the join indices (can't recall the implementation details to know if this is viable at the moment). Either way, preserving order for a hash join is not ideal for performance, which is why this hasn't been done yet. I am not sure if there's any better way -- we did find some clever ways to implement |
I think we should go ahead and implement this solution for now (when Pandas compatibility is enabled). |
|
@mroeschke -- by default, Is it an implementation detail of Pandas that when cc: @wence- |
|
|
If we pass sort=True to merges we are on the hook to sort the result in order with respect to the key columns. If those key columns have repeated values there is still some space for ambiguity. Currently we get a result back whose order (for the repeated key values) is determined by the gather map that libcudf returns for the join. This does not come with any ordering guarantees. When sort=False, pandas has join-type dependent ordering guarantees which we also do not match. To fix this, in pandas-compatible mode only, reorder the gather maps according to the order of the input keys. When sort=False this means that our result matches pandas ordering. When sort=True, it ensures that (if we use a stable sort) the tie-break for equal sort keys is the input dataframe order. While we're here, switch from argsort + gather to sort_by_key when sorting results. - Closes rapidsai#14001 - Closes rapidsai/xdf#385
If we pass sort=True to merges we are on the hook to sort the result in order with respect to the key columns. If those key columns have repeated values there is still some space for ambiguity. Currently we get a result back whose order (for the repeated key values) is determined by the gather map that libcudf returns for the join. This does not come with any ordering guarantees. When sort=False, pandas has join-type dependent ordering guarantees which we also do not match. To fix this, in pandas-compatible mode only, reorder the gather maps according to the order of the input keys. When sort=False this means that our result matches pandas ordering. When sort=True, it ensures that (if we use a stable sort) the tie-break for equal sort keys is the input dataframe order. While we're here, switch from argsort + gather to sort_by_key when sorting results. - Closes rapidsai#14001
If we pass sort=True to merges we are on the hook to sort the result in order with respect to the key columns. If those key columns have repeated values there is still some space for ambiguity. Currently we get a result back whose order (for the repeated key values) is determined by the gather map that libcudf returns for the join. This does not come with any ordering guarantees. When sort=False, pandas has join-type dependent ordering guarantees which we also do not match. To fix this, in pandas-compatible mode only, reorder the gather maps according to the order of the input keys. When sort=False this means that our result matches pandas ordering. When sort=True, it ensures that (if we use a stable sort) the tie-break for equal sort keys is the input dataframe order. While we're here, switch from argsort + gather to sort_by_key when sorting results. - Closes rapidsai#14001
) If we pass sort=True to merges we are on the hook to sort the result in order with respect to the key columns. If those key columns have repeated values there is still some space for ambiguity. Currently we get a result back whose order (for the repeated key values) is determined by the gather map that libcudf returns for the join. This does not come with any ordering guarantees. When sort=False, pandas has join-type dependent ordering guarantees which we also do not match. To fix this, in pandas-compatible mode only, reorder the gather maps according to the order of the input keys. When sort=False this means that our result matches pandas ordering. When sort=True, it ensures that (if we use a stable sort) the tie-break for equal sort keys is the input dataframe order. While we're here, switch from argsort + gather to sort_by_key when sorting results. - Closes #14001 Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Ashwin Srinath (https://github.com/shwina) - Bradley Dice (https://github.com/bdice) URL: #14428
|
Cleaning up assigned issues, this is now fixed in 24.12, sorry for the delay! |
Describe the bug
DataFrame.join(Series) does not maintain order of Series values
Steps/Code to reproduce bug
Expected behavior
I would expect the order of the joined values to be maintained.
Environment overview (please complete the following information)
docker pull&docker runcommands usedEnvironment details
Please run and paste the output of the
cudf/print_env.shscript here, to gather any other relevant environment detailsAdditional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: