Ignore the order locally for repartition tests #163
Conversation
| @@ -37,6 +38,7 @@ def test_coalesce_df(num_parts, length): | |||
|
|
|||
| @pytest.mark.parametrize('num_parts', [1, 10, 100, 1000, 2000], ids=idfn) | |||
| @pytest.mark.parametrize('length', [0, 2048, 4096], ids=idfn) | |||
| @ignore_order('local') | |||
There was a problem hiding this comment.
Why local? local sort does not scale well. Is it because of floating point -0.0 vs 0.0 comparison. If so we should mark it as such with a comment.
There was a problem hiding this comment.
no, just try to avoid involving any extra operation on the input DataFrame to make the test pure, since sort on Spark will shuffle the data again as far as i know.
Verifed sort on Spark also has the test passed.
There was a problem hiding this comment.
if you prefer, i can remove the 'local'
There was a problem hiding this comment.
The local sort is very slow compared to the distributed sort, and it does not scale for larger tests. I am fine with it being local, especially because this does deal with partitioning, but I would prefer a comment explaining why you picked local. Perhaps something like.
@ignore_order('local') # don't repartition again for sort
There was a problem hiding this comment.
I forgot to add that this is just a nit if you want to merge it in as-is that is fine.
There was a problem hiding this comment.
Thanks Bobby, really a good suggestion, updated to add a comment.
|
build |
|
build |
|
build |
* Ignore the order locally for repartition tests * Update repart_test.py Co-authored-by: Liangcai Li <[email protected]>
* Ignore the order locally for repartition tests * Update repart_test.py Co-authored-by: Liangcai Li <[email protected]>
Signed-off-by: spark-rapids automation <[email protected]>
This PR is to fix a comparison issue for repartition tests.
Repartition is likely to mess up the order of the rows in a DataFrame, especially when running on multiple executors, so better to ignore the order locally when comparing the result.