Do not materialize row partitions while doing reshuffling #5816
Labels
partitions reshuffling 🔀
Issues related to partitions reshuffling
Performance 🚀
Performance related issues and pull requests.
At the moment we're triggering
.force_materialization()
for row partitions in order to convert them to simple partitions that support the.split()
method:modin/modin/core/dataframe/pandas/partitioning/partition_manager.py
Lines 1584 to 1590 in d5f3f72
If we have implemented the
.split()
method for axis partitions we could have skipped this materialization step and thus save some time:CPU: Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz, 112 threads
The text was updated successfully, but these errors were encountered: