Modin is not always faster than pandas

[klopstock-dviz]

Truth, vanilla pandas is faster

On a xeon 2850v2, 10 cores, 50gb of ram
Used modin to speed up load times, filters, groupings and aggregation
For data loading, modin did better (x2)
But for everything else, pandas is ... faster ! and consume far less cpu ressource

The only context where modin beats clearly (x2 faster) pandas, it is jupyter notebook (in vscode)
For casual .py, i stick to pandas

[YarShev]

Hi @klopstock-dviz, thank you for creating this discussion. Would you mind sharing a code example with us so we can look into it? It would help us to look at the issues Modin has and try to speed up?

[klopstock-dviz]

Hi @YarShev
Sure
This is the script and ipynb:
explor-dataviz.tar.gz
Please keep only on year in the years array to avoid errors

For the dataset, you can get it here:
https://www.data.gouv.fr/fr/datasets/r/87038926-fb31-4959-b2ae-7a24321c599a

For the .py i measured 49s for pandas, 71s for modin
For ipynb modin is twice faster

I am on a fresh install of linux mint

[klopstock-dviz]

precision, i run the conda versions for both pandas and modin

[YarShev]

Thanks for the detailed information. I ran the code (py file) with 2022 year dataset and got the following results on our server machine.

# Modin
total time: 30.434391260147
# pandas
total time: 28.5141842365264

It is true that there are the cases when Modin is slower than pandas. In particular, exec time for groupBy_communes on pandas is 5.61414098739624 and on Modin is 11.1960875988006. When I specified the env var MODIN_RANGE_PARTITIONING_GROUPBY=True to use the optimized groupby version, the time of this operation on Modin reduced to 2.90554046630859. Can you try to specify this env var and recheck timings? Meanwhile, we will also take a look into other cases where Modin is slower.

Also, could you share some more details on your environment?

1. What Modin version are you using?
2. Which Modin engine are you using?
3. How many CPUs do you have on your machine?

[YarShev]

Clarification, I tested the code with 112 CPUs on latest master. When I used 32 CPUs, the total execution time on Modin reduced to 14.645833492279053. The line for running is:

MODIN_CPUS=32 MODIN_RANGE_PARTITIONING_GROUPBY=True python consolidation_aggregation_dvf.py

[YarShev]

Thus, Modin is twice faster than pandas with these settings. Look forward to your updates.

[klopstock-dviz]

Hi @YarShev
My config:
modin version 0.27.0, conda version
modin engine: the default, not changed
10 cpus, 20 threads, 3.2 to 3.6ghz boost

I tried with this exact line:
MODIN_CPUS=32 MODIN_RANGE_PARTITIONING_GROUPBY=True python consolidation_aggregation_dvf.py

Modin finished in 42s
Pandas in 45s

I got this warining:
"UserWarning: The size of /dev/shm is too small (24885424128 bytes). The required size at least half of RAM (25212215296 bytes). Please, delete files in /dev/shm or increase size of /dev/shm with --shm-size in Docker. Also, you can can override the memory size for each Ray worker (in bytes) to the MODIN_MEMORY environment variable."

So it is a great improvement, 71s to 42s
But the cpu usage is at 100%, vs 10% for pandas, to perform similar timing

For casual jupyter tasks whit high data load, i would use modin to get faster feedbacks, but for backend tasks, i think pandas is more efficient in the equation cpu resources / time spent

[YarShev]

@klopstock-dviz, I measured .info() and .dropna() operations by wrapping these lines up with time.

t1 = time.time()
print(all.info())
t2 = time.time()
print(f"df.info: {t2 - t1}")
# df.info: 4.46 s

t1 = time.time()
all["anneeCreationEtablissement"] = all["dateCreationEtablissement"].str[:4]
t2 = time.time()
all.dropna(axis=0, subset=["anneeCreationEtablissement"], inplace=True)
t3 = time.time()
all["anneeCreationEtablissement"] = all["anneeCreationEtablissement"].astype(int)
t4 = time.time()
all.head()
t5 = time.time()
print(f"extract year1: {t2 - t1}")
print(f"extract year2: {t3 - t2}")
print(f"extract year3: {t4 - t3}")
print(f"extract year4: {t5 - t4}")

# extract year1: 0.04 s
# extract year2: 0.06 s
# extract year3: 31.1 s
# extract year4: 6.1 s

.info() takes a few seconds since we materialize a portion of data in the main process. .dropna() performs less than a second since there is no data materialization in the main process. Worker processes execute the operation in background. However, .astype() takes ~30 s. This is because we materialize metadata in the main process during the operation. We will look at this bottleneck as to if it is possible to avoid metadata materialization. Again, as a recommendation, even if you are working in a jupyter notebook, try to avoid data materialization in the main process as long as you really need it.

[klopstock-dviz]

@YarShev thank you for the feedback
It seems your code is executed in a .py script
Do you have the same timings under .ipynb ?

What is data materialization ? how can i avoid it ?

[YarShev]

Do you have the same timings under .ipynb ?

Yes, I measured each single line with %time in the notebook and the timings are more or less the same.

What is data materialization ? how can i avoid it ?

Data materialization is the process of fetching data from worker processes, which execute operations in background, to the main process. This requires waiting for operations being executed in background and transerring data (via interprocess communication or shared memory) to the main process. Every print statement or just df.info() in a cell materializes data so you should avoid that as much as possible.

[klopstock-dviz]

@YarShev i understand, thank you
in fact the df.info() or df.dropna() can't be avoided at start, everybody need to perform these op. iot understand the dataset and clean it
can't you execute these tasks in the background process and post to the main process only the result ?

[YarShev]

can't you execute these tasks in the background process and post to the main process only the result ?

This is exactly what we do.

in fact the df.info() or df.dropna() can't be avoided at start, everybody need to perform these op. iot understand the dataset and clean it.

I understand that df.dropna() can't be avoided since it is an operation for cleaning the data to get the desired effect. However, df.info() just gives a consise summary of a dataframe. You could use it in the initial stages to make sure the data is what you except to be but, eventually, drop this operation.

[rootsmusic]

This Project published benchmark comparisons in 2022.

[YarShev]

Hi @rootsmusic, thanks for sharing this! Since the date of publishing the article it's already been ~2 years. A lot of improvements have been done since then but general concepts about Modin are still in place.