[FEA] Efficient partition_cols for parquet writer

[chinmaychandak]

I remember us discussing that the partition_cols mechanism in the current GPU-acc parquet writer can be improved considerably. Was this w.r.t. the CPU parquet writer, or just a boost in performance on how the GPU handles stuff? Has anyone started looking into this?

[OlivierNV]

From what I remember of the discussion a while back, the partitioning would happen mostly upstream from the writer. For the writer, there was talk about the multi-file partitioned sink support where the writer could be provided with a vector of {row, numrows} pairs to write pieces of a table in separate files (which speeds up the encoding for many small partitions vs multiple write_parquet calls per sliced table view).

[chinmaychandak]

Thanks for providing some clarity, Olivier! @kkraus14, any ideas here?

[OlivierNV]

Note that the biggest perf problem right now is that the python code will fall back to the pyarrow CPU writer if partition_cols is anything other than None.
It was recently pointed out that df.to_parquet() is faster than df.to_arrow(), so it's like the GPU is on its second parquet victory lap before the CPU even arrived at the starting line (any benefits from multifile parquet writer support are likely minor in comparison).

[kkraus14]

It was recently pointed out that df.to_parquet() is faster than df.to_arrow(), so it's like the GPU is on its second parquet victory lap before the CPU even arrived at the starting line (any benefits from multifile parquet writer support are likely minor in comparison).

Yea we need to spend a bit of time looking at this code, but likely the overheads of doing a memory copy per buffer serially is killing us and there's a lot of room for improvement.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.