[FEA] improve performance of chunked parquet write by not using pinned memory #12186
Labels
0 - Backlog
In queue waiting for assignment
cuIO
cuIO issue
feature request
New feature or request
libcudf
Affects libcudf (C++/CUDA) code.
Performance
Performance related issue
Spark
Functionality that helps Spark RAPIDS
Milestone
Comments
revans2
added
feature request
GregoryKimball
added
0 - Backlog
3 tasks
rapids-bot bot
pushed a commit
that referenced
this issue
Dec 2, 2022
…vice_vector` (#12251) Enables experimentation related to issue #12186 Frequent allocation of pinned memory becomes costly in multi-GPU environment, where all devices synchronize and allocation creates huge bubbles in GPU utilization. This PR adds environment variable `LIBCUDF_PREFER_PAGEABLE_MEMORY` that controls whether `hostdevice_vector` uses pinned memory for the host side. `hostdevice_vector` makes up most of pinned memory allocations in cuIO, so this environment variable should greatly impact the execution when using multiple GPUs. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Nghia Truong (https://github.com/ttnghia) - Mike Wilson (https://github.com/hyperbolic2346) - Robert Maynard (https://github.com/robertmaynard) URL: #12251
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
I am not sure if this is a bug or a feature request but I am leaning towards a feature request. In Spark we typically have multiple streams processing data at the same time. Some of these may be copying data to or from the GPU. We have noticed that when doing a chunked parquet write that we can get some really large sections, when a
cudaHostFreeis being called, which will synchronize with the GPU.These tend to correspond with being blocked behind a large data transfer to the GPU. Part of this is interesting because the
cudf::io::data_sinkwe use returnstrueforsupports_device_write. So there should be very little need for pinned memory at all. I really would like to understand if we really need pinned memory? In all cases? Even in the case where we accept device buffers for the write?The text was updated successfully, but these errors were encountered: