Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add to_arrow_device function to cudf interop using nanoarrow #15047

Merged
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
912a636
cudf to ArrowDeviceArray initial pass
zeroshade Feb 13, 2024
5f16d82
fix some whitespace
zeroshade Feb 13, 2024
2607e2e
Update cpp/include/cudf/interop.hpp
zeroshade Feb 15, 2024
e573201
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Feb 15, 2024
eb58f00
update nanoarrow
zeroshade Feb 20, 2024
124055e
update device release from feedback
zeroshade Feb 20, 2024
b3fcdfd
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Feb 20, 2024
d2b14fa
updates from feedback
zeroshade Feb 20, 2024
044511c
use validation minimal
zeroshade Feb 20, 2024
2c63261
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Feb 23, 2024
9f1ac7c
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Feb 27, 2024
f881e0c
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Feb 28, 2024
c670a8b
Update cpp/include/cudf/interop.hpp
zeroshade Feb 29, 2024
abbf465
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Feb 29, 2024
7b7a19d
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Feb 29, 2024
d7e0384
updates from feedback
zeroshade Feb 29, 2024
c085b1d
use fetchcontent to pull nanoarrow at build time
zeroshade Mar 1, 2024
5085f9e
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 1, 2024
7a41380
updates from feedback and implementation for struct/list/dict types
zeroshade Mar 1, 2024
16701e8
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 1, 2024
2217ae4
linting
zeroshade Mar 1, 2024
85c330c
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 4, 2024
43eb1bc
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 5, 2024
c0d97d3
Update cpp/src/interop/to_arrow_device.cu
zeroshade Mar 5, 2024
72bea95
Update cpp/src/interop/to_arrow_device.cu
zeroshade Mar 5, 2024
6f6c3fc
updates from feedback
zeroshade Mar 5, 2024
2005cce
switch to r-value ref
zeroshade Mar 5, 2024
98dbabc
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 5, 2024
b226904
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 6, 2024
a06203e
Update cpp/include/cudf/interop.hpp
zeroshade Mar 6, 2024
4f8c17e
updates from feedback
zeroshade Mar 6, 2024
ff629de
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 6, 2024
79af08e
style and lint fixes
zeroshade Mar 6, 2024
0947d2e
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 6, 2024
add0fd0
trailing whitespace and warning fix
zeroshade Mar 6, 2024
e5b9877
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 6, 2024
5b4d2b4
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 7, 2024
fa19d9a
fixing build
zeroshade Mar 7, 2024
8f25b79
use forward declarations and clean up
zeroshade Mar 7, 2024
7fac735
Update cpp/include/cudf/interop/detail/arrow.hpp
zeroshade Mar 7, 2024
fae0c13
Update cpp/src/interop/to_arrow_device.cu
zeroshade Mar 7, 2024
fea0bd8
shift to get_nanoarrow.cmake
zeroshade Mar 7, 2024
f96b73c
Fix clone and add docstring to satisfy linter
vyasr Mar 8, 2024
d39c0a0
Build nanoarrow statically so that we don't have to supply the library
vyasr Mar 8, 2024
6d0c867
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 8, 2024
76f81a4
Make nanoarrow compile with fpic
vyasr Mar 8, 2024
8c4d23c
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 8, 2024
56d3ab4
updates and first round of tests
zeroshade Mar 12, 2024
b578307
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 12, 2024
5e98b9d
fix linting style
zeroshade Mar 12, 2024
6f9946e
trailing whitespace
zeroshade Mar 12, 2024
f2de571
would be great if check-style gave ALL the failures...
zeroshade Mar 12, 2024
fc5a1c1
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 13, 2024
76370c7
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
davidwendt Mar 13, 2024
48ee25c
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
davidwendt Mar 13, 2024
5a2f08f
Add doxygen blocks for ArrowDeviceArray and ArrowSchema
zeroshade Mar 15, 2024
41152a2
trailing whitespace
zeroshade Mar 17, 2024
11eaf6d
add ingroup declarations
zeroshade Mar 19, 2024
baad47a
Remove documentation and switch to referencing nanoarrow's docs directly
vyasr Mar 19, 2024
19403f1
Merge remote-tracking branch 'upstream/branch-24.04' into arrow-devic…
vyasr Mar 19, 2024
7733064
Update cpp/tests/interop/to_arrow_device_test.cpp
zeroshade Mar 19, 2024
240af3b
change assert to expect to avoid mem leaks
zeroshade Mar 20, 2024
3022fbb
applying feedback
zeroshade Mar 25, 2024
e1e320c
adding comments and other feedback
zeroshade Mar 25, 2024
63407a3
Add cudf::column version of to_arrow_device
zeroshade Mar 25, 2024
d4d19f8
Merge branch 'branch-24.04' into arrow-device-array-nanoarrow
zeroshade Mar 26, 2024
301b278
Merge branch 'branch-24.06' into arrow-device-array-nanoarrow
zeroshade Mar 26, 2024
84ddc58
use custom deleter
zeroshade Mar 26, 2024
38e7fc3
check if released
zeroshade Mar 26, 2024
6e267c6
add documentation comment and fix tests
zeroshade Mar 26, 2024
f3dd2e0
forgot to format lint
zeroshade Mar 26, 2024
6cc5769
Merge branch 'branch-24.06' into arrow-device-array-nanoarrow
vyasr Mar 27, 2024
bb3d8e1
Merge branch 'branch-24.06' into arrow-device-array-nanoarrow
vyasr Mar 27, 2024
41ab3b8
Update cpp/include/cudf/interop.hpp
zeroshade Apr 1, 2024
f0c8ac6
Update cpp/include/cudf/interop.hpp
zeroshade Apr 1, 2024
3f7344c
Update cpp/include/cudf/interop.hpp
zeroshade Apr 1, 2024
d864e90
Update cpp/tests/interop/to_arrow_device_test.cpp
zeroshade Apr 1, 2024
b36a06c
Merge branch 'branch-24.06' into arrow-device-array-nanoarrow
zeroshade Apr 1, 2024
e2c858c
adding comment for scalar zero
zeroshade Apr 1, 2024
a0472d4
Update cpp/include/cudf/interop.hpp
zeroshade Apr 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -355,7 +355,10 @@ add_library(
src/interop/dlpack.cpp
src/interop/from_arrow.cu
src/interop/to_arrow.cu
src/interop/to_arrow_device.cu
src/interop/detail/arrow_allocator.cpp
src/interop/vendor/nanoarrow/nanoarrow.c
src/interop/vendor/nanoarrow/nanoarrow_device.c
src/io/avro/avro.cpp
src/io/avro/avro_gpu.cu
src/io/avro/reader_impl.cu
Expand Down
45 changes: 45 additions & 0 deletions cpp/include/cudf/interop.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
#pragma nv_diag_suppress 2810
#endif
#include <arrow/api.h>
#include <cudf/interop/nanoarrow/nanoarrow.hpp>
#include <cudf/interop/nanoarrow/nanoarrow_device.h>
zeroshade marked this conversation as resolved.
Show resolved Hide resolved
#ifdef __CUDACC__
#pragma nv_diag_default 611
#pragma nv_diag_default 2810
Expand Down Expand Up @@ -162,6 +164,49 @@ std::shared_ptr<arrow::Scalar> to_arrow(cudf::scalar const& input,
column_metadata const& metadata = {},
rmm::cuda_stream_view stream = cudf::get_default_stream(),
arrow::MemoryPool* ar_mr = arrow::default_memory_pool());

/**
* @brief Create ArrowSchema from cudf table and metadata
*
* Populates and returns an ArrowSchema C struct using a table and metadata.
*
* @param input table_view to create a schema from
* @param metadata Contains the hierarchy of names of columns and children
zeroshade marked this conversation as resolved.
Show resolved Hide resolved
* @return ArrowSchema generated from `input`
*
* @note For decimals, since the precision is not stored for them in libcudf,
* it will be converted to an Arrow decimal128 that has the widest-precision the cudf decimal type
zeroshade marked this conversation as resolved.
Show resolved Hide resolved
* supports. For example, numeric::decimal32 will be converted to Arrow decimal128 of the precision
* 9 which is the maximum precision for 32-bit types. Similarly, numeric::decimal128 will be
* converted to Arrow decimal128 of the precision 38.
*/
nanoarrow::UniqueSchema to_arrow_schema(cudf::table_view const& input,
std::vector<column_metadata> const& metadata);

/**
* @brief Create `ArrowDeviceArray` from cudf table and metadata
*
* Populates the C struct ArrowDeviceArray without performing if possible.
* This maintains the data on the GPU device and gives ownership of the table
* and its buffers to the ArrowDeviceArray struct.
*
* After calling this function, the release callback on the returned ArrowDeviceArray
* must be called to clean up the memory.
*
* @note For decimals, since the precision is not stored for them in libcudf
* it will be converted to an Arrow decimal128 with the widest-precision the cudf decimal type
* supports. For example, numeric::decimal32 will be converted to Arrow decimal128 of the precision
* 9 which is the maximum precision for 32-bit types. Similarly, numeric::decimal128 will be
* converted to Arrow decimal128 of the precision 38.
*
* @note Copies will be performed in the cases where cudf differs from Arrow
* such as in the representation of bools (Arrow uses a bitmap, cudf uses 1-byte per value).
*/
zeroshade marked this conversation as resolved.
Show resolved Hide resolved
struct ArrowDeviceArray to_arrow_device(
cudf::table&& table,
rmm::cuda_stream_view stream = cudf::get_default_stream(),
rmm::mr::device_memory_resource* mr = rmm::mr::get_current_device_resource());
vyasr marked this conversation as resolved.
Show resolved Hide resolved

/**
* @brief Create `cudf::table` from given arrow Table input
*
Expand Down
Loading
Loading