[BUG] Fail to convert the data to arrow format when there is a child column of struct type.

[firestarman]

Describe the bug
The cudf is complaining the error as below when writing a child column of struct type to an arrow table.

Caused by: ai.rapids.cudf.CudfException: cuDF failure at: /home/liangcail/work/projects/on_github/cudf/cpp/src/interop/to_arrow.cpp:207: Number of field names and number of children doesn't match

	at ai.rapids.cudf.Table.convertCudfToArrowTable(Native Method)
	at ai.rapids.cudf.Table.access$1500(Table.java:46)
	at ai.rapids.cudf.Table$ArrowIPCTableWriter.write(Table.java:1010)

Steps/Code to reproduce bug
Apply this patch and run the test testArrowIPCWriteToBufferChunked .

cd <cudf_root>/java
mvn test -Dtest=ai.rapids.cudf.TableTest#testArrowIPCWriteToBufferChunked

Expected behavior
The test applied the patch above should pass.

[firestarman]

It the below a reasonable fix (It works)? Since I see there is a simliar check here https://github.com/rapidsai/cudf/blob/branch-0.19/cpp/src/interop/to_arrow.cpp#L248 for list_view .

diff --git a/cpp/src/interop/to_arrow.cpp b/cpp/src/interop/to_arrow.cpp
index 7daffc1a3c..bf269ebfe1 100644
--- a/cpp/src/interop/to_arrow.cpp
+++ b/cpp/src/interop/to_arrow.cpp
@@ -202,20 +202,22 @@ std::shared_ptr<arrow::Array> dispatch_to_arrow::operator()<cudf::struct_view>(
   arrow::MemoryPool* ar_mr,
   rmm::cuda_stream_view stream)
 {
-  CUDF_EXPECTS(metadata.children_meta.size() == static_cast<std::size_t>(input.num_children()),
+  auto children_meta =
+    metadata.children_meta.empty() ? std::vector<column_metadata>(input.num_children()) : metadata.children_meta;
+  CUDF_EXPECTS(children_meta.size() == static_cast<std::size_t>(input.num_children()),
                "Number of field names and number of children doesn't match\n");
   std::unique_ptr<column> tmp_column = nullptr;
 
   if (input.offset() != 0) { tmp_column = std::make_unique<cudf::column>(input); }
 
   column_view input_view = (tmp_column != nullptr) ? tmp_column->view() : input;
-  auto child_arrays      = fetch_child_array(input_view, metadata.children_meta, ar_mr, stream);
+  auto child_arrays      = fetch_child_array(input_view, children_meta, ar_mr, stream);
   auto mask              = fetch_mask_buffer(input_view, ar_mr, stream);
 
   std::vector<std::shared_ptr<arrow::Field>> fields;
   std::transform(child_arrays.cbegin(),
                  child_arrays.cend(),
-                 metadata.children_meta.cbegin(),
+                 children_meta.cbegin(),
                  std::back_inserter(fields),
                  [](auto const array, auto const meta) {

[jrhemstad]

@rgsl888prabhu I believe you implemented this originally. Could you take a look?

[rgsl888prabhu]

@firestarman When you try to read the struct column written with the patch you have provided, what are the names of children/struct members. Would it be empty ?

[firestarman]

@firestarman When you try to read the struct column written with the patch you have provided, what are the names of children/struct members. Would it be empty ?

@rgsl888prabhu Yes, i believe it is empty here. Are the names necessary for this operation ? Or could it be empty ? If names are required, we may need to fix this from JNI side.

[firestarman]

I add the support in JNI to carry the column names onto native by this change, however seems native to_arrow does NOT support array of struct?

@rgsl888prabhu Could you help confirm whether to_arrow support the column with array of struct type ?
If not, shall we going to support it ?

[rgsl888prabhu]

to_arrow supports array of structs, here are some tests that were added for structs

cudf/cpp/tests/interop/to_arrow_test.cpp

Line 266 in 5fea6ad

TEST_F(ToArrowTest, StructColumn)

.

For structs, it is better to have name for members which is required for accessing the element, else it would be same as list of list.

What error are you getting currently with the changes you mentioned ?

[firestarman]

Thanks, i made the PR #7598 to close this issue.