[BUG] Calling to_pandas() on a dask dataframe containing a 'struct' column can result in a raised PyArrow exception

[drobison00]

Describe the bug
Calling to_pandas() on a dask dataframe created from a cuDF dataframe containing a 'struct' column can result in a raised PyArrow exception:

/opt/anaconda/envs/morpheus/lib/python3.10/site-packages/cudf/core/column/struct.py:56: in to_arrow
    return pa.StructArray.from_buffers(
pyarrow/array.pxi:1110: in pyarrow.lib.Array.from_buffers
    ???
pyarrow/array.pxi:1504: in pyarrow.lib.Array.validate
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Buffer #0 too small in array of type struct<> and length 10: expected at least 2 byte(s), got 0

pyarrow/error.pxi:100: ArrowInvalid

Steps/Code to reproduce bug

Using this example file azure_ad_logs.json: https://github.com/nv-morpheus/Morpheus/blob/branch-23.07/tests/tests_data/azure_ad_logs.json

def test_cudf_struct_type_conversion_dask():
    import cudf
    import dask_cudf

    cdf = cudf.read_json("azure_ad_logs.json")
    ddf = dask_cudf.from_cudf(cdf, npartitions=1)

    result = ddf['properties'].compute().to_pandas()

    print(result)

Expected behavior
A pandas dataframe should be returned.

Environment overview (please complete the following information)
Conda environment

grep 'cudf|arrow|dask'
arrow                     1.2.3                    pypi_0    pypi
arrow-cpp                 10.0.1          ha770c72_14_cuda    conda-forge
cudf                      23.02.00        cuda_11_py310_g5ad4a85b9d_0    rapidsai
dask                      2023.1.1           pyhd8ed1ab_0    conda-forge
dask-core                 2023.1.1           pyhd8ed1ab_0    conda-forge
dask-cuda                 23.2.1             pyhd8ed1ab_1    conda-forge
dask-cudf                 23.02.00        cuda_11_py310_g5ad4a85b9d_0    rapidsai
libarrow                  10.0.1          h255618e_14_cuda    conda-forge
libcudf                   23.02.00        cuda11_g5ad4a85b9d_0    rapidsai
pyarrow                   10.0.1          py310hc81d9b2_14_cuda    conda-forge

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Additional context
Add any other context about the problem here.

[shwina]

Thanks for reporting, @drobison00 -- is there any chance you're able to share the file azure_ad_logs.json? Even better would be if it's possible to reproduce this error from a much smaller dataset or by hand-constructing the inputs but that's not necessary. Thanks!

I apologize! I missed that you dropped a link to it :)

[wence-]

Here is a more minimal reproducer, it appears to be some interaction between a column of empty structs with nulls and the way dask_cudf initialises the data.

import cudf
import dask_cudf
s = cudf.Series([None, {}])
ds = dask_cudf.from_cudf(s, npartitions=1)
ds.compute()

[wence-]

OK, this is a bug slicing struct columns:

import cudf
s = cudf.Series([None, {}])
s.iloc[0:2] # => `ArrowInvalid` error

[wence-]

The null mask returned from libcudf's slice for a struct column with zero children is incorrectly sized as 0 bytes...

The null mask wrapped up by the cython layer's from_column_view is incorrectly sized as zero bytes.