[BUG] dask-cudf.to_parquet(write_metadata_file=True, append=True) fails

[ayushdg]

Describe the bug
When writing a dask-cudf data frame with the append=True and write_metadata_file=True flag set, the operation seems to fail when trying to merge metadata files.

Steps/Code to reproduce bug

import dask_cudf
import cudf

df = cudf.DataFrame({"a":[1,2,3]})
ddf = dask_cudf.from_cudf(df,1)
ddf.to_parquet("test.parquet", append=True, write_metadata_file=True, write_index=False)

ddf.to_parquet("test.parquet", append=True, write_metadata_file=True, write_index=False)

File /datasets/adattagupta/mambaforge/envs/curator-latest/lib/python3.10/site-packages/dask_cudf/io/parquet.py:423, in CudfEngine.write_metadata(parts, fmd, fs, path, append, **kwargs)
    420     _meta = [fmd]
    421 _meta.extend([parts[i][0]["meta"] for i in range(len(parts))])
    422 _meta = (
--> 423     cudf.io.merge_parquet_filemetadata(_meta)
    424     if len(_meta) > 1
    425     else _meta[0]
    426 )
    427 with fs.open(metadata_path, "wb") as fil:
    428     fil.write(memoryview(_meta))

File /datasets/adattagupta/mambaforge/envs/curator-latest/lib/python3.10/site-packages/cudf/io/parquet.py:1109, in merge_parquet_filemetadata(filemetadata_list)
   1105 @ioutils.doc_merge_parquet_filemetadata()
   1106 def merge_parquet_filemetadata(filemetadata_list):
   1107     """{docstring}"""
-> 1109     return libparquet.merge_filemetadata(filemetadata_list)

File parquet.pyx:829, in cudf._lib.parquet.merge_filemetadata()

File parquet.pyx:842, in cudf._lib.parquet.merge_filemetadata()

File <stringsource>:47, in vector.from_py.__pyx_convert_vector_from_py_uint8_t()

TypeError: 'pyarrow._parquet.FileMetaData' object is not iterable

Expected behavior
No errors

Environment overview (please complete the following information)

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
• Method of cuDF install: [conda, Docker, or from source]
  • If method of install is [Docker], provide docker pull & docker run commands used

Environment details
Tested with cudf/dask-cudf 24.8 though it probably persists in newer releases as well.

Additional context
Add any other context about the problem here.

[vyasr]

@rjzamora it looks like this issue has to do with how the pyarrow metadata is being handled at the dask layer when passed down to cudf, yeah?

[rjzamora]

@rjzamora it looks like this issue has to do with how the pyarrow metadata is being handled at the dask layer when passed down to cudf, yeah?

Yeah, we are passing down list[pq.FileMetaData, bytes], and cudf is expecting list[bytes, ...]. I can submit a fix.