[BUG] dask-cudf fails to read ORC files created by Spark

[randerzander]

I'm trying to use dask-cudf to read ORC files generated by a Spark job.

from pyspark.sql import SparkSession
spark = SparkSession \
    .builder \
    .getOrCreate()

df = spark.createDataFrame(
    [
        (1, 1.0),
        (2, 2.0),
    ],
    ['id', 'doubleCol']
)
df.write.mode('overwrite').orc('test')

The above snippet results in 4 files in test:

ls test
part-00000-ddf3787b-958c-4c57-9968-f3afdec1eb7e-c000.snappy.orc  part-00079-ddf3787b-958c-4c57-9968-f3afdec1eb7e-c000.snappy.orc
part-00039-ddf3787b-958c-4c57-9968-f3afdec1eb7e-c000.snappy.orc  _SUCCESS

import dask_cudf

dask_cudf.read_orc('test/*.orc')

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-74c488c779a0> in <module>
      1 import dask_cudf
      2 
----> 3 dask_cudf.read_orc('test/*.orc')

~/conda/envs/rapids/lib/python3.8/site-packages/dask_cudf/io/orc.py in read_orc(path, columns, filters, storage_options, **kwargs)
     86 
     87     with fs.open(paths[0], "rb") as f:
---> 88         meta = cudf.read_orc(f, stripes=[0], columns=columns, **kwargs)
     89 
     90     name = "read-orc-" + tokenize(fs_token, path, columns, **kwargs)

~/conda/envs/rapids/lib/python3.8/site-packages/cudf/io/orc.py in read_orc(filepath_or_buffer, engine, columns, filters, stripes, skiprows, num_rows, use_index, timestamp_type, **kwargs)
    258     if engine == "cudf":
    259         df = DataFrame._from_table(
--> 260             liborc.read_orc(
    261                 filepath_or_buffer,
    262                 columns,

cudf/_lib/orc.pyx in cudf._lib.orc.read_orc()

cudf/_lib/orc.pyx in cudf._lib.orc.read_orc()

RuntimeError: cuDF failure at: ../src/io/orc/orc.cpp:478: Invalid stripe index

The ORC files in test can all be read individually by cudf.read_orc. Note that some of the files have 0 rows. It seems like dask-cudf is failing when attempting to read an empty ORC file.

This does not happen with Parquet files.

[rjzamora]

Thanks for raising this @randerzander - Looks like dask_cudf may failing to generate metadata from an empty file. I should be able to look into this tomorrow :)