[FEA] Support S3 writes in chunked writer - ParquetDatasetWriter

[sauravdev]

Parquet writes to external storage such as s3 should be possible using ParquetDatasetWriter, right now it errors out.

[GregoryKimball]

Thanks @sauravdev for sharing this use case. Would you please post a code sample and the error message, if any?

[sauravdev]

Sure, below is sample code

stream = Stream.from_kafka_batched(
    kafka_input_topic, consumer_conf,asynchronous=True,
    poll_interval="60s", max_batch_size=90000,
    engine="cudf", dask=True
) 

from cudf.io.parquet import ParquetDatasetWriter

def cudf_passthrough_to_parquet_chunk_test(gdf):
    batch_process_start_time = time.time()
    size = len(gdf)
    with ParquetDatasetWriter("s3://samplepath/", partition_cols=["samplecol"], index=False) as cw:
        cw.write_table(gdf)
    batch_process_finish_time = time.time()
    return (batch_process_start_time, batch_process_finish_time, size)

output = stream.map(cudf_passthrough_to_parquet_chunk_test).gather().sink_to_list()
stream.start()
output

and below is error:
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <zmq.eventloop.ioloop.ZMQIOLoop object at 0x7f27e7e25e50>>, <Future finished exception=RuntimeError('cuDF failure at: ../src/io/utilities/data_sink.cpp:36: Cannot open output file')>)