[BUG] Conversion of list column to pandas produces array(..., dtype=object) entries

[wence-]

Describe the bug

import cudf
import pandas as pd

s = cudf.Series([[["abc", "def"]]])
expect = pd.Series([[["abc", "def"]]])
got = s.to_pandas()

I would expect that the entries of got.values are the same as expect.values. However, type(got.values[0]) is numpy.ndarray whereas type(expect.values[0]) is list.

In and of itself this doesn't seem to be a big problem, since cudf.from_pandas(expect) produces the same as cudf.from_pandas(got). But, IO of the two pandas objects (specifically CSV IO) produces different results, since print of a numpy.ndarray produces something different to printing a list.

Consequently, I now can't round-trip through to_csv/read_csv

from io import StringIO
import cudf
import pandas as pd

s = cudf.Series([[["abc", "def"]]])
expect = pd.Series([[["abc", "def"]]])
got = s.to_pandas()

ebuf = StringIO()
gbuf = StringIO()

expect.to_frame().to_csv(ebuf, header=False, index=False)
got.to_frame().to_csv(gbuf, header=False, index=False)

ebuf.seek(0)
gbuf.seek(0)

expect_io = pd.read_csv(ebuf, index_col=None, header=None)
got_io = pd.read_csv(gbuf, index_col=None, header=None)

print(expect.to_frame())
              0
0  [[abc, def]]
print(expect_io)
                  0
0  [['abc', 'def']]
print(got.to_frame())
              0
0  [[abc, def]]
print(got_io)
                                       0
0  [array(['abc', 'def'], dtype=object)]

Expected behavior

I think this should work.

The way to_pandas() is implemented, we just go via arrow and then call to_pandas() on the arrow array. Plausibly this is therefore a bug in arrow, but I am not sure, converting a ListArray to a series of numpy array objects. I suspect that, at least in the short term, we'll need to patch up ListColumn.to_pandas() to post-process the result from arrow.