[BUG] JSON reader fails to parse files with empty rows

[vuule]

The following test fails in both cases:

@pytest.mark.parametrize(
    "buffer",
    [
        "[ ]\n[ ]",
        '{ }\n{ }',
    ],
)
def test_json_empty(buffer):
    cu_df = cudf.read_json(buffer, lines=True)
    pd_df = pd.read_json(buffer, lines=True)

    np.testing.assert_array_equal(pd_df.dtypes, cu_df.dtypes)

With array rows, the reader creates a table with int8 column instead of an empty table.
With object rows, a CUDA error happens.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[dagardner-nv]

I can't reproduce this, however with cudf 23.02.00 I have a similar issue where cudf is failing to parse an empty json array '[]' I get an re.error exception, while an empty list with a blank space in the body '[ ]' parses regardless of the engine value:

import pandas as pd
import cudf

print(cudf.__version__)
print(pd.__version__)

try:
    cudf.read_json('[]', lines=False, engine='cudf')
except Exception as e:
    print(e)

try:
    cudf.read_json('[]', lines=False, engine='pandas')
except Exception as e:
    print(e)

# Pandas works fine
print(pd.read_json('[]', lines=False))

# Blank space works fine
print(cudf.read_json('[ ]', lines=False, engine='cudf'))
import pandas as pd
import cudf

print(cudf.__version__)
print(pd.__version__)

try:
    cudf.read_json('[]', lines=False, engine='cudf')
except Exception as e:
    print(e)

try:
    cudf.read_json('[]', lines=False, engine='pandas')
except Exception as e:
    print(e)

# Pandas works fine
print(pd.read_json('[]', lines=False))

# Blank space works fine
print(cudf.read_json('[ ]', lines=False, engine='cudf'))

output:

23.02.00
1.3.5
unterminated character set at position 31
/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/cudf/io/json.py:121: UserWarning: Using CPU via Pandas to read JSON dataset, this may be GPU accelerated in the future
  warnings.warn(
unterminated character set at position 31
Empty DataFrame
Columns: []
Index: []
Empty DataFrame
Columns: []
Index: []

[GregoryKimball]

Thank you @dagardner-nv for sharing this. This is an issue we had missed in the overall JSON prioritization and it seems to impact our new nested JSON reader.

[karthikeyann]

This error does not happen in new nested JSON reader anymore.
Only difference I see is that, Index: [0, 1] in pandas, Index: [] in cudf.

Question: is it possible to have empty column with non-empty indices in cudf Dataframe?

[vuule]

Question: is it possible to have empty column with non-empty indices in cudf Dataframe?

@galipremsagar

[galipremsagar]

Question: is it possible to have empty column with non-empty indices in cudf Dataframe?

@galipremsagar

Yup, it's possible to have a non-empty index when no columns exists:

In [4]: cudf.DataFrame()
Out[4]: 
Empty DataFrame
Columns: []
Index: []

In [5]: cudf.DataFrame(index=[1, 2, 3])
Out[5]: 
Empty DataFrame
Columns: []
Index: [1, 2, 3]

But if a column exists and non-empty index exist, the column will be the length of index:

In [6]: cudf.DataFrame(index=[1, 2, 3], columns=['a', 'b'])
Out[6]: 
      a     b
1  <NA>  <NA>
2  <NA>  <NA>
3  <NA>  <NA>

[revans2]

Spark has a similar issue, but it is a bit different. If we are trying to read a JSON lines file with something like.

{}
{}

in it, then we get back a TableWithMeta that is empty, but we have no idea how many rows there were in the original input. This is even more problematic if we have something like

{}

{}

Where empty lines are supposed to be filtered out. All I need is the number of rows that was read.

[vyasr]

I think all of the cases that previously caused problems now work as expected.

In [1]: import cudf
   ...: import pandas as pd
   ...: import numpy as np                                                                                                                                                                                     ...:
   ...: buffers = ["[ ]\n[ ]", '{ }\n{ }']
   ...: for buffer in buffers:
   ...:     cu_df = cudf.read_json(buffer, lines=True)
   ...:     pd_df = pd.read_json(buffer, lines=True)
   ...:     np.testing.assert_array_equal(pd_df.dtypes, cu_df.dtypes)
   ...:
   ...:                                                                                                                                                                                                        ...: print(cudf.read_json('[]', lines=False, engine='cudf'))
   ...: print(cudf.read_json('[]', lines=False, engine='pandas'))
   ...: print(pd.read_json('[]', lines=False))
   ...: print(cudf.read_json('[ ]', lines=False, engine='cudf'))
/home/coder/cudf/python/cudf/cudf/utils/ioutils.py:1745: FutureWarning: Passing literal json to read_json is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  warnings.warn(
<ipython-input-1-1c788c1a77cf>:8: FutureWarning: Passing literal json to 'read_json' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  pd_df = pd.read_json(buffer, lines=True)
/home/coder/cudf/python/cudf/cudf/utils/ioutils.py:1733: FutureWarning: Passing literal json to read_json is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  warnings.warn(
<ipython-input-1-1c788c1a77cf>:8: FutureWarning: Passing literal json to 'read_json' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  pd_df = pd.read_json(buffer, lines=True)
/home/coder/cudf/python/cudf/cudf/utils/ioutils.py:1745: FutureWarning: Passing literal json to read_json is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  warnings.warn(
Empty DataFrame
Columns: []
Index: []
/home/coder/cudf/python/cudf/cudf/io/json.py:108: UserWarning: Using CPU via Pandas to read JSON dataset, this may be GPU accelerated in the future
  warnings.warn(
/home/coder/cudf/python/cudf/cudf/io/json.py:130: FutureWarning: Passing literal json to 'read_json' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  pd_value = pd.read_json(
Empty DataFrame
Columns: []
Index: []
<ipython-input-1-1c788c1a77cf>:14: FutureWarning: Passing literal json to 'read_json' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  print(pd.read_json('[]', lines=False))
Empty DataFrame
Columns: []
Index: []
/home/coder/cudf/python/cudf/cudf/utils/ioutils.py:1745: FutureWarning: Passing literal json to read_json is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  warnings.warn(
Empty DataFrame
Columns: []
Index: []

I'm going to close for now, but feel free to reopen if I missed something.

[revans2]

This is still happening. I put in a work around into the java API, but it is not a good solution to the problem.

[karthikeyann]

@galipremsagar What information does cython/python layer need from libcudf read_json to ensure you can create index columns even if there are no columns in result table?

[ttnghia]

The input in the example above actually does not contain "empty rows". They are "empty JSON objects/arrays" instead. Empty rows should mean only whitespace, like " "\n" ".

With recent changes in JSON reader, I believe that this should be fixed (at least for the case of empty JSON objects). Please confirm again.

[karthikeyann]

It's fixed for spark because passing schema with prune columns will create null columns (of size number of rows).
For python, it's still an issue, where number of rows is not returned for an empty table.
One option is to return number of rows as metadata num_rows_per_source.