[BUG] JSON mixed_types_as_strings feature incorrectly returns some structs as strings

[andygrove]

Describe the bug

When reading JSON containing structs where some fields are optional and when the mixed_types_as_strings feature is enabled, the data is returned as a string column instead of a struct column.

Steps/Code to reproduce bug

Input file:

{ "teacher": "Bob" }
{ "student": { "name": "Carol", "age":  21 } }
{ "teacher": "Bob", "student": { "name": "Carol", "age": 21 } }

Java test code:

  Schema schema = Schema.builder()
          .column(DType.STRING, "teacher")
          .column(DType.STRUCT, "student")
          .build();
  JSONOptions opts = JSONOptions.builder()
          .withLines(true)
          .withMixedTypesAsStrings(true)
          .build();
  Table table = Table.readJSON(schema, opts, TEST_STRUCTS_JSON);
  System.out.println(table.getColumn(0).getType());

This outputs STRING instead of STRUCT.

Expected behavior
I would not consider this input to be "mixed types". We expect this to be returned as a struct column.

Environment overview (please complete the following information)
N/A

Environment details
N/A

Additional context

[GregoryKimball]

@karthikeyann Would you please take a look?

[karthikeyann]

In this example, "teacher" column appears first. so, that will be first column, which is a string column. The struct column is index 1.
System.out.println(table.getColumn(1).getType()); prints STRUCT.
(tested via spark unit test, and in cudf-python)

[andygrove]

Thanks @karthikeyann. I have either misunderstood something or my repro case is not correct. I will take a look.

[andygrove]

I found another issue while working on a better repro.

    Schema schema = Schema.builder()
            .column(DType.STRUCT, "student")
            .build();
    JSONOptions opts = JSONOptions.builder()
            .withLines(true)
            .withMixedTypesAsStrings(true)
            .build();
    Table table = Table.readJSON(schema, opts, TestUtils.getResourceAsFile("structs.json"));

{"teacher": "Ntflgt","student": {"name": "Odhut", "age": 10}}
{"teacher": "Pnjugo","student": {"name": "Xqnqpg", "age": 13}}
{"student":null}

ai.rapids.cudf.CudfException: CUDF failure at: /home/andy/git/nvidia/cudf/cpp/include/cudf/column/column_factories.hpp:342: Invalid, non-fixed-width type.
        at ai.rapids.cudf.Table.readJSON(Native Method)

[andygrove]

I now have an accurate repro:

@Test
void testMixedTypes() throws IOException {
  MultiBufferDataSource source = sourceFrom(
          TestUtils.getResourceAsFile("structs.json"));
  JSONOptions opts = JSONOptions.builder()
          .withLines(true)
          .withMixedTypesAsStrings(true)
          .build();
  TableWithMeta table = Table.readJSON(opts, source.getHostBuffers()[0],
          0, source.size());
  Table t = table.releaseTable();
  for (int i=0; i<t.getNumberOfColumns(); i++) {
    System.out.println("TYPE " + i + ": " + t.getColumn(i).getType());
  }
  TableDebug.builder().build().debug("TABLE", t);
}

Note that this requires adding a method to MultiBufferDataSource:

public HostMemoryBuffer[] getHostBuffers() {
  return hostBuffers;
}

Test data:

{"teacher": "Ntflgt","student": {"name": "Odhut", "age": 10}}
{"teacher": "Pnjugo","student": {"name": "Xqnqpg", "age": 13}}
{"student":null}
{"teacher": "Ntflgt","student": {"name": "Odhut", "age": 10}}
{"teacher": "Pnjugo","student": {"name": "Xqnqpg", "age": 13}}

Output:

TYPE 0: STRING
TYPE 1: STRING
DEBUG TABLE Table{columns=[ColumnVector{rows=5, type=STRING, nullCount=Optional.empty, offHeap=(ID: 40057 7fe71c588380)}, ColumnVector{rows=5, type=STRING, nullCount=Optional.empty, offHeap=(ID: 40058 7fe7275745d0)}], cudfTable=140630589746128, rows=5}
GPU COLUMN 0 - NC: 1 DATA: DeviceMemoryBufferView{address=0x7fe746001600, length=24, id=-1} VAL: DeviceMemoryBufferView{address=0x7fe746001500, length=64, id=-1}
COLUMN 0 - STRING
0 "Ntflgt" 4e74666c6774
1 "Pnjugo" 506e6a75676f
2 NULL
3 "Ntflgt" 4e74666c6774
4 "Pnjugo" 506e6a75676f
GPU COLUMN 1 - NC: 1 DATA: DeviceMemoryBufferView{address=0x7fe746001800, length=114, id=-1} VAL: DeviceMemoryBufferView{address=0x7fe746001f00, length=64, id=-1}
COLUMN 1 - STRING
0 "{"name": "Odhut", "age": 10}" 7b226e616d65223a20224f64687574222c2022616765223a2031307d
1 "{"name": "Xqnqpg", "age": 13}" 7b226e616d65223a202258716e717067222c2022616765223a2031337d
2 NULL
3 "{"name": "Odhut", "age": 10}" 7b226e616d65223a20224f64687574222c2022616765223a2031307d
4 "{"name": "Xqnqpg", "age": 13}" 7b226e616d65223a202258716e717067222c2022616765223a2

The student column here is returned as a string instead of a struct.

[karthikeyann]

Fix available with PR #14939
tested the repro case in python

In [5]: df
Out[5]: 
  teacher                        student
0  Ntflgt   {'name': 'Odhut', 'age': 10}
1  Pnjugo  {'name': 'Xqnqpg', 'age': 13}
2    <NA>                           None
3  Ntflgt   {'name': 'Odhut', 'age': 10}
4  Pnjugo  {'name': 'Xqnqpg', 'age': 13}

In [6]: df.info()
<class 'cudf.core.dataframe.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 2 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   teacher  4 non-null      object
 1   student  4 non-null      struct
dtypes: object(1), struct(1)
memory usage: 390.0+ bytes