[BUG] cudf::experimental::read_json returns column with corrupted data when the input is invalid

[ttnghia]

From the current json test:

TEST_F(JsonReaderTest, JsonNestedDtypeSchema)

The test specifies an input:

std::string json_string = R"( [{"a":[123, {"0": 123}], "b":1.0}, {"b":1.1}, {"b":2.1}])";

Notice that the input for list a (lists of structs) is invalid: the first element is 123, which is not a struct. As such, the test expects the result of parsing the first list a to be [null, 123.0] which is reasonable.

However, if at the end of the test, we print out the result by:

cudf::test::print(result.tbl->get_column(0));

then we will get this:

List<Struct<float,>>:
Length : 3
Offsets : 0, 1, 1, 1
Null count: 2
001
   Struct<float,>:
   Length : 2:
   Null count: 1
   10
      10
      NULL, 123

The output column seems to have corrupted data. In particular, it is a lists column having only one non-null row (the first row), which in turn has only one element (offsets 0, 1]). However, the child column has 2 elements.

To overcome this bug, I temporarily disable the JsonReaderTest#JsonNestedDtypeSchema test in #12370. It should be enabled back after fixing this.

[ttnghia]

CC @karthikeyann.

Also CC @GregoryKimball.

[GregoryKimball]

Thank you @ttnghia for identifying this issue. It seems that reading this string from #11682 in the Nested JSON reader results in an unsanitized column. @elstehle do you think this was an issue also in the host-side tree algorithms and column creation, or something introduced as part of the device-side tree algorithms that @karthikeyann developed?

It would be better to not materialize a column with unsanitized nulls, but if that is difficult we could always run the sanitization function on the columns before returning them.

[karthikeyann]

Actual List offsets are [1, 2, 2, 2].
Column printer in test utilities subtracts the first offset with rest of the offsets. so, it prints [0, 1, 1, 1].

This issue arises due to mixed types in lists, and it is not considered invalid input. Specification is,

If string/value type is mixed with struct, the string/value locations are considered null.

Issue is in list offset device-side algorithm.
Column deduction infers 123 as seperate column, and {"0": 123} struct as seperate column. Parsing of 123 is ignored, but the list offsets considered {"0": 123} as first item. so, it wrote its offset 1 to lists's first offset.
I changed the device-side algorithm to depend on parent_col_id instead of col_id. bug fix in the works now.

Also, this issue shows the need for more test cases for list types. (// TODO comment in list offset algorithm). More tests will be added as part of the fix.