Make map column non-nullable when it's a key in another map.

[res-life]

fixes #9129

Fix map column can not be non-nullable.
Depends on rapidsai/cudf#14003.

[revans2]

Why do we need to set isMapKey instead of just going off of nullable? We set nullable to false specifically each place we set isMapKey to true.

[res-life]

Consider 2 scenarios.
First scenario:

struct(
  field(column_1, map(int, int), nullable = true),
  field(column_1, map(int, int), nullable = false)
)

We should honor the nullable in the else branch

if (isMapKey) false else nullable

Second scenario:

struct(
  field(column_1, map1(map2(int, int), int), nullable = true)
)

The isMapKey means this map is a map key in another map?.
The isMapKey here means the m: MapType is a map key of another map, and its type is also Map.
We can not know isMapKey unless that the outer writerOptionsFromField passes it in.
Consider the map1(map2(int, int), int), the logic is from outer to inner.
writerOptionsFromField first handles map1, then handle its (key(map2(int, int)), value), here isMapKey means (map2(int, int) is key of map1.
writerOptionsFromField handles map2(int, int) (the key of outer map1), isMapKey is true, isMapKey is passed in from outer writerOptionsFromField.
writerOptionsFromField handles int (the value of outer map1)

[revans2]

I don't think you understand what I am saying. isMapKey is only ever set to true on line 296. On line 295 nullable is also always set to false for this same method call.

              nullable = false,
              writeInt96, fieldMeta, parquetFieldIdWriteEnabled, isMapKey = true).build()

There is never a situation if the current code when

is (isMapKey) false else nullable

will return anything different from nullable by itself. If it did, then it would be an error in the code, because all map keys, no matter the type, are required to never be nullable. There is not even an option in the Spark MapType to make them nullable, which is why line 295 is hard coded.

[res-life]

Thanks for your detailed review. Fixed.
And also tested succefully with rapidsai/cudf#14003.
Once this PR is approved, I'll merge rapidsai/cudf#14003 and this PR.

[res-life]

The test code is in: #9090

  test("Statistics tests for Parquet files written by GPU, map(map)") {
    assume(false, "https://github.com/NVIDIA/spark-rapids/issues/9129")
    assume(true, "Move to scale test")

After this PR is merged, I'll update #9090

[res-life]

build

[res-life]

build

[res-life]

build