[BUG] integration test test_cast_nested can fail with non-empty nulls

[revans2]

Describe the bug
To be able to support sorting and comparisons on nested types we needed to get rid of all non-empty nulls. To help with this we added in some asserts when we see them. It looks like the Cast implementation is using mergeAndSetValidity which is producing an exception in this case.

Steps/Code to reproduce bug
Run the integration tests, but revert #8183 first

E                   : org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 1165.0 failed 1 times, most recent failure: Lost task 3.0 in stage 1165.0 (TID 3750) (10.28.9.123 executor driver): java.lang.AssertionError: Column has non-empty nulls
E                       at ai.rapids.cudf.AssertEmptyNulls.assertNullsAreEmpty(AssertEmptyNulls.java:33)
E                       at ai.rapids.cudf.ColumnView.<init>(ColumnView.java:71)
E                       at ai.rapids.cudf.ColumnVector.<init>(ColumnVector.java:58)
E                       at ai.rapids.cudf.ColumnView.mergeAndSetValidity(ColumnView.java:835)
E                       at com.nvidia.spark.rapids.GpuCast$.$anonfun$castStructToString$5(GpuCast.scala:903)
E                       at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:29)
E                       at com.nvidia.spark.rapids.GpuCast$.$anonfun$castStructToString$1(GpuCast.scala:903)
E                       at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:65)
E                       at com.nvidia.spark.rapids.GpuCast$.doCastStructToString$1(GpuCast.scala:877)
E                       at com.nvidia.spark.rapids.GpuCast$.$anonfun$castStructToString$9(GpuCast.scala:916)
E                       at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:47)
E                       at com.nvidia.spark.rapids.GpuCast$.$anonfun$castStructToString$7(GpuCast.scala:914)
E                       at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:47)
E                       at com.nvidia.spark.rapids.GpuCast$.castStructToString(GpuCast.scala:909)
E                       at com.nvidia.spark.rapids.GpuCast$.doCast(GpuCast.scala:253)
E                       at com.nvidia.spark.rapids.GpuCast$.$anonfun$doCast$24(GpuCast.scala:471)
E                       at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:29)
E                       at com.nvidia.spark.rapids.GpuCast$.doCast(GpuCast.scala:469)
E                       at com.nvidia.spark.rapids.GpuCast.doColumnar(GpuCast.scala:1567)

Expected behavior
This should not happen, we either need to fix mergeAndSetValidity, which I thought we had done, but it might have been bitwiseMergeAndSetValidity. Not sure if we have two or not. Or we need to #7485 + #7698

This is likely very closely related to #8185

[revans2]

Other tests are also failing because of cast using mergeAndSetValidity

• test_cast_array_to_string
• test_cast_map_to_string
• test_two_col_struct_legacy_cast
• test_one_nested_null_field_legacy_cast
• test_struct_cast_groupby_count
• test_struct_count_distinct_cast
• test_cast_struct_to_string

Possibly more, but I will just leave it with this for now.

[razajafri]

closed by rapidsai/cudf#13335