[BUG] Integration test test_window_running_no_part can produce non-empty nulls (cudf scan)

[revans2]

Describe the bug
To be able to support sorting and comparisons on nested types we needed to get rid of all non-empty nulls. To help with this we added in some asserts when we see them. It looks like the ColumnView.scan in some cases can return non-empty nulls.

Steps/Code to reproduce bug
Run the integration tests, but revert #8183 first

E                   py4j.protocol.Py4JJavaError: An error occurred while calling o362293.collectToPython.
E                   : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9329.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9329.0 (TID 37178) (10.28.9.123 executor driver): java.lang.Assert
ionError: Column has non-empty nulls
E                       at ai.rapids.cudf.AssertEmptyNulls.assertNullsAreEmpty(AssertEmptyNulls.java:33)
E                       at ai.rapids.cudf.ColumnView.<init>(ColumnView.java:71)
E                       at ai.rapids.cudf.ColumnVector.<init>(ColumnVector.java:58)
E                       at ai.rapids.cudf.ColumnView.scan(ColumnView.java:1712)
E                       at com.nvidia.spark.rapids.GroupedAggregations.$anonfun$doRunningWindowScan$4(GpuWindowExec.scala:925)
E                       at com.nvidia.spark.rapids.GroupedAggregations.$anonfun$doRunningWindowScan$4$adapted(GpuWindowExec.scala:918)
E                       at scala.collection.immutable.Range.foreach(Range.scala:158)
E                       at com.nvidia.spark.rapids.GroupedAggregations.$anonfun$doRunningWindowScan$3(GpuWindowExec.scala:918)
E                       at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:65)
E                       at com.nvidia.spark.rapids.GroupedAggregations.$anonfun$doRunningWindowScan$1(GpuWindowExec.scala:917)
E                       at com.nvidia.spark.rapids.GroupedAggregations.$anonfun$doRunningWindowScan$1$adapted(GpuWindowExec.scala:908)

This line corresponds to doing a Scan, specifically
inputCol.scan(agg, ScanType.INCLUSIVE, NullPolicy.EXCLUDE)

I don't know the exact agg being done, but it is being done where the data type is a String.

[revans2]

I should add that this is likely a bug in CUDF that needs to be fixed.

[razajafri]

@ttnghia you suggested that cudf has removed all instances of creating non-empty nulls. Is this on the cudf agenda?
how do you want to proceed on this?
@revans2 should we handle this in Java for now and then revert when it's fixed in cudf?

[revans2]

@sameerz what is the priority on this? Do we need to get full nested type aggregation key and sorts into 23.06 or can it wait for 23.08.

@razajafri if we need it for 23.06 then we have to do something in java. If we are going to wait for 23.08, then we should just do it in CUDF correctly.

Either way we should come up with a repro case in CUDF C++ and file an issue against them.

[sameerz]

Do we need to get full nested type aggregation key and sorts into 23.06 or can it wait for 23.08.

This can wait until 23.08.

[razajafri]

This depends on rapidsai/cudf#13455

[razajafri]

This has been fixed by the cudf PR rapidsai/cudf#13455