[BUG] Max() can return wrong results when column consists of Nans

[kuhushukla]

Describe the bug
Max() on a column with NaN values , with or without a grouping key which has NaNs generates wrong results on the GPU.
Steps/Code to reproduce bug

//GPU plugin on
val rdd = sc.parallelize(Seq(Float.NaN, 1.0, 2.0), 2)
val df = rdd.toDF("c0")
val res= df.agg(max("c0"))
res.collect
 res.explain

Output:

res1: Array[org.apache.spark.sql.Row] = Array([2.0])
== Physical Plan ==
*(2) GpuColumnarToRow false
+- GpuHashAggregate(keys=[], functions=[gpumax(c0#17)]), filters=List(None))
+- GpuCoalesceBatches TargetSize(2147483647)
+- GpuColumnarExchange gpusinglepartitioning(), true, [id=#75]
+- GpuHashAggregate(keys=[], functions=[partial_gpumax(c0#17)]), filters=List(None))
+- !GpuProject [value#14 AS c0#17]
+- GpuRowToColumnar TargetSize(2147483647)
+- *(1) SerializeFromObject [input[0, double, false] AS value#14]
+- Scan[obj#13]

//GPU Plugin off
 spark.conf.set("spark.rapids.sql.enabled", "false")
val res= df.agg(max("c0"))
scala> res.collect
 res.explain

Output:

res5: Array[org.apache.spark.sql.Row] = Array([NaN])

== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[max(c0#17)])
+- Exchange SinglePartition, true, [id=#106]
+- *(1) HashAggregate(keys=[], functions=[partial_max(c0#17)])
+- *(1) Project [value#14 AS c0#17]
+- *(1) SerializeFromObject [input[0, double, false] AS value#14]
+- Scan[obj#13]

Expected behavior
CPU and GPU answers should match.

Additional context
rapidsai/cudf#4753 is related on cudf side.

[revans2]

This looks like an issue that we cannot work around without the help of cudf. Because float aggregations are off by default I think in the short term if we just update the documentation we can live with this. But we should file a cudf specific issue to try and address this properly.

[jlowe]

This has been fixed.