Skip to content

Commit

Permalink
Disable coalesce batch spilling to avoid cudf contiguous_split bug (N…
Browse files Browse the repository at this point in the history
…VIDIA#1871)

Signed-off-by: Jason Lowe <[email protected]>
  • Loading branch information
jlowe authored Mar 4, 2021
1 parent d859683 commit 224285f
Showing 1 changed file with 9 additions and 7 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -608,13 +608,15 @@ case class GpuCoalesceBatches(child: SparkPlan, goal: CoalesceGoal)
// cache in local vars to avoid serializing the plan
val outputSchema = schema
val decompressMemoryTarget = maxDecompressBatchMemory
val cannotSpill = child.schema.fields.exists { f =>
f.dataType match {
case MapType(_, _, _) | ArrayType(_, _) | StructType(_) => true
case _ => false
}
}

// disabling spillable batches due to
// https://github.com/rapidsai/cudf/issues/7514
val cannotSpill = true
// val cannotSpill = child.schema.fields.exists { f =>
// f.dataType match {
// case MapType(_, _, _) | ArrayType(_, _) | StructType(_) => true
// case _ => false
// }
// }
val batches = child.executeColumnar()
batches.mapPartitions { iter =>
if (outputSchema.isEmpty) {
Expand Down

0 comments on commit 224285f

Please sign in to comment.