Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fallback to ARENA if ASYNC configured and driver < 11.5.0 #4947

Merged
merged 2 commits into from
Mar 14, 2022
Merged

Fallback to ARENA if ASYNC configured and driver < 11.5.0 #4947

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c4c9404
Warn if using CUDA driver less than 11.5.0 with ASYNC
abellina Mar 14, 2022
0569e80
Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.s…
abellina Mar 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1492,14 +1492,23 @@ class RapidsConf(conf: Map[String, String]) extends Logging {
lazy val isPooledMemEnabled: Boolean = get(POOLED_MEM)

lazy val rmmPool: String = {
val pool = get(RMM_POOL)
if ("ASYNC".equalsIgnoreCase(pool) &&
(Cuda.getRuntimeVersion < 11020 || Cuda.getDriverVersion < 11020)) {
logWarning("CUDA runtime/driver does not support the ASYNC allocator, falling back to ARENA")
"ARENA"
} else {
pool
var pool = get(RMM_POOL)
if ("ASYNC".equalsIgnoreCase(pool)) {
val driverVersion = Cuda.getDriverVersion
val runtimeVersion = Cuda.getRuntimeVersion
var fallbackMessage: Option[String] = None
if (runtimeVersion < 11020 || driverVersion < 11020) {
fallbackMessage = Some("CUDA runtime/driver does not support the ASYNC allocator")
tgravescs marked this conversation as resolved.
Show resolved Hide resolved
} else if (driverVersion < 11050) {
fallbackMessage = Some("CUDA drivers before 11.5 have known incompatibilities with " +
"the ASYNC allocator")
}
if (fallbackMessage.isDefined) {
logWarning(s"${fallbackMessage.get}, falling back to ARENA")
pool = "ARENA"
}
}
pool
}

lazy val rmmAllocFraction: Double = get(RMM_ALLOC_FRACTION)
Expand Down