Force parallel world in Shim caller's classloader (#3763)

* Simplify shim classloader logic - uprev spark320 breaking DiskManager changes - use the Serializer instance to find mutable classloader - make the the update logic oblivious to the executor/driver side Signed-off-by: Gera Shegalov <[email protected]> * Restore ExecutorClassLoader treatment for userClassPathFirst Signed-off-by: Gera Shegalov <[email protected]> * Review comments Signed-off-by: Gera Shegalov <[email protected]> * wip * wip * fix forcing AppClassLoader update Signed-off-by: Gera Shegalov <[email protected]> * Undo doc changes * Provide an option to update the ShimLoader's classloader - making this option default because it's equivalent to the old flat jar - making it optional because we still debug how we miss the non-default classloader in ##3704 and it's not the right behavior for addJar with userClassPathFirst. However, I think we shoudl generally stop documenting --jars as the plugin deploy option Fixes #3704 Signed-off-by: Gera Shegalov <[email protected]> * more logging Signed-off-by: Gera Shegalov <[email protected]> * undo run_pyspark_from_build.sh change
NVIDIA · Oct 7, 2021 · 16fc3aa · 16fc3aa
1 parent a19486f
commit 16fc3aa
Show file tree

Hide file tree

Showing 2 changed files with 20 additions and 3 deletions.
diff --git a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala
@@ -1296,6 +1296,14 @@ object RapidsConf {
     .booleanConf
     .createWithDefault(true)
 
+  val FORCE_SHIMCALLER_CLASSLOADER = conf("spark.rapids.force.caller.classloader")
+    .doc("Option to statically add shim's parallel world classloader URLs to " +
+      "the classloader of the ShimLoader class, typically Bootstrap classloader. This option" +
+      " uses reflection with setAccessible true on a classloader that is not created by Spark.")
+    .internal()
+    .booleanConf
+    .createWithDefault(value = true)
+
   private def printSectionHeader(category: String): Unit =
     println(s"\n### $category")
 

diff --git a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/ShimLoader.scala b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/ShimLoader.scala
@@ -135,7 +135,15 @@ object ShimLoader extends Logging {
     // org/apache/spark/serializer/KryoSerializer.scala#L134
 
     Option(SparkEnv.get)
-      .map(_.serializer)
+      .flatMap {
+        case env if !env.conf.get("spark.rapids.force.caller.classloader",
+          true.toString).toBoolean => Option(env.serializer)
+        case _ =>
+          logInfo("Forcing shim caller classloader update (default behavior). " +
+            "If it causes issues with userClassPathFirst, set " +
+            "spark.rapids.force.caller.classloader to false!")
+          None
+      }
       .flatMap { serializer =>
         logInfo("Looking for a mutable classloader (defaultClassLoader) in SparkEnv.serializer " +
           serializer)
@@ -153,8 +161,9 @@ object ShimLoader extends Logging {
         logInfo("Extracted Spark classloader from SparkEnv.serializer " + serdeClassLoader)
         findURLClassLoader(serdeClassLoader)
       }.orElse {
-        logInfo("Spark-less use case: RapidsConf.help?")
-        Option(getClass.getClassLoader)
+        val shimLoaderCallerCl = getClass.getClassLoader
+        logInfo("Falling back on ShimLoader caller's classloader " + shimLoaderCallerCl)
+        Option(shimLoaderCallerCl)
       }
   }