Force parallel world in Shim caller's classloader

[gerashegalov]

This PR codes up the approach that was tested on Databricks as an alternative while working on #3756

• making this option default because it's mostly equivalent to the old flat jar classloader
• making it optional because we still debug how we miss the non-default classloader in [BUG] Executor-side ClassCastException when testing with Spark 3.2.1-SNAPSHOT in k8s environment #3704, and it's not the right behavior for addJar with userClassPathFirst. However, I think we should generally stop documenting --jars as the plugin deploy option

Fixes #3704

Signed-off-by: Gera Shegalov [email protected]

[gerashegalov]

build

[tgravescs]

generally stop documenting --jars as the plugin deploy option

Why would we do this? That is generally how you are supposed to distribute jars on spark, what is your alternative? this needs more explanation. for instance, on yarn you use --jars to get things into distributed cache to get sent to the nodes, without this you expect people to install it on every node?

[tgravescs]

nit, if changes are made to this file perhaps update function name to be more generic

[gerashegalov]

Why would we do this? That is generally how you are supposed to distribute jars on spark, what is your alternative? this needs more explanation. for instance, on yarn you use --jars to get things into distributed cache to get sent to the nodes, without this you expect people to install it on every node?

Good point. My main point that we should decide which of multiple ways to deploy plugin jars we recommend and support, and make sure that it works with all the features we provide such as the shuffle manager in all Spark deploy modes.

[abellina]

Taking a look at this patch with UCX

[gerashegalov]

While talking through the list of pros and cons with @tgravescs we concluded that my concern userClassPathFirst is moot with the following reasoning:
pre-21.10 the whole jar was visible to the ShimLoader's classloader, and we are not changing what classloader was chosen by Spark to load the shim. Thus by making the right sections of the jar accessible to the same class loader makes it only equivalent to the pre-21.10 setup.

So maybe having a reflection call into a non-public API is the only known downside at this moment.

[abellina]

Taking a look at this patch with UCX

It came up fine with and without earlyStart. I tried with the latest spark 3.2.1