Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement per-shim parallel world jar classloader #3381

Merged
merged 21 commits into from
Sep 10, 2021
Merged

Implement per-shim parallel world jar classloader #3381

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
07526a3
Parallel world ServiceLoader
gerashegalov Sep 3, 2021
a8f5949
Implement per-shim parallel world jar classloader
gerashegalov Sep 3, 2021
0d5b11c
review and test fix
gerashegalov Sep 3, 2021
fbacdbe
Adding shuffle manager
gerashegalov Sep 4, 2021
72dd879
Merge remote-tracking branch 'origin/branch-21.10' into shimLoader320
gerashegalov Sep 4, 2021
59503cd
wip
gerashegalov Sep 4, 2021
aca6e8e
test fix
gerashegalov Sep 4, 2021
9fbe630
test fix
gerashegalov Sep 4, 2021
2ce74fc
reviews
gerashegalov Sep 7, 2021
0e5ad4e
commens
gerashegalov Sep 7, 2021
e59f44f
Merge remote-tracking branch 'origin/branch-21.10' into shimLoader320
gerashegalov Sep 7, 2021
21873ef
ProxyShuffleManager cannot be dedupped, so undoing
gerashegalov Sep 9, 2021
c89a65c
dir cleanup
gerashegalov Sep 9, 2021
bf79e74
Merge remote-tracking branch 'origin/branch-21.10' into shimLoader320
gerashegalov Sep 9, 2021
871e12d
source code layout doc
gerashegalov Sep 9, 2021
9a43800
preview
gerashegalov Sep 9, 2021
6ec561a
whitespace
gerashegalov Sep 9, 2021
44fbfac
inclusive
gerashegalov Sep 9, 2021
d905db3
scalastyle violations
gerashegalov Sep 9, 2021
742383d
Merge remote-tracking branch 'origin/branch-21.10' into shimLoader320
gerashegalov Sep 9, 2021
4696232
Merge remote-tracking branch 'origin/branch-21.10' into shimLoader320
gerashegalov Sep 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 38 additions & 26 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,13 @@
<module>udf-examples</module>
</modules>

<!--
301+ : beginning with spark301 including spark301
311- : until 311 excluding spark 311
all : applies to all shims
apache: applies only to upstream shims
cdh, emr, db: applies to one of the distribution shims
-->
<profiles>
<profile>
<id>default</id>
Expand All @@ -91,8 +98,8 @@
<module>api_validation</module>
<module>tools</module>
</modules>
</profile>
<profile>
</profile>
<profile>
<id>no-buildver-default</id>
<activation>
<property>
Expand All @@ -114,18 +121,17 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark30+all/java</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
<source>${project.basedir}/src/main/spark301+all/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

<profile>
</profile>
<profile>
<id>buildver-default</id>
<activation>
<property>
Expand Down Expand Up @@ -153,8 +159,8 @@
</plugin>
</plugins>
</build>
</profile>
<profile>
</profile>
<profile>
<id>release301</id>
<activation>
<property>
Expand All @@ -178,7 +184,8 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark301+311-all/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -214,7 +221,8 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark301+311-all/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -243,7 +251,8 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark301+311-all/scala</source>
gerashegalov marked this conversation as resolved.
Show resolved Hide resolved
<source>${project.basedir}/src/main/spark301+all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -283,7 +292,8 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark301+311-all/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -319,7 +329,8 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark301+311-all/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -355,9 +366,9 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark31+all/scala</source>
<source>${project.basedir}/src/main/spark31+apache/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
<source>${project.basedir}/src/main/spark311+all/scala</source>
<source>${project.basedir}/src/main/spark311+apache/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -448,9 +459,9 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark31+all/scala</source>
<source>${project.basedir}/src/main/spark31+apache/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
<source>${project.basedir}/src/main/spark311+all/scala</source>
<source>${project.basedir}/src/main/spark311+apache/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -486,9 +497,10 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark31+all/scala</source>
<source>${project.basedir}/src/main/spark31+apache/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
<source>${project.basedir}/src/main/spark311+all/scala</source>
<source>${project.basedir}/src/main/spark311+320-all/scala</source>
<source>${project.basedir}/src/main/spark311+apache/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -524,7 +536,7 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark31+all/scala</source>
<source>${project.basedir}/src/main/spark311+all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -557,8 +569,8 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/spark30+all/scala</source>
<source>${project.basedir}/src/main/spark31+all/scala</source>
<source>${project.basedir}/src/main/spark301+all/scala</source>
<source>${project.basedir}/src/main/spark311+all/scala</source>
</sources>
</configuration>
</execution>
Expand Down
38 changes: 38 additions & 0 deletions shims/spark301/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,4 +82,42 @@
<scope>provided</scope>
</dependency>
</dependencies>
<profiles>
<profile>
<!--
enables Shim class consolidation without breaking the existing build,
to be removed when the build architecture swap is complete
-->
<id>no-buildver-default</id>
<activation>
<property>
<name>!buildver</name>
</property>
</activation>
<properties>
<spark-rapids.sql-plugin.root>${project.basedir}/../../sql-plugin</spark-rapids.sql-plugin.root>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-profile-src-default</id>
<goals><goal>add-source</goal></goals>
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${spark-rapids.sql-plugin.root}/src/main/spark301+all/scala</source>
<source>${spark-rapids.sql-plugin.root}/src/main/spark301+311-all/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
6 changes: 3 additions & 3 deletions shims/spark301/src/main/scala/com/nvidia/spark/rapids/spark301/RapidsShuffleManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2020, NVIDIA CORPORATION.
* Copyright (c) 2020-2021, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand All @@ -17,10 +17,10 @@
package com.nvidia.spark.rapids.spark301

import org.apache.spark.SparkConf
import org.apache.spark.sql.rapids.shims.spark301.RapidsShuffleInternalManager
import org.apache.spark.rapids.shims.v2.ProxyRapidsShuffleInternalManager

/** A shuffle manager optimized for the RAPIDS Plugin for Apache Spark. */
sealed class RapidsShuffleManager(
conf: SparkConf,
isDriver: Boolean) extends RapidsShuffleInternalManager(conf, isDriver) {
isDriver: Boolean) extends ProxyRapidsShuffleInternalManager(conf, isDriver) {
}
4 changes: 2 additions & 2 deletions ...s/spark301db/src/main/scala/com/nvidia/spark/rapids/spark301db/RapidsShuffleManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
package com.nvidia.spark.rapids.spark301db

import org.apache.spark.SparkConf
import org.apache.spark.sql.rapids.shims.spark301db.RapidsShuffleInternalManager
import org.apache.spark.rapids.shims.v2.ProxyRapidsShuffleInternalManager

/** A shuffle manager optimized for the RAPIDS Plugin for Apache Spark. */
sealed class RapidsShuffleManager(
conf: SparkConf,
isDriver: Boolean) extends RapidsShuffleInternalManager(conf, isDriver) {
isDriver: Boolean) extends ProxyRapidsShuffleInternalManager(conf, isDriver) {
}
39 changes: 39 additions & 0 deletions shims/spark301emr/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,4 +82,43 @@
<scope>provided</scope>
</dependency>
</dependencies>

<profiles>
<profile>
<!--
enables Shim class consolidation without breaking the existing build,
to be removed when the build architecture swap is complete
-->
<id>no-buildver-default</id>
<activation>
<property>
<name>!buildver</name>
</property>
</activation>
<properties>
<spark-rapids.sql-plugin.root>${project.basedir}/../../sql-plugin</spark-rapids.sql-plugin.root>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-profile-src-default</id>
<goals><goal>add-source</goal></goals>
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${spark-rapids.sql-plugin.root}/src/main/spark301+all/scala</source>
<source>${spark-rapids.sql-plugin.root}/src/main/spark301+311-all/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
6 changes: 3 additions & 3 deletions ...spark301emr/src/main/scala/com/nvidia/spark/rapids/spark301emr/RapidsShuffleManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2020, NVIDIA CORPORATION.
* Copyright (c) 2020-2021, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand All @@ -17,10 +17,10 @@
package com.nvidia.spark.rapids.spark301emr

import org.apache.spark.SparkConf
import org.apache.spark.sql.rapids.shims.spark301emr.RapidsShuffleInternalManager
import org.apache.spark.rapids.shims.v2.ProxyRapidsShuffleInternalManager

/** A shuffle manager optimized for the RAPIDS Plugin for Apache Spark. */
sealed class RapidsShuffleManager(
conf: SparkConf,
isDriver: Boolean) extends RapidsShuffleInternalManager(conf, isDriver) {
isDriver: Boolean) extends ProxyRapidsShuffleInternalManager(conf, isDriver) {
}
38 changes: 38 additions & 0 deletions shims/spark302/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,4 +82,42 @@
<scope>provided</scope>
</dependency>
</dependencies>
<profiles>
<profile>
<!--
enables Shim class consolidation without breaking the existing build,
to be removed when the build architecture swap is complete
-->
<id>no-buildver-default</id>
<activation>
<property>
<name>!buildver</name>
</property>
</activation>
<properties>
<spark-rapids.sql-plugin.root>${project.basedir}/../../sql-plugin</spark-rapids.sql-plugin.root>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-profile-src-default</id>
<goals><goal>add-source</goal></goals>
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${spark-rapids.sql-plugin.root}/src/main/spark301+all/scala</source>
<source>${spark-rapids.sql-plugin.root}/src/main/spark301+311-all/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
4 changes: 2 additions & 2 deletions shims/spark302/src/main/scala/com/nvidia/spark/rapids/spark302/RapidsShuffleManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
package com.nvidia.spark.rapids.spark302

import org.apache.spark.SparkConf
import org.apache.spark.sql.rapids.shims.spark302.RapidsShuffleInternalManager
import org.apache.spark.rapids.shims.v2.ProxyRapidsShuffleInternalManager

/** A shuffle manager optimized for the RAPIDS Plugin for Apache Spark. */
sealed class RapidsShuffleManager(
conf: SparkConf,
isDriver: Boolean) extends RapidsShuffleInternalManager(conf, isDriver) {
isDriver: Boolean) extends ProxyRapidsShuffleInternalManager(conf, isDriver) {
}
Loading