Add support for Spark 2.x Explain Api

docs/get-started/getting-started-workload-qualification.md

@@ -63,13 +63,8 @@ the other is to modify your existing Spark application code to call a function d
 
 Please note that if using adaptive execution in Spark the explain output may not be perfect
 as the plan could have changed along the way in a way that we wouldn't see by looking at just
-the CPU plan.
-
-### Requirements
-
-- A Spark 3.x CPU cluster
-- The `rapids-4-spark` and `cudf` [jars](../download.md)
-- Ability to modify the existing Spark application code if using the function call directly
+the CPU plan. The same applies if you are using an older version of Spark. Spark planning
+may be slightly different if you go up to a newer version of Spark.
 
 ### Using the Configuration Flag for Explain Only Mode
 
@@ -78,6 +73,13 @@ This mode allows you to run on a CPU cluster and can help us understand the pote
 if there are any unsupported features. Basically it will log the output which is the same as
 the driver logs with `spark.rapids.sql.explain=all`.
 
+#### Requirements
+
+- A Spark 3.x CPU cluster
+- The `rapids-4-spark` and `cudf` [jars](../download.md)
+
+#### Usage
+
 1. In `spark-shell`, add the `rapids-4-spark` and `cudf` jars into --jars option or put them in the
    Spark classpath and enable the configs `spark.rapids.sql.mode=explainOnly` and
    `spark.plugins=com.nvidia.spark.SQLPlugin`.
@@ -125,20 +127,41 @@ pretty accurate.
 
 ### How to use the Function Call
 
-Starting with version 21.12 of the RAPIDS Accelerator, a new function named
-`explainPotentialGpuPlan` is added which can help us understand the potential GPU plan and if there
-are any unsupported features on a CPU cluster.  Basically it can return output which is the same as
-the driver logs with `spark.rapids.sql.explain=all`.
+A function named `explainPotentialGpuPlan` is available which can help us understand the potential
+GPU plan and if there are any unsupported features on a CPU cluster. Basically it can return output
+which is the same as the driver logs with `spark.rapids.sql.explain=all`.
 
-1. In `spark-shell`, add the `rapids-4-spark` and `cudf` jars into --jars option or put them in the
+#### Requirements with Spark 3.X
+
+- A Spark 3.X CPU cluster
+- The `rapids-4-spark` and `cudf` [jars](../download.md)
+- Ability to modify the existing Spark application code
+- RAPIDS Accelerator for Apache Spark version 21.12 or newer
+
+#### Requirements with Spark 2.4.X
+
+- A Spark 2.4.X CPU cluster
+- The `rapids-4-spark-sql-meta` [jar](../download.md)
+- Ability to modify the existing Spark application code
+- RAPIDS Accelerator for Apache Spark version 22.02 or newer
+
+#### Usage
+
+1. In `spark-shell`, add the necessary jars into --jars option or put them in the
    Spark classpath.
 
-   For example:
+   For example, on Spark 3.X:
 
    ```bash
    spark-shell --jars /PathTo/cudf-<version>.jar,/PathTo/rapids-4-spark_<version>.jar
    ```
 
+   For example, on Spark 2.4.X:
+
+   ```bash
+   spark-shell --jars /PathTo/rapids-4-spark-sql-meta-<version and classifier>.jar
+   ```
+
 2. Test if the class can be successfully loaded or not.
 
    ```scala
@@ -148,8 +171,8 @@ the driver logs with `spark.rapids.sql.explain=all`.
 3. Enable optional RAPIDS Accelerator related parameters based on your setup.
 
    Enabling optional parameters may allow more operations to run on the GPU but please understand
-   the meaning and risk of above parameters before enabling it. Please refer to [configs
-   doc](../configs.md) for details of RAPIDS Accelerator parameters.
+   the meaning and risk of above parameters before enabling it. Please refer to the
+   [configuration documentation](../configs.md) for details of RAPIDS Accelerator parameters.
 
    For example, if your jobs have `double`, `float` and `decimal` operators together with some Scala
    UDFs, you can set the following parameters:

jenkins/spark-nightly-build.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright (c) 2020-2021, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2020-2022, NVIDIA CORPORATION. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -74,6 +74,14 @@ function distWithReducedPom {
         $mvnExtaFlags
 }
 
+# build the Spark 2.x explain jar
+mvn -B $MVN_URM_MIRROR -Dmaven.repo.local=$M2DIR -Dbuildver=24X clean install -DskipTests
+[[ $SKIP_DEPLOY != 'true' ]] && \
+    mvn -B deploy $MVN_URM_MIRROR \
+        -Dmaven.repo.local=$M2DIR \
+        -DskipTests \
+        -Dbuildver=24X
+
 # build, install, and deploy all the versions we support, but skip deploy of individual dist module since we
 # only want the combined jar to be pushed.
 # Note this does not run any integration tests

jenkins/spark-premerge-build.sh

@@ -35,6 +35,9 @@ mvn_verify() {
     # file size check for pull request. The size of a committed file should be less than 1.5MiB
     pre-commit run check-added-large-files --from-ref $BASE_REF --to-ref HEAD
 
+    # build the Spark 2.x explain jar
+    env -u SPARK_HOME mvn -B $MVN_URM_MIRROR -Dbuildver=24X clean install -DskipTests
+
     # build all the versions but only run unit tests on one 3.0.X version (base version covers this), one 3.1.X version, and one 3.2.X version.
     # All others shims test should be covered in nightly pipelines
     env -u SPARK_HOME mvn -U -B $MVN_URM_MIRROR -Dbuildver=302 clean install -Drat.skip=true -DskipTests -Dmaven.javadoc.skip=true -Dskip -Dmaven.scalastyle.skip=true -Dcuda.version=$CUDA_CLASSIFIER -pl aggregator -am

pom.xml

@@ -69,18 +69,25 @@
             <timezone>-6</timezone>
         </developer>
     </developers>
-
-    <modules>
-        <module>dist</module>
-        <module>integration_tests</module>
-        <module>shims</module>
-        <module>shuffle-plugin</module>
-        <module>sql-plugin</module>
-        <module>tests</module>
-        <module>udf-compiler</module>
-        <module>udf-examples</module>
-    </modules>
     <profiles>
+        <profile>
+            <id>release24X</id>
+            <activation>
+                <property>
+                    <name>buildver</name>
+                    <value>24X</value>
+                </property>
+            </activation>
+            <properties>
+                <scala.binary.version>2.11</scala.binary.version>
+                <scala.version>2.11.12</scala.version>
+                <spark.version>${spark248.version}</spark.version>
+                <spark.test.version>${spark248.version}</spark.test.version>
+            </properties>
+            <modules>
+                <module>spark2-sql-plugin</module>
+            </modules>
+        </profile>
         <profile>
             <id>release301</id>
             <activation>
@@ -121,6 +128,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>api_validation</module>
                 <module>tools</module>
                 <module>aggregator</module>
@@ -166,6 +181,14 @@
                 <spark.version.classifier>spark302</spark.version.classifier>
             </properties>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>aggregator</module>
                 <module>tools</module>
                 <module>api_validation</module>
@@ -210,6 +233,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>api_validation</module>
                 <module>tools</module>
                 <module>aggregator</module>
@@ -254,6 +285,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>api_validation</module>
                 <module>tools</module>
                 <module>aggregator</module>
@@ -302,6 +341,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>api_validation</module>
                 <module>tools</module>
                 <module>aggregator</module>
@@ -359,6 +406,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>aggregator</module>
             </modules>
         </profile>
@@ -416,6 +471,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>aggregator</module>
             </modules>
         </profile>
@@ -462,6 +525,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>tools</module>
                 <module>aggregator</module>
                 <module>api_validation</module>
@@ -511,6 +582,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>tools</module>
                 <module>aggregator</module>
                 <module>api_validation</module>
@@ -557,6 +636,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>tools</module>
                 <module>aggregator</module>
                 <module>tests-spark310+</module>
@@ -602,6 +689,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>tools</module>
                 <module>aggregator</module>
                 <module>tests-spark310+</module>
@@ -647,6 +742,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>tools</module>
                 <module>aggregator</module>
                 <module>tests-spark310+</module>
@@ -691,6 +794,14 @@
                 </plugins>
             </build>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>tools</module>
                 <module>aggregator</module>
                 <module>tests-spark310+</module>
@@ -745,6 +856,14 @@
                 </repository>
             </repositories>
             <modules>
+                <module>dist</module>
+                <module>integration_tests</module>
+                <module>shims</module>
+                <module>shuffle-plugin</module>
+                <module>sql-plugin</module>
+                <module>tests</module>
+                <module>udf-compiler</module>
+                <module>udf-examples</module>
                 <module>tools</module>
                 <module>aggregator</module>
                 <module>tests-spark310+</module>
@@ -826,6 +945,7 @@
         <!--
          If you update a dependency version so it is no longer a SNAPSHOT
          please update the snapshot-shims profile as well so it is accurate -->
+        <spark248.version>2.4.8</spark248.version>
         <spark301.version>3.0.1</spark301.version>
         <spark301db.version>3.0.1-databricks</spark301db.version>
         <spark302.version>3.0.2</spark302.version>
@@ -1240,14 +1360,15 @@
                         <exclude>rmm_log.txt</exclude>
                         <exclude>dependency-reduced-pom*.xml</exclude>
                         <exclude>**/.*/**</exclude>
-                        <exclude>src/main/java/com/nvidia/spark/rapids/format/*</exclude>
+                        <exclude>**/src/main/java/com/nvidia/spark/rapids/format/*</exclude>
                         <exclude>**/*.csv</exclude>
                         <exclude>dist/*.txt</exclude>
                         <exclude>**/META-INF/com.nvidia.spark.rapids.SparkShimServiceProvider</exclude>
                         <!-- Apache Rat excludes target folder for projects that are included by
                         default, but there are some projects that are conditionally included.  -->
                         <exclude>**/target/**/*</exclude>
                         <exclude>**/cufile.log</exclude>
+                        <exclude>scripts/spark2diffs/*.diff</exclude>
                     </excludes>
                 </configuration>
             </plugin>