Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a two-shim jar by default on Databricks [databricks] #9013

Merged
merged 5 commits into from
Aug 11, 2023
Merged

Create a two-shim jar by default on Databricks [databricks] #9013

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e1cf497
Create a two-shim jar by default on Databricks
Aug 11, 2023
be3167c
Merge remote-tracking branch 'origin/branch-23.08' into dbBuild2Shims
Aug 11, 2023
d6c521f
Update jenkins/databricks/build.sh
gerashegalov Aug 11, 2023
3805fd1
remove hardcoding for 311
Aug 11, 2023
0713522
Merge branch 'dbBuild2Shims' of github.com:gerashegalov/spark-rapids …
Aug 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions jenkins/databricks/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,9 @@ initialize()
SKIP_DEP_INSTALL=${SKIP_DEP_INSTALL:-'0'}
# export 'M2DIR' so that shims can get the correct Spark dependency info
export M2DIR=/home/ubuntu/.m2/repository
# whether to build a two-shim jar with the lowest supported upstream Spark version
WITH_DEFAULT_UPSTREAM_SHIM=${WITH_DEFAULT_UPSTREAM_SHIM:-1}


# Print a banner of the build configurations.
printf '+ %*s +\n' 100 '' | tr ' ' =
Expand All @@ -108,6 +111,7 @@ initialize()
echo "Rapids build jar : ${RAPIDS_BUILT_JAR}"
echo "Build Version : ${BUILDVER}"
echo "Skip Dependencies : ${SKIP_DEP_INSTALL}"
echo "Include Default Spark Shim : ${WITH_DEFAULT_UPSTREAM_SHIM}"
printf '+ %*s +\n' 100 '' | tr ' ' =
}

Expand Down Expand Up @@ -143,5 +147,10 @@ fi
# Build the RAPIDS plugin by running package command for databricks
mvn -B -Ddatabricks -Dbuildver=$BUILDVER clean package -DskipTests $MVN_OPT

if [[ "$WITH_DEFAULT_UPSTREAM_SHIM" != "0" ]]; then
echo "Building the default Spark shim and creatign a two-shim dist jar"
gerashegalov marked this conversation as resolved.
Show resolved Hide resolved
mvn -B package -pl dist -am -Dincluded_buildvers=311,$BUILDVER -DskipTests -Dskip $MVN_OPT
fi

cd /home/ubuntu
tar -zcf spark-rapids-built.tgz spark-rapids