Add dynamic configs for the spark-rapids IT pipelines (NVIDIA#1450)

Add dynamic spark-rapids configs for the IT pipelines via parameter: 'BASE_SPARK_SUBMIT_ARGS=--conf abc=xxx --conf xyz=xxx --conf def=xxx ...'. e.g. We can enable spark 'AQE' for spark-rapids integration tests pipeline by: 'BASE_SPARK_SUBMIT_ARGS=--conf spark.sql.adaptive.enabled=true' The default value 'BASE_SPARK_SUBMIT_ARGS' is unset Signed-off-by: Tim Liu <[email protected]>
nartal1 · Jan 7, 2021 · 53442d5 · 53442d5
1 parent e16493b
commit 53442d5
Showing 1 changed file with 2 additions and 1 deletion.
diff --git a/jenkins/spark-tests.sh b/jenkins/spark-tests.sh
@@ -67,7 +67,8 @@ OUTPUT="$WORKSPACE/output"
 # spark.sql.cache.serializer conf is ignored for versions prior to 3.1.0
 SERIALIZER="--conf spark.sql.cache.serializer=com.nvidia.spark.rapids.shims.spark310.ParquetCachedBatchSerializer"
 
-BASE_SPARK_SUBMIT_ARGS="--master spark://$HOSTNAME:7077 \
+BASE_SPARK_SUBMIT_ARGS="$BASE_SPARK_SUBMIT_ARGS \
+    --master spark://$HOSTNAME:7077 \
     --executor-memory 12G \
     --total-executor-cores 6 \
     --conf spark.sql.shuffle.partitions=12 \