Skip to content

Commit

Permalink
[GoSDK Infra] Bugfix: Parallelism ignored when using endpoint flag.
Browse files Browse the repository at this point in the history
The parallelism flag is intended to be ignored when using Flink, Spark, or Samza runners, but due to a bug, this wasn't happening if one of those runners was used with an endpoint address instead of a job server jar. This fixes that.
  • Loading branch information
youngoli committed Aug 26, 2021
1 parent 4c0493f commit 83efa34
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions sdks/go/test/run_validatesrunner_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -262,22 +262,19 @@ elif [[ "$RUNNER" == "flink" || "$RUNNER" == "spark" || "$RUNNER" == "samza" ||
--job-port $JOB_PORT \
--expansion-port 0 \
--artifact-port 0 &
SIMULTANEOUS=1
elif [[ "$RUNNER" == "samza" ]]; then
java \
-jar $SAMZA_JOB_SERVER_JAR \
--job-port $JOB_PORT \
--expansion-port 0 \
--artifact-port 0 &
SIMULTANEOUS=1
elif [[ "$RUNNER" == "spark" ]]; then
java \
-jar $SPARK_JOB_SERVER_JAR \
--spark-master-url local \
--job-port $JOB_PORT \
--expansion-port 0 \
--artifact-port 0 &
SIMULTANEOUS=1 # Spark runner fails if jobs are run concurrently.
elif [[ "$RUNNER" == "portable" ]]; then
python3 \
-m apache_beam.runners.portability.local_job_service_main \
Expand Down Expand Up @@ -305,6 +302,11 @@ elif [[ "$RUNNER" == "flink" || "$RUNNER" == "spark" || "$RUNNER" == "samza" ||
fi
fi

# Disable parallelism on runners that don't support it.
if [[ "$RUNNER" == "flink" || "$RUNNER" == "spark" || "$RUNNER" == "samza" ]]; then
SIMULTANEOUS=1
fi

if [[ "$RUNNER" == "dataflow" ]]; then
# Verify docker and gcloud commands exist
command -v docker
Expand Down

0 comments on commit 83efa34

Please sign in to comment.