Add Load Tests SideInput Flink Batch Go workflow #28662
Conversation
| GCLOUD_ZONE: us-central1-a | ||
| CLUSTER_NAME: beam-loadtests-go-sideinput-flink-batch-${{ github.run_id }} | ||
| GCS_BUCKET: gs://beam-flink-cluster | ||
| FLINK_DOWNLOAD_URL: https://archive.apache.org/dist/flink/flink-1.15.0/flink-1.15.0-bin-scala_2.12.tgz | ||
| HADOOP_DOWNLOAD_URL: https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar | ||
| FLINK_TASKMANAGER_SLOTS: 1 | ||
| DETACHED_MODE: true | ||
| HARNESS_IMAGES_TO_PULL: gcr.io/apache-beam-testing/beam-sdk/beam_go_sdk:latest | ||
| JOB_SERVER_IMAGE: gcr.io/apache-beam-testing/beam_portability/beam_flink1.15_job_server:latest | ||
| ARTIFACTS_DIR: gs://beam-flink-cluster/beam-loadtests-go-sideinput-flink-batch-${{ github.run_id }} |
There was a problem hiding this comment.
Can we move these variables to the env section for the - name: Start Flink with parallelism 10 step as you did it for FLINK_NUM_WORKERS ?
There was a problem hiding this comment.
In this case we will need to do this 2-3 times, for start / restart / teardown flink.
| env: | ||
| FLINK_NUM_WORKERS: 10 | ||
| run: | | ||
| cd ./.test-infra/dataproc; ./flink_cluster.sh create |
There was a problem hiding this comment.
Can we run .${{ github.workspace }}/.test-infra/dataproc/flink_cluster.sh create instead of drilling down to the dataproc directory first?
There was a problem hiding this comment.
No, we should change directory, because there are relative paths inside this script
| beam_LoadTests_Go_SideInput_Flink_Batch_test_arguments_1: '' | ||
| beam_LoadTests_Go_SideInput_Flink_Batch_test_arguments_2: '' |
There was a problem hiding this comment.
Let's remove these variables and leave a comment on line #99:
# The env variables are created and populated in the test-arguments-action as "<github.job>_test_arguments_<argument_file_paths_index>"
|
Assigning reviewers. If you would like to opt out of this review, comment R: @Abacn for label build. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
The following jenkins job has been migrated:
https://ci-beam.apache.org/job/beam_LoadTests_Go_SideInput_Flink_Batch
https://ci-beam.apache.org/job/beam_LoadTests_Go_SideInput_Flink_Batch_PR
How has it been tested?
The job has been tested on Self-hosted GKE runners ubuntu 20.04.
The following test cases are passed:
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.