-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[25119] Change test-infra Kafka outside service to NodePort #25171
[25119] Change test-infra Kafka outside service to NodePort #25171
Conversation
Run Python xlang KafkaIO Performance Test |
after a seedjob looks like it still trying to access to loadBalancer somewhere (which is replaced to NodePort in this PR)
|
Run Python xlang KafkaIO Performance Test looks like there is still some references to loadBalancer, though also see NodePort in log : https://ci-beam.apache.org/job/beam_PerformanceTests_xlang_KafkaIO_Python/38/ |
@Abacn is the jenkins job seeding from https://github.com/damondouglas/beam/tree/25119-fix-test-infra-kafka-service? |
Run Python xlang KafkaIO Performance Test |
Run Java KafkaIO Performance Test |
Run Python xlang KafkaIO Performance Test |
Run Java KafkaIO Performance Test |
Run Python xlang KafkaIO Performance Test |
Run Java KafkaIO Performance Test |
Run Python xlang KafkaIO Performance Test |
@Abacn I set this to draft to take a break and focus on other projects. The job set the parameters like this: kafkaBootstrapServerAddresses=$NODE_IP_0:32401,$NODE_IP_1:32402,$NODE_IP_2:32403","--kafkaTopic=beam-sdf","--readTimeout=1500","--numWorkers=5","--autoscalingAlgorithm=NONE","--experiments=use_runner_v2,use_unified_****" I was expecting instead of $NODE_IP_ to be an actual IP address. |
Run Java KafkaIO Performance Test |
Run Python xlang KafkaIO Performance Test |
Run Python xlang KafkaIO Performance Test |
This PR closes #25119 and addresses #19465 by:
OUTSIDE_HOST
from k8s nodeInternalIP
andOUTSIDE_PORT
to queryoutside-${KAFKA_BROKER_ID}
's NodePortnodeIPAddress
andnodePort
methods to .test-infra/jenkins/Kubernetes.groovy and cooresponding .test-infra/kubernetes/kubernetes.sh to enable Node InternalIP and NodePort querying within a jenkins job contextusePublicIps=false
usePublicIps=false
(Note: I already executed the terraform code, per personal communication, on apache-beam-testing GCP project)I was unable to test end-to-end the new workflow from jenkins job creation to seeing a Dataflow Job executing successfully in my own Google Cloud environment. However, I was able to validate:
--bootstrap-server
job_PerformanceTests_xlang_KafkaIO_Python.groovy
from github.com/damondouglas/beam from branch25119-fix-test-infra-kafka-service
built the test job using the queried kubernetes Node InternalIP and NodePort as the bootstrap_server argumentsThank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.UpdateCHANGES.md
with noteworthy changes.If this contribution is large, please file an Apache Individual Contributor License Agreement.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.