Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[25119] Change test-infra Kafka outside service to NodePort #25171

Conversation

damondouglas
Copy link
Contributor

This PR closes #25119 and addresses #19465 by:

  1. Change .test-infra/kubernetes/kafka-cluster/04-outside-services from LoadBalancer to NodePort
  2. Refactor .test-infra/kubernetes/kafka-cluster/05-kafka/10broker-config.yml to query OUTSIDE_HOST from k8s node InternalIP and OUTSIDE_PORT to query outside-${KAFKA_BROKER_ID}'s NodePort
  3. Added nodeIPAddress and nodePort methods to .test-infra/jenkins/Kubernetes.groovy and cooresponding .test-infra/kubernetes/kubernetes.sh to enable Node InternalIP and NodePort querying within a jenkins job context
  4. Refactor the following with usePublicIps=false
    • .test-infra/jenkins/job_PerformanceTests_xlang_KafkaIO_Python.groovy
    • .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
  5. terraform code to validate and provision Google Cloud networking requirements to support usePublicIps=false (Note: I already executed the terraform code, per personal communication, on apache-beam-testing GCP project)

I was unable to test end-to-end the new workflow from jenkins job creation to seeing a Dataflow Job executing successfully in my own Google Cloud environment. However, I was able to validate:

  1. Using a private IP compute engine instance in the same GCP network, https://kafka.apache.org/quickstart worked using the kubernetes cluster's node InternalIP and NodePort as the --bootstrap-server
  2. Spinning up .test-infra/dockerized-jenkins and seeding job_PerformanceTests_xlang_KafkaIO_Python.groovy from github.com/damondouglas/beam from branch 25119-fix-test-infra-kafka-service built the test job using the queried kubernetes Node InternalIP and NodePort as the bootstrap_server arguments

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@github-actions github-actions bot added the infra label Jan 25, 2023
@johnjcasey johnjcasey requested a review from Abacn January 25, 2023 21:27
@Abacn
Copy link
Contributor

Abacn commented Jan 25, 2023

Run Python xlang KafkaIO Performance Test

@Abacn
Copy link
Contributor

Abacn commented Jan 25, 2023

after a seedjob looks like it still trying to access to loadBalancer somewhere (which is replaced to NodePort in this PR)

eval 'kubectl --kubeconfig=/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_xlang_KafkaIO_Python/config-beam-performancetests-xlang-kafkaio-python-37 --namespace=beam-performancetests-xlang-kafkaio-python-37 get svc outside-0 -ojsonpath='\''{.status.loadBalancer.ingress[0].ip}'\'''

@Abacn
Copy link
Contributor

Abacn commented Jan 25, 2023

Run Python xlang KafkaIO Performance Test

looks like there is still some references to loadBalancer, though also see NodePort in log : https://ci-beam.apache.org/job/beam_PerformanceTests_xlang_KafkaIO_Python/38/

@damondouglas
Copy link
Contributor Author

@Abacn
Copy link
Contributor

Abacn commented Jan 26, 2023

Run Python xlang KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Java KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Python xlang KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Java KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Python xlang KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Java KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Python xlang KafkaIO Performance Test

@damondouglas damondouglas marked this pull request as draft January 26, 2023 21:16
@damondouglas
Copy link
Contributor Author

@Abacn I set this to draft to take a break and focus on other projects. The job set the parameters like this: kafkaBootstrapServerAddresses=$NODE_IP_0:32401,$NODE_IP_1:32402,$NODE_IP_2:32403","--kafkaTopic=beam-sdf","--readTimeout=1500","--numWorkers=5","--autoscalingAlgorithm=NONE","--experiments=use_runner_v2,use_unified_****"

I was expecting instead of $NODE_IP_ to be an actual IP address.

@damondouglas damondouglas marked this pull request as ready for review January 31, 2023 17:00
@damondouglas
Copy link
Contributor Author

Run Java KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Python xlang KafkaIO Performance Test

@damondouglas
Copy link
Contributor Author

Run Python xlang KafkaIO Performance Test

@damondouglas damondouglas marked this pull request as draft February 2, 2023 15:59
@damondouglas damondouglas deleted the 25119-fix-test-infra-kafka-service branch September 25, 2024 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task]: Refactor kubernetes kafka External load balancer dependency in tests
2 participants