Explore possibilities to lower in-use IP address quota footprint.

[kennknowles]

 

java.lang.RuntimeException: Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s)
to execute this workflow with 1 instances in region us-central1. Quota summary (required/available):
1/11743 instances, 1/170 CPUs, 250/278399 disk GB, 0/4046 SSD disk GB, 1/188 instance groups, 1/188
managed instance groups, 1/126 instance templates, 1/0 in-use IP addresses.

 

Sample jobs:
https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/440/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testGlobalWindowContainsInAnyOrder/

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/67/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyIterableSideInput/

 

Imported from Jira BEAM-7410. Original Jira may contain additional context.
Reported by: Ardagan.

[apilloud]

Issue ongoing: https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/809/testReport/junit/org.apache.beam.examples/WindowedWordCountIT/testWindowedWordCountInBatchDynamicSharding/

java.lang.RuntimeException: Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/38755 instances, 2/1265 CPUs, 25/330889 disk GB, 0/553 SSD disk GB, 1/283 instance groups, 1/286 managed instance groups, 1/402 instance templates, 1/0 in-use IP addresses.

[damondouglas]

.add-labels beam-fixit

[damondouglas]

Communicated assessment/recommendation internally.

[damondouglas]

.take-issue

[aaltay]

Is this still an issue worth addressing soon? Are we running into infra issues for issues related to this quota?

[damondouglas]

Please see personal communication.

[damondouglas]

@kennknowles Unassigned this issue as it the fix requires access to the jenkins environment for efficient debugging and testing that I do not have at this time.

[Abacn]

Each Jenkins node uses a static IP address, so 16 addresses in total. This is required to connect with the Jenkins master that is managed by Apache Infra. From gcp console there are actually two complete set of Jenkins node, one in use (apache-ci-beam-jenkins-nodes) and another are long stopped (apache-beam-jenkins-jnlp-group). The former was created on Jun 10, 2020 and latter was created on Apr 9, 2019. Likely the latter was no longer in use and resources could be released.

[Abacn]

deleted 17 instances and released these static IP address. This should suffice to resolve the quota issue.

IP address quota footprint would be further decreased once we migrate to k8s based github action runner, and finally teardown jenkins

close this issue for now