test/kokoro: Add bootstrap generator test into Go Kokoro script #6463

zasweq · 2023-07-20T16:46:58Z

This PR adds the bootstrap generator test into Go kokoro script.

RELEASE NOTES: N/A

sergiitk · 2023-07-20T17:33:11Z

test/kokoro/xds_k8s_lb.sh

@@ -167,6 +167,7 @@ main() {
    "outlier_detection_test"
    "remove_neg_test"
    "round_robin_test"
+    "bootstrap_generator_test"


We need to figure out the logic on only including this when the branch under test is master. See an example here: https://github.com/grpc/grpc/blob/5b46c8bdbad733fa74ddf07c5b6fa3c9a600dbb9/tools/internal_ci/linux/grpc_xds_k8s_xlang.sh#L59-L62

You kinda need to do the opposite - add the bootstrap_generator_test to the list of tests only if $TESTING_VERSION is `master.

sergiitk · 2023-07-20T18:42:15Z

test/kokoro/xds_k8s_lb.sh

  )
+  if [ "${TESTING_VERSION}" != "master" ]; then


We need the opposite: only add it when the version IS master

Whoops, sorry, missed that.

sergiitk · 2023-07-20T18:43:12Z

test/kokoro/xds_k8s_lb.sh

  )
+  if [ "${TESTING_VERSION}" != "master" ]; then


Prefer [ over [[: https://google.github.io/styleguide/shellguide.html#s6.3-tests.

Ah, interesting, changed. Noted for future.

sergiitk · 2023-07-21T00:01:58Z

Test run:

grpc/go/master/branch/xds_k8s_lb

easwars · 2023-07-21T00:32:01Z

I don't know enough about this script. Since you will probably need an approver from one of the maintainers, I will approve once @sergiitk approves. Thanks.

sergiitk · 2023-07-21T18:15:45Z

Hm, something didn't work with the test run. It didn't run the bootstrap_generator_test.
@zasweq - could you please add set -x before the test_suites=( line so we can debug this?

zasweq · 2023-07-24T16:19:20Z

zasweq · 2023-07-24T16:20:00Z

From your invocation, I see the that outlier detection test passed, and did not "not run"? Am I missing something?

zasweq · 2023-07-24T16:20:42Z

https://source.cloud.google.com/results/invocations/11ca23e1-e69c-4d16-b781-8c146ba9e88b/targets/grpc%2Fgo%2Fmaster%2Fbranch%2Fxds_k8s_lb%2Foutlier_detection_test/log logs seems ok to me (just took a while 371 seconds).

sergiitk · 2023-07-24T22:34:00Z

@zasweq https://source.cloud.google.com/results/invocations/11ca23e1-e69c-4d16-b781-8c146ba9e88b/targets

There's no bootstrap_generator_test in the list of passed tests.

sergiitk · 2023-07-24T22:37:25Z

New test run with debugging enabled:

grpc/go/master/branch/xds_k8s_lb

sergiitk · 2023-07-24T23:02:44Z

test/kokoro/xds_k8s_lb.sh

@@ -168,6 +169,9 @@ main() {
    "remove_neg_test"
    "round_robin_test"
  )
+  if [[ "${TESTING_VERSION}" == "master" ]]; then


Ah, that's on me. This would've worked when merged. The reason the test wasn't running is for manual test run we set TESTING_VERSION to dev-master, and not master.

This should take care of it:

Suggested change

if [[ "${TESTING_VERSION}" == "master" ]]; then

if [[ "${TESTING_VERSION}" =~ "master" ]]; then

sergiitk · 2023-07-25T22:14:14Z

grpc/go/master/branch/xds_k8s_lb

sergiitk · 2023-07-26T00:04:45Z

Ok, now we see the test ran, and it failed

BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')

Traceback (most recent call last):
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/absl/testing/parameterized.py", line 316, in bound_param_test
    return test_method(self, *testcase_params)
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/tests/bootstrap_generator_test.py", line 202, in test_baseline_in_server_with_bootstrap_version
    self.test_server = self.startTestServer(
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/framework/bootstrap_generator_testcase.py", line 143, in startTestServer
    test_server = server_runner.run(
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_server_runner.py", line 212, in run
    pod = self._wait_pod_started(pod_name)
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 552, in _wait_pod_started
    self.k8s_namespace.wait_for_pod_started(name, **kwargs)
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 559, in wait_for_pod_started
    retryer(self.get_pod, pod_name)
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/tenacity/__init__.py", line 369, in iter
    return self.retry_error_callback(retry_state=retry_state)
  File "/tmp/tmp.6gukgbTRj6/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
    raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_pod: timeout 0:03:00 (h:mm:ss) exceeded. Check result callback returned False.

sergiitk · 2023-07-26T00:07:14Z

Target log provided extra info:

init_container_statuses:
- container_id: containerd://8ca568563b76cd48371ebbd040e2e467303a0cf19286dea46b2a88277796d2ad
  image: [gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2](https://www.google.com/url?q=http://gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2&sa=D)
  image_id: [gcr.io/grpc-testing/td-grpc-bootstrap@sha256:ea80ec66017c7bc2a661e2403d01e6689b9c0ef6110a892dfaa3a405ecdb3344](https://www.google.com/url?q=http://gcr.io/grpc-testing/td-grpc-bootstrap@sha256:ea80ec66017c7bc2a661e2403d01e6689b9c0ef6110a892dfaa3a405ecdb3344&sa=D)
  last_state:
    running: null
    terminated:
      container_id: containerd://8ca568563b76cd48371ebbd040e2e467303a0cf19286dea46b2a88277796d2ad
      exit_code: 2
      finished_at: 2023-07-25 23:54:05+00:00
      message: null
      reason: Error
      signal: null
      started_at: 2023-07-25 23:54:05+00:00
    waiting: null
  name: grpc-td-init
  ready: false
  restart_count: 5
  started: null
  state:
    running: null
    terminated: null
    waiting:
      message: back-off 2m40s restarting failed container=grpc-td-init pod=psm-grpc-server-d8d484ffd-vz9wr_psm-interop-server-20230725-2339-0d55u(84cdf144-9128-4feb-a66d-10a306b67d47)
      reason: CrashLoopBackOff

Image gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2 failed.

sergiitk · 2023-07-26T00:09:38Z

And here's the reason: https://cloudlogging.app.goo.gl/KiZ7U2zjhD1anVAP8

ERROR 2023-07-25T23:51:08.071335398Z flag provided but not defined: -node-metadata

zasweq · 2023-07-26T00:13:03Z

It failing for server v.0.12.0 and v.0.11.0 locally (me and Richard found that error too), and also failed client v.0.14.0 (which might be a real bootstrap generator failure) 1/3 times locally with ErrorStartingTestClient, which Richard mentioned if the bootstrap generator fails than you fail that.

easwars · 2023-07-26T17:07:20Z

It failing for server v.0.12.0 and v.0.11.0 locally

The -node-metadata flag was added in v0.13.0 here. So, that failure is not surprising, but failing in v0.14.0 is mildly bothering.

zasweq · 2023-07-26T20:38:53Z

Failing due to that flag, test isn't cleaning up resources properly (url map) and a worrysome client failure on 14 that might be the bootstrap generator.

sergiitk · 2023-07-29T00:49:09Z

grpc/go/master/branch/xds_k8s_lb

Based on grpc/grpc-go#6463

Add bootstrap generator test into Go Kokoro script

670a336

zasweq added the Type: Testing label Jul 20, 2023

zasweq added this to the 1.57 Release milestone Jul 20, 2023

zasweq requested a review from easwars July 20, 2023 16:47

zasweq assigned easwars Jul 20, 2023

sergiitk suggested changes Jul 20, 2023

View reviewed changes

Responded to Sergii's comment

7b5d946

sergiitk suggested changes Jul 20, 2023

View reviewed changes

Sergiis comments

74ceda2

easwars assigned sergiitk and zasweq and unassigned easwars Jul 21, 2023

arvindbr8 unassigned sergiitk Jul 24, 2023

Add debugging

e4428f0

zasweq assigned sergiitk and unassigned zasweq Jul 24, 2023

yashykt mentioned this pull request Jul 24, 2023

[PSM Interop] Add bootstrap generator test grpc/grpc#33833

Merged

sergiitk reviewed Jul 24, 2023

View reviewed changes

Sergiis comment

9a560a5

dfawley modified the milestones: 1.57 Release, 1.58 Release Jul 26, 2023

Removed set -x

ddf767a

easwars approved these changes Aug 1, 2023

View reviewed changes

easwars assigned zasweq and unassigned sergiitk Aug 1, 2023

zasweq added fixit and removed fixit labels Aug 1, 2023

zasweq closed this Aug 1, 2023

zasweq reopened this Aug 1, 2023

zasweq merged commit 8f496b2 into grpc:master Aug 1, 2023
1 check passed

github-actions bot locked as resolved and limited conversation to collaborators Jan 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test/kokoro: Add bootstrap generator test into Go Kokoro script #6463

test/kokoro: Add bootstrap generator test into Go Kokoro script #6463

zasweq commented Jul 20, 2023 •

edited

Loading

sergiitk Jul 20, 2023

zasweq Jul 24, 2023

sergiitk Jul 20, 2023

zasweq Jul 20, 2023

sergiitk Jul 20, 2023

zasweq Jul 20, 2023

sergiitk commented Jul 21, 2023

easwars commented Jul 21, 2023

sergiitk commented Jul 21, 2023

zasweq commented Jul 24, 2023

zasweq commented Jul 24, 2023

zasweq commented Jul 24, 2023

sergiitk commented Jul 24, 2023

sergiitk commented Jul 24, 2023

sergiitk Jul 24, 2023 •

edited

Loading

zasweq Jul 25, 2023

sergiitk commented Jul 25, 2023

sergiitk commented Jul 26, 2023

sergiitk commented Jul 26, 2023 •

edited

Loading

sergiitk commented Jul 26, 2023

zasweq commented Jul 26, 2023

easwars commented Jul 26, 2023

zasweq commented Jul 26, 2023

sergiitk commented Jul 29, 2023

	if [[ "${TESTING_VERSION}" == "master" ]]; then
	if [[ "${TESTING_VERSION}" =~ "master" ]]; then

test/kokoro: Add bootstrap generator test into Go Kokoro script #6463

test/kokoro: Add bootstrap generator test into Go Kokoro script #6463

Conversation

zasweq commented Jul 20, 2023 • edited Loading

sergiitk Jul 20, 2023

Choose a reason for hiding this comment

zasweq Jul 24, 2023

Choose a reason for hiding this comment

sergiitk Jul 20, 2023

Choose a reason for hiding this comment

zasweq Jul 20, 2023

Choose a reason for hiding this comment

sergiitk Jul 20, 2023

Choose a reason for hiding this comment

zasweq Jul 20, 2023

Choose a reason for hiding this comment

sergiitk commented Jul 21, 2023

easwars commented Jul 21, 2023

sergiitk commented Jul 21, 2023

zasweq commented Jul 24, 2023

zasweq commented Jul 24, 2023

zasweq commented Jul 24, 2023

sergiitk commented Jul 24, 2023

sergiitk commented Jul 24, 2023

sergiitk Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

zasweq Jul 25, 2023

Choose a reason for hiding this comment

sergiitk commented Jul 25, 2023

sergiitk commented Jul 26, 2023

sergiitk commented Jul 26, 2023 • edited Loading

sergiitk commented Jul 26, 2023

zasweq commented Jul 26, 2023

easwars commented Jul 26, 2023

zasweq commented Jul 26, 2023

sergiitk commented Jul 29, 2023

zasweq commented Jul 20, 2023 •

edited

Loading

sergiitk Jul 24, 2023 •

edited

Loading

sergiitk commented Jul 26, 2023 •

edited

Loading