Migrate triage dashboard to wg-k8s-infra

[spiffxp]

Part of migrating away from gcp-project k8s-gubernator: #1308

Triage is made up of a few components

• periodic jobs run via prow.k8s.io - these should be changed to run on k8s-infra-prow-build
• data files stored in gs://k8s-gubernator/triage - these should be migrated to a new k8s-infra owned GCS bucket
• a static site hosted in gs://k8s-gubernator/triage - ditto
• a http://go.k8s.io/triage redirect that points to the static site - this should be updated to point to the new location

Where should we move things to? My suggestions:

• gcp project: k8s-gubernator -> kubernetes-public
• gcs bucket: gs://k8s-gubernator -> gs://k8s-triage

One other wrinkle: visiting https://go.k8s.io/triage redirects to https://storage.googleapis.com/k8s-gubernator/triage/index.html which exposes the bucket. If we change to use a new bucket, we're probably going to break a lot of existing URI's. Ideally we can serve a 301 redirect pointing to the new location. Bonus points if we could mask out the bucket name (eg: have triage.k8s.io be our location)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ameukam]

/remove-lifecycle stale

[spiffxp]

/assign @spiffxp
/milestone v1.21

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ameukam]

/remove-lifecycle stale
/milestone clear

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[spiffxp]

/remove lifecycle-stale
/milestone v1.23

[spiffxp]

Ahh right, I had hopes this would be just as simple as gs://k8s-metrics, but it's not.

gs://k8s-gubernator/triage is the path for triage, so gs://k8s-gubernator is the bucket to be migrated

The problem is it's much more complicated because it uses ACLs, so there's no guarantee a delete/recreate as I did last time would recreate all ACLs (and whether there are google-internal things hidden in there)

$ gsutil ubla get gs://k8s-gubernator
'NoneType' object has no attribute 'bucketPolicyOnly'

$ gsutil iam get gs://k8s-gubernator | yq -y
bindings:
  - members:
      - projectEditor:k8s-gubernator
      - projectOwner:k8s-gubernator
      - serviceAccount:[email protected]
    role: roles/storage.legacyBucketOwner
  - members:
      - projectViewer:k8s-gubernator
    role: roles/storage.legacyBucketReader
  - members:
      - serviceAccount:[email protected]
    role: roles/storage.objectAdmin
  - members:
      - serviceAccount:[email protected]
    role: roles/storage.objectCreator
  - members:
      - serviceAccount:[email protected]
    role: roles/storage.objectViewer
etag: CAQ=

I'm going to at least start with:

• syncing the contents of gs://k8s-gubernator/triage to gs://k8s-project-triage
• looking at what k8s.io redirects or reverse-proxy could do for us
• looking at what a 301 redirect from the old triage link would do for us

[spiffxp]

I toyed around with trying to create gs://triage.k8s.io so as to be able to serve content directly from GCS vs. having to proxy through nginx. It's requiring that I verify domain ownership. Which I feel like we have done, the question is who owns it?

[spiffxp]

For now I'm opting to:

• s/k8s-project-triage/k8s-triage
• setup a canary triage job in k8s-infra to run to populate gs://k8s-triage (config/jobs: setup canary triage job, mv other triage jobs test-infra#23126)
• mv the "push static files" job to k8s-infra, and update to push to gs://k8s-triage (config/jobs: setup canary triage job, mv other triage jobs test-infra#23126)
• mv the "push triage image" job to k8s-staging-test-infra (config/jobs: setup canary triage job, mv other triage jobs test-infra#23126)
• switch from using k8s-testimage/triage to k8s-staging-test-infra/triage
• setup a redirect index.html in gs://k8s-gubernator/triage that will javascript-redirect (preserving querystring + hash) to gs://k8s-triage/index.html
  • I experimented with a redirect from gs://k8s-triage back to gubernator
  • The redirect (with the buckets swapped) is in config/jobs: setup canary triage job, mv other triage jobs test-infra#23126 but not yet auto-deployed
• verify all still works
• cleanup (e.g. remove all triage folders from gs://k8s-gubernator)

[spiffxp]

PR's in-flight:

• Migrate k8s-triage, k8s-metrics to terraform, setup k8s-triage dataset #2461
• config/jobs: setup canary triage job, mv other triage jobs test-infra#23126

[spiffxp]

The canary job is refusing to schedule: https://testgrid.k8s.io/wg-k8s-infra-canaries#triage

So:

• change the prow-build-trusted nodepool to match the prow-build nodepool config (except autoscaling settings): prow: match prow-build-trusted nodepool to prow-build #2479
• update the job to ask for slightly fewer resources: config/jobs: misc fixes for k8s-infra triage jobs test-infra#23154

[spiffxp]

For some reason the triage image isn't in staging like I would have expected with kubernetes/test-infra#23126

$ gcloud container images list --project=k8s-staging-test-infra
NAME
gcr.io/k8s-staging-test-infra/alpine
gcr.io/k8s-staging-test-infra/docker-buildx
gcr.io/k8s-staging-test-infra/gcb-docker-gcloud
gcr.io/k8s-staging-test-infra/git

[spiffxp]

... that would be because nothing has landed that would trigger the push to staging after the job config was updated

I was originally going to make a dummy change just to push a new image, but in grepping for k8s-gubernator in the triage directory I realized it's still hardcoded in the update_summaries.sh script, to the canary was bound to fail trying to use a project it wasn't authorized to

Next steps:

• PR to add env var based configuration for projects, buckets, and datasets: triage: prepare for alternate projects, datasets, buckets test-infra#23168
• See how far the canary job gets
• Look into provisioning a bq dataset for the canary job to write temp results to

[spiffxp]

We are very nearly done now:

• fix static upload job to work with UBLA bucket, fix gcr.io/k8s-staging-test-infra/triage image build: triage: fix postsubmits test-infra#23180
• write to a community-owned bigquery dataset: config/jobs: update ci-test-infra-triage-canary test-infra#23235
• fix triage job to work with UBLA bucket: triage: don't set ACLs when uploading test-infra#23256
• point go.k8s.io/triage to gs://k8s-triage: apps/k8s.io: flip go.k8s.io/triage to community-owned bucket #2544
• drop old google.com job: config/jobs: use community-owned infra for triage test-infra#23258

[spiffxp]

When the final PR merges I will:

• go through and find some old triage links to make sure the redirect appropriately
• validate the triage job shows on https://testgrid.k8s.io/wg-k8s-infra-prow

[spiffxp]

Will close after one last PR:

• Pushing to gs://k8s-gubernator can't run from community, and do need the public_read ACL since the bucket as a whole can't be public read
• I manually fixed and ran the appropriate make commands
• PR with the changes: triage: fix k8s-gubernator triage redirect test-infra#23261

Arbitrary old link I verified the redirect with:

• https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=Pods%20should%20support%20pod%20readiness%20gates
• redirects to: https://storage.googleapis.com/k8s-triage/index.html?test=Pods%20should%20support%20pod%20readiness%20gates

[spiffxp]

I'm a bit hesitant to shout "success!" because kettle being down (kubernetes/test-infra#23135) makes triage look broken, but it's just as broken as it was before migration. We're reading and clustering data on community infra, but there's no new data to cluster right now...

[spiffxp]

/close
Calling this done!

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Calling this done!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.