Migrate test-infra metrics to wg-k8s-infra

[spiffxp]

Part of migrating away from gcp-project k8s-gubernator: #1308

These queries: https://github.com/kubernetes/test-infra/tree/master/metrics

• prow jobs that run via prow.k8s.io - these should be migrated to run on k8s-infra-prow-build (-trusted?)
• results are stored in gs://k8s-metrics - we should use a k8s-infra owned bucket instead

Proposal for new projects/names

• gcp project: k8s-gubernator -> kubernetes-public
• gcs bucket: gs://k8s-metrics -> k8s-project-metrics

I know of at least some people who link directly to the in-bucket json, eg: http://storage.googleapis.com/k8s-metrics/flakes-latest.json. It would be helpful if we could instead serve 301 redirects to the new location of these files

EDIT(spiffxp):

The plan

• setup new bucket, allow google and community prow to write to it (infra/gcp/main: add k8s-project-metrics special-case #2442)
• rsync data from old bucket to new bucket
• setup canary job on community prow to write to new bucket (config/jobs: add metrics-bigquery-canary test-infra#23097)
• confirm bigquery can be queried and bucket can be written
• delete old bucket, rename new bucket to old bucket
• swap/delete canary so "real" job runs on community infra

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ameukam]

/remove-lifecycle stale

[spiffxp]

/assign @MushuEE
for help fleshing out steps, I think this could qualify for /help

[spiffxp]

/milestone v1.21

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ameukam]

/remove-lifecycle stale
/milestone clear

[spiffxp]

/milestone v1.22

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
/milestone v1.23

[spiffxp]

Starting on this:

• infra/gcp/main: add k8s-project-metrics special-case #2442
• infra/gcp/main: fix typos #2443

[spiffxp]

$ gsutil -m rsync gs://k8s-metrics gs://k8s-project-metrics
# ...
| [7.5k/7.5k files][  5.0 GiB/  5.0 GiB] 100% Done  35.2 MiB/s ETA 00:00:00
Operation completed over 7.5k objects/5.0 GiB.

[spiffxp]

Working on using the new infra via a canary job:

• config/jobs: add metrics-bigquery-canary test-infra#23097
• Would like to try billing bq usage against kubernetes-public after I've confirmed it works with k8s-infra-prow-build
• No strong opinions on where we bill, just want to prototype what permissions are needed so that we can bill against a project of our choice

[ameukam]

$ gsutil -m rsync gs://k8s-metrics gs://k8s-project-metrics
# ...
| [7.5k/7.5k files][  5.0 GiB/  5.0 GiB] 100% Done  35.2 MiB/s ETA 00:00:00
Operation completed over 7.5k objects/5.0 GiB.

No auto sync planned regularly?

[spiffxp]

If I can get away with moving the bucket vs. having to migrate all links to the new bucket, no, no periodic sync

[spiffxp]

https://testgrid.k8s.io/wg-k8s-infra-canaries#metrics-bigquery-canary - currently failing

e.g. https://storage.googleapis.com/kubernetes-jenkins/logs/metrics-bigquery-canary/1422317015489581056/build-log.txt

BigQuery error in show operation: Unknown project 'k8s-infra-prow-build-trusted'

Hm, ok, but when I run locally with my google.com account or my personal account

$ bq query --format=prettyjson --project_id=k8s-infra-prow-build-trusted 'select count(*) from [k8s-gubernator:build.day]'
Waiting on bqjob_r5b61d60c431f26e5_0000017b08eceeab_1 ... (0s) Current status: DONE
[
  {
    "f0_": "6257"
  }
]

I ended up changing this job from a bootstrap job to decorated, I wonder if it needs to be updated to work with workload identity

[spiffxp]

kubernetes/test-infra#23102 - workload identity is my best guess for now

[spiffxp]

Turns out it was adding roles/bigquery.user to the service account for the project against which the query was being billed. A thing I did manually, so will PR a terraform change for that in a bit.

'm sure there's a much more locked down set of permissions we could try, such that the service account doesn't accidentally create datasets in k8s-infra-prow-build-trusted, but I think this is good enough for now.

[spiffxp]

Did the dance to empty out the old bucket, remove it, create a new bucket with the same name in the new GCP org/project, and sync contents back

$ gsutil -m rsync gs://k8s-metrics gs://k8s-project-metrics
$ gsutil -m rm -r gs://k8s-metrics
$ ./infra/gcp/ensure-main-projects.sh # edited to use k8s-metrics instead of k8s-project-metrics
$ gsutil -m rsync -r gs://k8s-project-metrics gs://k8s-metrics

[spiffxp]

Alright! The bucket lives in kubernetes.io, the latest job ran in kubernetes.io, and everything works:

• audit PR to prove:"
• latest job: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/metrics-bigquery/1422619249486073856
• flakes-latest.json: http://storage.googleapis.com/k8s-metrics/flakes-latest.json

Cleaning up:

• promote the k8s-infra job / get rid of the google.com job: config/jobs: promote metrics-bigquery-canary test-infra#23115
• cleanup the gs://k8s-metrics bucket, add the roles/bigquery.user binding: infra/gcp: k8s-metrics move cleanup #2451
• manually remove the gs://k8s-project-metrics bucket: Migrate test-infra metrics to wg-k8s-infra #1306 (comment)
• open a PR to remove the gs://k8s-metrics binding cleanup code: infra/gcp: setup k8s-project-triage special-case #2454

[spiffxp]

$ gsutil -m rm -r gs://k8s-project-metrics
# ...
/ [7.5k/7.5k objects] 100% Done 194.46 objects/s ETA 00:00:00
Operation completed over 7.5k objects.
Removing gs://k8s-project-metrics/...

[spiffxp]

/miletone v1.22
Looks like we are about to wrap this up before v1.22 gets released

[spiffxp]

#2454 has merged

Calling this done!

/close

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

#2454 has merged

Calling this done!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

/milestone v1.22