Use image promoter release process

[serathius]

Using k8s release process would allow to migrate to official k8s repositories without depending on Googlers. It also supports automated build process which will make it more auditable and secure.

We should also stop depending on quay.io registry if official option can be used.

Staging repo was already setup by @brancz kubernetes/k8s.io#384

/kind feature
Tasks based on https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io

Migrate to promotion process:

• Create google group image release (@dims)
• Create staging repo (@justinsb)
• Add image promotion config in test-infra
• Add image promotion instructions in RELEASE.md
• Replace quay.io repository with k8s.gcr.io as default - we decided to support both

Automate builds:

• Create cloudbuild.yaml instructions example
• Add image build job in prow
• Update image build instructions in RELEASE.md

/cc @lilic please confirm that plan makes sense for you.

[serathius]

@dims
Can you create [email protected] google group?

[serathius]

/cc @dims

[serathius]

/cc @justinsb
Can you create staging repo for kube-state-metrics?

[lilic]

Last time there was still an issue with the tool, is this solved now? kubernetes/k8s.io#384 (comment)

[serathius]

I think yes, because gcr.io/google-containers is deprecated. Googler no longer can release new images to it.
https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/DO-NOT-MODIFY-legacy-backfill

I will confirm it with someone from sig-release

[serathius]

@bartsmykla Can you confirm that new pipeline is fully functional?

[bartsmykla]

Yes, it's working :-)

[bartsmykla]

@serathius it looks like staging repo already exists: https://console.cloud.google.com/gcr/images/k8s-staging-kube-state-metrics/GLOBAL and as kubernetes/k8s.io#384 was merged almost half a year ago I'm sure the google group was already created. :-)

[serathius]

How can I verify this? I can mark it as done

[bartsmykla]

These are the IAM roles for this project: https://github.com/kubernetes/k8s.io/blob/master/audit/projects/k8s-staging-kube-state-metrics/iam.json

I'm not entirely sure how to verify that but if I would guess looking at IAMs members of this group should have access to GCB for the project. Try to go here: https://console.cloud.google.com/navigation-error;errorUrl=%2Fcloud-build%3ForganizationId%3D&project%3Dk8s-staging-kube-state-metrics and check if it will let you in :-)

[bartsmykla]

Oooor... it should be easier to just check if you are a member of that group: https://groups.google.com/a/kubernetes.io/forum/#!myforums :-D

[serathius]

I'm not owner of kube-state-metrics so I cannot check that.

[bartsmykla]

@serathius so the only way is to wait for one of the members or @dims to look and confirm, but the only case when the group would not exist is when the reconcile tool would not work and as far as I'm aware it's not the case and every project which was added later is able to get the access to its repositories and push images to them.

[serathius]

/help

[k8s-ci-robot]

@serathius:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[serathius]

@lilic Can you confirm that group exits?
It should be visible in https://groups.google.com/a/kubernetes.io/forum/#!search/k8s-infra-staging-kube-state-metrics

[serathius]

/assign @justinsb @lilic

[lilic]

I am part of k8s-infra-staging-kube-state-metrics group yes and I can confirm I can access the. https://console.cloud.google.com/cloud-build/builds?organizationId=&project=k8s-staging-kube-state-metrics
But only for staging, do we need the production ones as per @nikhita comment:

When it is ready, we can start having prod ones too.

[serathius]

Acces to production images should only be done by automated CI. This is done by image promoter, which watches content of file https://github.com/kubernetes/k8s.io/blob/master/k8s.gcr.io/images/k8s-staging-kube-state-metrics/images.yaml

[bartsmykla]

It's exactly as @serathius said :-)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[serathius]

/remove-lifecycle stale

[lilic]

@tariq1890 said he is interested in doing this, if not I can take it. Tariq do you have time for this?

[lilic]

Both of my PRs are merged. Now how to check the process actually works 🤔

[serathius]

Thanks for handling this Lili
When we were migrating Metrics Server it required additional changes in our build rules.

• Fixing version tagging to be compatible with cloudbuild Fix cloud build kubernetes-sigs/metrics-server#475
• Stop building binary locally and docker use multi stage build Use multi-stage image building kubernetes-sigs/metrics-server#352

Hope this helps

[mrueg]

@lilic @tariq1890 anything I can do to help out here? I would be interested in getting kube-state-metrics images for arm64.

[lilic]

@mrueg the above mentioned by Marek sounds like great two things to be done in kube-state-metrics as well, thanks! :)

[mrueg]

Looks like it's not working as expected, see: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-kube-state-metrics-push-images/1285961432038379522 anyone with access to gcp can look at those logs?

[lilic]

Yes opened PR to fix this, looking at the current failure as well. PR for that out as well.

[lilic]

I believe everything was done on this list. Last PR should be #1175 We can close this issue now?

[serathius]

Could we make one full release using new pipeline before? Would make sense to test if everything works e2e

[lilic]

Sounds good, we plan on doing this with the 2.0 release.

[mrueg]

@serathius @lilic looking at https://console.cloud.google.com/gcr/images/k8s-staging-kube-state-metrics/GLOBAL/kube-state-metrics@sha256:23eb6eda398dca8a488dd49817605b914dea8ccf740168a8bf7328bc8fc5cbb6/details?tab=info it seems like there is no non-amd64 image pushed. anything that's missing here?

[mrueg]

Ah I figured, cloudbuild.yaml is calling make pushhere: https://github.com/kubernetes/kube-state-metrics/blob/master/Makefile#L114 which only creates and pushes a regular and am64 container image. That will require some rework in the Makefile then. I'll try to have a look at that in the next couple of days.

[lilic]

Seems like this worked, we are now seeing multiple arch images in gcr. Thanks @mrueg for the work!

[mrueg]

Great, let me know if you want a backport to 1.9 (I already see a 1.9.7 tag appeared on gcr?).

[lilic]

Backport sounds great!

[mrueg]

Great, let me know if you want a backport to 1.9 (I already see a 1.9.7 tag appeared on gcr?).

I can do the backport, one thing I'm wondering about is, that Makefile defines the version per cat VERSION (which leads to the docker image being updated on every push so a version becomes a moving target).

I would suggest the following logic:

• Read VERSION file
• Check if current commit is tagged with that version
• If it does, create a docker image with the version of VERSION ->
  kube-state-metrics:v1.9.8
• If it does not, create a docker image with the version of VERSION and add git shortcommit (that creates a tag per commit) e.g. kube-state-metrics:v1.9.8-aabbccdd
  or add "-next" or similar (that's a moving target then) kube-state-metrics:v1.9.8-next

@lilic @paulfantom @serathius any preference here?

[lilic]

Yes, I noticed as well have plans to correct this for when we do the 2.0 release as we are not using the image promotion process until 2.0 release, we have not discussed this yet with the maintainers. Do you mind opening a separate issue instead thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[lilic]

Since this image promotion now works and we have a good workflow, closing this issue. Thanks all!