add pull-k8sio-backup presubmit check #15398

listx · 2019-11-26T02:57:13Z

This checks the backup_tools scripts by testing them. Specifically, it
runs the backup_test.sh script, which performs the backup from

us.gcr.io/k8s-gcr-backup-test-prod

to

us.gcr.io/k8s-gcr-backup-test-prod-bak

for a set of chosen images. Both this test and the real backup logic in
backup_prod.sh share the same "build_gcrane" and "copy_with_date"
functions --- the main difference is the repositories where the backups
happen and the images that are backed up.

NOTE: I need to provide the test-environment secret k8s-gcr-backup-test-prod-bak-service-account (i.e., k8s-infra-gcr-promoter@k8s-gcr-backup-test-prod-bak.iam.gserviceaccount.com) for this job to work.

/hold

/cc @thockin @dims @justinsb

listx · 2019-12-03T01:40:11Z

/assign @cjwagner

listx · 2019-12-03T23:33:45Z

/hold cancel

listx · 2019-12-03T23:34:50Z

Oops, forgot that I need to provide the secret for this job to work.

Re-holding...

/hold

Assigning to test-infra oncall:
/assign @cjwagner

cjwagner · 2019-12-06T01:59:16Z

config/jobs/kubernetes/sig-release/cip/container-image-promoter.yaml

+    volumes:
+    - name: creds
+      secret:
+        secretName: k8s-gcr-backup-test-prod-bak-service-account


Is it safe to have these credentials used in a presubmit and hosted on the untrusted build cluster? You should assume that any presubmit job can mount and use credentials in that cluster.

Yes --- the cred will only ever have write access to the ephemeral test resources an GCP.

cjwagner · 2019-12-06T01:59:47Z

config/jobs/kubernetes/sig-release/cip/container-image-promoter.yaml

+  # Check that changes to backup scripts are valid.
+  - name: pull-k8sio-backup
+    decorate: true
+    skip_report: false


This is the default, you can omit it.

cjwagner · 2019-12-06T02:00:58Z

config/jobs/kubernetes/sig-release/cip/container-image-promoter.yaml

+  - name: pull-k8sio-backup
+    decorate: true
+    skip_report: false
+    run_if_changed: 'infra/gcp/backup_tools/.*'


nit: ^infra/gcp/backup_tools/ probably more closely matches what you want.

cjwagner · 2019-12-06T02:02:22Z

config/jobs/kubernetes/sig-release/cip/container-image-promoter.yaml

+      - image: gcr.io/k8s-testimages/kubekins-e2e:v20191124-4beb966-1.17
+        command:
+        - infra/gcp/backup_tools/backup_test.sh
+      env:


I think your indentation is off starting on this line.

The other comments can be ignored, but I'm pretty sure this won't work as you expect.

For the record, I suppose it's troublesome that the presubmit checks passed regardless.

It is a known issue, but I haven't addressed it out of fear that strictly parsing pod specs will fail if new fields that Prow doesn't know about are used. It may not be a legitimate concern though, it is kind of hard for me to tell if that would actually be a common problem. It may be fine if

new podspec fields aren't used much in prowjobs

we keep this dependency up to date and continue to keep prow.k8s.io up to date:

test-infra/repos.bzl

Lines 1448 to 1456 in 44c22ec

go_repository(

name = "io_k8s_apimachinery",

build_file_generation = "on",

build_file_proto_mode = "disable",

importpath = "k8s.io/apimachinery",

replace = "k8s.io/apimachinery",

sum = "h1:7Kns6qqhMAQWvGkxYOLSLRZ5hJO0/5pcE5lPGP2fxUw=",

version = "v0.0.0-20190817020851-f2f3a405f61d",

)

This checks the backup_tools scripts by testing them. Specifically, it runs the backup_test.sh script, which performs the backup from us.gcr.io/k8s-gcr-backup-test-prod to us.gcr.io/k8s-gcr-backup-test-prod-bak for a set of chosen images. Both this test and the real backup logic in backup_prod.sh share the same "build_gcrane" and "copy_with_date" functions --- the main difference is the repositories where the backups happen and the images that are backed up.

listx · 2019-12-12T06:12:47Z

@cjwagner PTAL

k8s-ci-robot · 2019-12-17T00:24:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cjwagner, listx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/sig-release/cip/OWNERS~~ [cjwagner,listx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

listx · 2020-01-02T17:31:00Z

/hold cancel

k8s-ci-robot · 2020-01-02T17:39:47Z

@listx: Updated the job-config configmap in namespace default at cluster default using the following files:

key container-image-promoter.yaml using file config/jobs/kubernetes/sig-release/cip/container-image-promoter.yaml

In response to this:

This checks the backup_tools scripts by testing them. Specifically, it
runs the backup_test.sh script, which performs the backup from

us.gcr.io/k8s-gcr-backup-test-prod

to

us.gcr.io/k8s-gcr-backup-test-prod-bak

for a set of chosen images. Both this test and the real backup logic in
backup_prod.sh share the same "build_gcrane" and "copy_with_date"
functions --- the main difference is the repositories where the backups
happen and the images that are backed up.

NOTE: I need to provide the test-environment secret k8s-gcr-backup-test-prod-bak-service-account (i.e., k8s-infra-gcr-promoter@k8s-gcr-backup-test-prod-bak.iam.gserviceaccount.com) for this job to work.

/hold

/cc @thockin @dims @justinsb

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot requested review from dims, justinsb and thockin November 26, 2019 02:57

listx force-pushed the backup-presubmit-test branch from 12a8e65 to e99f047 Compare November 26, 2019 03:01

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 26, 2019

k8s-ci-robot assigned cjwagner Dec 3, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2019

cjwagner requested changes Dec 6, 2019

View reviewed changes

listx force-pushed the backup-presubmit-test branch from e99f047 to 85fcf2a Compare December 12, 2019 06:12

listx requested a review from cjwagner December 12, 2019 06:16

cjwagner approved these changes Dec 17, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 17, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2020

k8s-ci-robot merged commit f7f462d into kubernetes:master Jan 2, 2020

k8s-ci-robot added this to the v1.18 milestone Jan 2, 2020

listx mentioned this pull request Mar 21, 2020

Optimize backups kubernetes/k8s.io#677

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add pull-k8sio-backup presubmit check #15398

add pull-k8sio-backup presubmit check #15398

listx commented Nov 26, 2019 •

edited

Loading

listx commented Dec 3, 2019

listx commented Dec 3, 2019

listx commented Dec 3, 2019

cjwagner Dec 6, 2019

listx Dec 6, 2019

cjwagner Dec 6, 2019

cjwagner Dec 6, 2019

cjwagner Dec 6, 2019

cjwagner Dec 10, 2019

listx Dec 12, 2019

listx Dec 12, 2019

cjwagner Dec 17, 2019

listx commented Dec 12, 2019

k8s-ci-robot commented Dec 17, 2019

listx commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

	go_repository(
	name = "io_k8s_apimachinery",
	build_file_generation = "on",
	build_file_proto_mode = "disable",
	importpath = "k8s.io/apimachinery",
	replace = "k8s.io/apimachinery",
	sum = "h1:7Kns6qqhMAQWvGkxYOLSLRZ5hJO0/5pcE5lPGP2fxUw=",
	version = "v0.0.0-20190817020851-f2f3a405f61d",
	)

add pull-k8sio-backup presubmit check #15398

add pull-k8sio-backup presubmit check #15398

Conversation

listx commented Nov 26, 2019 • edited Loading

listx commented Dec 3, 2019

listx commented Dec 3, 2019

listx commented Dec 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

listx commented Dec 12, 2019

k8s-ci-robot commented Dec 17, 2019

listx commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

listx commented Nov 26, 2019 •

edited

Loading