[testgridshot] upload images to a bucket

[hoegaarden]

As a followup to #882 (comment) we could store the images in a google storage bucket. This would supersede #882.

That would be relatively simple, assuming the person who calls the script is a a release-manager and therefore should have access to certain buckets and the gcloud tools setup correctly already. So no more special setup if my assumptions are correct.

I am not sure, to which bucket we'd want to push the images too. Should we create a new one, should we use an already existing one?

cc: @kubernetes/release-engineering
Closes: #882
/kind cleanup

[justaugustus]

@hoegaarden -- I really like this compared to the wiki option!

Let me give some thought to what we call the bucket before I create it.
In the meantime, can you tweak this so that the images will land in a path like testgrid/$release/$timestamp-$rand?

Adding the $release in there will make it a little easier to search in the future.

[hoegaarden]

Let me give some thought to what we call the bucket before I create it.

I think I'd slightly lean towards reusing an existing bucket, mostly because I don't have a good overview of all the buckets and their intended use, and don't want to introduce yet another one.
You probably have a better overview on all of that. Can you maybe give your reasoning why you'd prefer a new bucket over using an existing one, e.g. kubernetes-release or such?

But yeah, in the end I don't care too much in which bucket the images end up in.

[hoegaarden]

Example bucket layout after c32934e:

→ gsutil ls -lr gs://$BUCKET/
gs://$BUCKET/testgridshot/:
        11  2019-11-26T12:19:48Z  gs://$BUCKET/testgridshot/

gs://$BUCKET/testgridshot/1.15/:

gs://$BUCKET/testgridshot/1.15/2019-11-27_10-18-42_0f1c8d9a/:
   1677303  2019-11-27T10:18:51Z  gs://$BUCKET/testgridshot/1.15/2019-11-27_10-18-42_0f1c8d9a/sig_release_1_15_informing_gce_windows_stable1.jpg

gs://$BUCKET/testgridshot/1.16/:

gs://$BUCKET/testgridshot/1.16/2019-11-27_10-18-56_c485091f/:
   1611709  2019-11-27T10:19:04Z  gs://$BUCKET/testgridshot/1.16/2019-11-27_10-18-56_c485091f/sig_release_1_16_informing_gce_windows_beta.jpg

gs://$BUCKET/testgridshot/1.17/:

gs://$BUCKET/testgridshot/1.17/2019-11-27_09-58-32_7eba5288/:
   1182460  2019-11-27T09:58:46Z  gs://$BUCKET/testgridshot/1.17/2019-11-27_09-58-32_7eba5288/sig_release_1_17_informing_aks_engine_azure_1_17_windows.jpg
   1627746  2019-11-27T09:58:46Z  gs://$BUCKET/testgridshot/1.17/2019-11-27_09-58-32_7eba5288/sig_release_1_17_informing_gce_cos_k8sbeta_default.jpg
   1584127  2019-11-27T09:58:46Z  gs://$BUCKET/testgridshot/1.17/2019-11-27_09-58-32_7eba5288/sig_release_1_17_informing_gce_windows_1_17.jpg

gs://$BUCKET/testgridshot/master/:

gs://$BUCKET/testgridshot/master/2019-11-27_10-03-00_77288065/:
   1204008  2019-11-27T10:03:20Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-03-00_77288065/sig_release_master_informing_aks_engine_azure_1_17_windows.jpg
   1530266  2019-11-27T10:03:20Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-03-00_77288065/sig_release_master_informing_capa_conformance_stable_k8s_master.jpg
   1582567  2019-11-27T10:03:20Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-03-00_77288065/sig_release_master_informing_capg_conformance_stable_k8s_master.jpg
    309229  2019-11-27T10:03:19Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-03-00_77288065/sig_release_master_informing_gce_master_scale_performance.jpg
   1654753  2019-11-27T10:03:19Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-03-00_77288065/sig_release_master_informing_gce_windows_master.jpg

gs://$BUCKET/testgridshot/master/2019-11-27_10-19-08_a515ae9d/:
   1203871  2019-11-27T10:19:19Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-19-08_a515ae9d/sig_release_master_informing_aks_engine_azure_1_17_windows.jpg
   1530212  2019-11-27T10:19:19Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-19-08_a515ae9d/sig_release_master_informing_capa_conformance_stable_k8s_master.jpg
   1582281  2019-11-27T10:19:19Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-19-08_a515ae9d/sig_release_master_informing_capg_conformance_stable_k8s_master.jpg
    309279  2019-11-27T10:19:19Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-19-08_a515ae9d/sig_release_master_informing_gce_master_scale_performance.jpg
   1654590  2019-11-27T10:19:20Z  gs://$BUCKET/testgridshot/master/2019-11-27_10-19-08_a515ae9d/sig_release_master_informing_gce_windows_master.jpg
TOTAL: 16 objects, 20244412 bytes (19.31 MiB)

→ 

[justaugustus]

@hoegaarden -- Made some code suggestions for the bucket name.

I think I'd slightly lean towards reusing an existing bucket, mostly because I don't have a good overview of all the buckets and their intended use, and don't want to introduce yet another one.
You probably have a better overview on all of that.

Opened #944 to document bucket usage.

Can you maybe give your reasoning why you'd prefer a new bucket over using an existing one, e.g. kubernetes-release or such?

I was thinking more about which realm I wanted the images to land in: Google-owned or K8s-Infra, primarily because the permissions are slightly different.
We currently have two projects on k8s infra, k8s-staging-release-test and k8s-release-test-prod, both of which may end up being temporary locations (the naming is not quite right).

Our primary bucket (kubernetes-release-gcb) in the kubernetes-release-test project should work fine for this.

[hoegaarden]

The default bucket has been changed to kubernetes-release-gcb.

[justaugustus]

Thanks Hannes!
/lgtm
/approve
/milestone v1.17

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hoegaarden, justaugustus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hoegaarden,justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment