Enable GOPROXY in default preset #13648

cblecker · 2019-07-28T23:48:21Z

fixes #13645.

Create a preset that adds the GOPROXY=https://proxy.golang.org env var to all jobs on prow.k8s.io.

Summary of benefits:

proxy.golang.org is a Google-run service on Borg. Should be highly reliable.
As both prow and proxy.golang.org are on the Google network, it should be faster to download.
As the cached modules don't require a full version control checkout, the download size should be smaller (read: faster)
For any jobs that don't use go modules, this is a noop.

cblecker · 2019-07-29T00:16:31Z

/hold
for discussion

/cc @BenTheElder @cjwagner @fejta @stevekuznetsov

cjwagner · 2019-07-29T17:17:50Z

prow/config.yaml

@@ -635,3 +635,7 @@ presets:
      hostPath:
        path: /sys/fs/cgroup
        type: Directory
+# enable GOPROXY by default


I don't think we support a 'default' preset that applies to all jobs implicitly and I'm not sure that we'd want to since that would be pretty opaque to job authors.

test-infra/prow/config/jobs.go

Lines 42 to 45 in c042e20

func mergePreset(preset Preset, labels map[string]string, containers []v1.Container, volumes *[]v1.Volume) error {

for l, v := range preset.Labels {

if v2, ok := labels[l]; !ok || v2 != v {

return nil

FWIW:

presets upstream are meant to be opaque for injecting environment (primarily credential) details

this setting only affects jobs that use go modules directly from within the container (not bazel or docker in docker)

in many settings this would be configured with a default on the machine (EG a corporate GOPROXY)

Setting this by default across the board makes sense, I did mention in previous discussion that the prow owners may not be OK with that though 🤷‍♂

I can confirm that the code as it is today will apply this "default" preset to every job (CI actually errors if this is set in the job spec because it merges and detects the duplicate).

+1 to everything Ben said. I can't think of any job on prow.k8s.io that wouldn't either benefit from this being set, or disregard it as a noop.

I am fine with this, but let's make sure it's documented:

https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md#presets - mention that defaulting is possible

https://github.com/kubernetes/test-infra/tree/master/config/jobs#job-presets - link to the default we use and mention it sets GOPROXY by default

I'm assuming we have tests that exercise this defaulting behavior?

Created #13665 for follow up

k8s-ci-robot · 2019-07-29T17:56:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cblecker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/OWNERS~~ [cblecker]
~~prow/OWNERS~~ [cblecker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spiffxp

this was new behavior to me, I like its use for good, but can we doc it?

spiffxp · 2019-07-29T19:00:54Z

prow/config.yaml

@@ -635,3 +635,7 @@ presets:
      hostPath:
        path: /sys/fs/cgroup
        type: Directory
+# enable GOPROXY by default


I am fine with this, but let's make sure it's documented:

https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md#presets - mention that defaulting is possible

https://github.com/kubernetes/test-infra/tree/master/config/jobs#job-presets - link to the default we use and mention it sets GOPROXY by default

I'm assuming we have tests that exercise this defaulting behavior?

spiffxp · 2019-07-29T20:24:58Z

/lgtm

k8s-ci-robot · 2019-07-29T20:25:00Z

LGTM label has been added.

Git tree hash: 35eabee58731807806ba8fb89b2bcec46df40c71

subsequent lgtm

cblecker · 2019-07-29T21:54:53Z

/hold cancel

k8s-ci-robot · 2019-07-29T22:06:52Z

@cblecker: Updated the following 2 configmaps:

job-config configmap in namespace default using the following files:
- key generated-security-jobs.yaml using file config/jobs/kubernetes-security/generated-security-jobs.yaml
- key dependencies.yaml using file config/jobs/kubernetes/sig-testing/dependencies.yaml
config configmap in namespace default using the following files:
- key config.yaml using file prow/config.yaml

In response to this:

fixes #13645.

Create a preset that adds the GOPROXY=https://proxy.golang.org env var to all jobs on prow.k8s.io.

Summary of benefits:

proxy.golang.org is a Google-run service on Borg. Should be highly reliable.

As both prow and proxy.golang.org are on the Google network, it should be faster to download.

As the cached modules don't require a full version control checkout, the download size should be smaller (read: faster)

For any jobs that don't use go modules, this is a noop.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…roxy" This reverts commit b84ad08, reversing changes made to 57a4dc6.

Revert "Merge pull request #13648 from cblecker/default-goproxy"

spiffxp · 2019-07-30T16:30:13Z

This broke pull-kubernetes-dependencies and was reverted via #13674

We can probably try this again once kubernetes/kubernetes#80732 merges

cblecker · 2019-07-30T18:14:56Z

PR to re-enable: #13686

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 28, 2019

k8s-ci-robot requested review from alvaroaleman and matthyx July 28, 2019 23:48

cblecker removed request for alvaroaleman and matthyx July 28, 2019 23:48

cblecker changed the title ~~[WIP] Enabled GOPROXY preset by default~~ Enable GOPROXY preset by default Jul 29, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2019

cblecker force-pushed the default-goproxy branch from fa19394 to e65a949 Compare July 29, 2019 00:11

cblecker changed the title ~~Enable GOPROXY preset by default~~ Enable GOPROXY in default preset Jul 29, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 29, 2019

k8s-ci-robot requested review from BenTheElder, cjwagner, fejta and stevekuznetsov July 29, 2019 00:16

cblecker mentioned this pull request Jul 29, 2019

RFC: go module proxy #13645

Closed

cjwagner reviewed Jul 29, 2019

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 29, 2019

cblecker added 2 commits July 29, 2019 10:51

Enable GOPROXY in default preset

b351a57

Remove undeeded GOPROXY from pull-kubernetes-dependencies-canary

246b49c

cblecker force-pushed the default-goproxy branch from e65a949 to 246b49c Compare July 29, 2019 17:56

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 29, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2019

spiffxp previously requested changes Jul 29, 2019

View reviewed changes

k8s-ci-robot assigned spiffxp Jul 29, 2019

cblecker mentioned this pull request Jul 29, 2019

prow: document default preset behaviour #13665

Closed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 29, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 29, 2019

k8s-ci-robot merged commit b84ad08 into kubernetes:master Jul 29, 2019

k8s-ci-robot added this to the v1.16 milestone Jul 29, 2019

cblecker deleted the default-goproxy branch July 29, 2019 22:15

cblecker added a commit to cblecker/test-infra that referenced this pull request Jul 30, 2019

Revert "Merge pull request kubernetes#13648 from cblecker/default-gop…

2fdd9cc

…roxy" This reverts commit b84ad08, reversing changes made to 57a4dc6.

k8s-ci-robot added a commit that referenced this pull request Jul 30, 2019

Merge pull request #13674 from cblecker/revert-13648

028b90c

Revert "Merge pull request #13648 from cblecker/default-goproxy"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable GOPROXY in default preset #13648

Enable GOPROXY in default preset #13648

cblecker commented Jul 28, 2019 •

edited

Loading

cblecker commented Jul 29, 2019

cjwagner Jul 29, 2019

BenTheElder Jul 29, 2019

cblecker Jul 29, 2019

spiffxp Jul 29, 2019

cblecker Jul 29, 2019

k8s-ci-robot commented Jul 29, 2019

spiffxp left a comment

spiffxp Jul 29, 2019

spiffxp commented Jul 29, 2019

k8s-ci-robot commented Jul 29, 2019

cblecker commented Jul 29, 2019

k8s-ci-robot commented Jul 29, 2019

spiffxp commented Jul 30, 2019

cblecker commented Jul 30, 2019

	func mergePreset(preset Preset, labels map[string]string, containers []v1.Container, volumes *[]v1.Volume) error {
	for l, v := range preset.Labels {
	if v2, ok := labels[l]; !ok \|\| v2 != v {
	return nil

Enable GOPROXY in default preset #13648

Enable GOPROXY in default preset #13648

Conversation

cblecker commented Jul 28, 2019 • edited Loading

cblecker commented Jul 29, 2019

cjwagner Jul 29, 2019

Choose a reason for hiding this comment

BenTheElder Jul 29, 2019

Choose a reason for hiding this comment

cblecker Jul 29, 2019

Choose a reason for hiding this comment

spiffxp Jul 29, 2019

Choose a reason for hiding this comment

cblecker Jul 29, 2019

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 29, 2019

spiffxp left a comment

Choose a reason for hiding this comment

spiffxp Jul 29, 2019

Choose a reason for hiding this comment

spiffxp commented Jul 29, 2019

k8s-ci-robot commented Jul 29, 2019

cblecker commented Jul 29, 2019

k8s-ci-robot commented Jul 29, 2019

spiffxp commented Jul 30, 2019

cblecker commented Jul 30, 2019

cblecker commented Jul 28, 2019 •

edited

Loading