RFC: go module proxy

[cblecker]

We're using go modules in a number of different repos, including k/k. I'm wondering if it's time we explore either using or deploying our own go module proxy.

Benefits:

• Faster download times when doing online verification in CI
• Repos wouldn't require direct in-tree vendoring, while still maintaining speed and reducing network flakes.

Downsides:

• Overhead of running/maintaining a caching proxy layer; or
• Dependency on a single outside proxy, such as proxy.golang.org

cc: @kubernetes/sig-testing

[BenTheElder]

kind has had great success with using proxy.golang.org by default.

IMHO whatever we use needs to be overridable (eg regional availability concerns) but that's easily accomplished with GOPROXY=${GOPROXY:-proxy.golang.org}

As I understand we are slated to essentially get this built in to golang 1.13 soon (defaulting to the golang.org proxy).

[BenTheElder]

See: https://proxy.golang.org/ in your browser which discusses go 1.13 and defaults.

[cblecker]

Well I'm also wondering if our CI should run a local proxy, and populate the environment variable right into the job spec. I think it could reduce network flakes, as well as speed up jobs if we don't have to re-download stuff each time.

[BenTheElder]

I think that might make sense for a non GCP Prow but significantly less sense given Prow is also in Google data centers.

Well I'm also wondering if our CI should run a local proxy, and populate the environment variable right into the job spec.

proxy.golang.org is already local to Google data centers (as is CI) -- I don't think it will get significantly more local or have better uptime. If anything I'd expect our Prow local cache to be more likely to have downtime and no SLA / SRE ...

I think it could reduce network flakes, as well as speed up jobs if we don't have to re-download stuff each time.

You still have to download, even locally -- just from a faster more optimal service with better uptime and a database of known module hashes (and I think we're better protected by sharing those with more projects). Local won't be less download-y than the standard proxy?

FWIW I've had zero flakes using it (did have plenty without it) and haven't seen any reported issues or downtime.

[akutz]

I like vendor.

[cblecker]

@akutz vendor still makes sense for k/k's project dependencies, but I'm thinking that for repos that either have simple dependencies or only are vendoring tools (like misspell), being able to remove the vendor is a positive thing.

[akutz]

Maybe. Then again, regardless of whether a dependency is small, large, or tooling — if it’s Golang-based and is or generates a compile-time requirement, then it’s beneficial to have that dependency vendored as well.

Pre-building tooling such as linters or post-build tooling such as static code analyzers, then sure, shove them into an external location all day.

[cblecker]

Example PR ditching vendor in favour of the proxy: kubernetes/community#3934
+10 −45,044

[BenTheElder]

Maybe. Then again, regardless of whether a dependency is small, large, or tooling — if it’s Golang-based and is or generates a compile-time requirement, then it’s beneficial to have that dependency vendored as well.

I used to think this but ... .git is 922M currently on my github.com/kubernetes/kubernetes clone (!) With the current source this makes the whole checkout 1.1GB (!)

Checking in vendor/ has a very real price. Cloning from GitHub is slow and this problem only increases with time.

Modules + GOPROXY allows you to only track the versions and hashes in-tree while externalizing storage to a central host (the proxy) and the local (system-wide) cache, which is much more efficient but still reproducible.

Shallow clones are nifty but don't really work with git tags (which most go repos use) and generally aren't used by most systems.

We also get the benefit of any shared proxy collectively observing hashes for modules and catching any changes to published packages. Downstream users are free to use their own local cache (EG company hosted) to further improve download speed and implement security & compliance controls.

Additionally, with checked in vendor we need to run slow CI verification downloading from many arbitrary hosts to ensure that these dependencies download as we expect (pull-kubernetes-dependencies). This job has been especially problematic as hosts go down (bitbucket mercurial comes to mind as a frequent offender). With GOPROXY we can trivially switch to an alternate cache for all downloads and move on, and eliminate this particular job all-together.

Pre-building tooling such as linters or post-build tooling such as static code analyzers, then sure, shove them into an external location all day.

This is an especially huge win if you use a separate module for these tools (repos can have multiple) as it means consumers that are not doing development do not need to download any of this linter code etc.

[akutz]

Hi @BenTheElder,

I used to think this but

I was the same way and came back around. I started coding in C and moved to C#, in a land where MS didn't provide any artifact management. Going to Java and Maven seemed like a godsend. So much so that I created nvn -- Maven for .NET -- nearly 10 years ago (and eventually contributed to codeplex and nuget).

When I first started working with Golang I immediately adopted the same behavior, using Glide to pull in dependencies at build-time and .gitignore to ensure vendor was not committed. However, this brought with it its own issues. Eventually we started committing vendor, and I haven't looked back.

I understand that for larger projects this is a hassle, but if I may misquote Cat Stevens and say the first clone is the deepest. As @cblecker said, the proxy model helps to reduce network flakes. Know what completely gets ride of them? Having all the code you need to build a project committed alongside the project. It's become such an important and useful part of my personal experience with the Go language that I truly hope Russ never completely removes the concept of vendor.

I hope @mattfarina and @sdboyer don't mind me pinging them. I'm curious what the beautiful minds primarily responsible for glide and dep think...

[BenTheElder]

... and I haven't looked back.

Looking back and re-evaluating is how we improve things, right? 🙃

understand that for larger projects this is a hassle, but if I may misquote Cat Stevens and say the first clone is the deepest.

The first clone is the one we do ~10,000 times a day in CI, and for any new contributors. It's also true for modules, but strictly smaller (quite a lot so for Kubernetes).

As @cblecker said, the proxy model helps to reduce network flakes. Know what completely gets ride of them? Having all the code you need to build a project committed alongside the project.

That's not true at all. Nothing can completely get rid of them, we have flakes today cloning repos. Cloning from source control is more bandwidth and more endpoints than downloading the current version of the sources exclusively (not to mention the caching etc.)

[akutz]

Hi @BenTheElder,

Looking back and re-evaluating is how we improve things, right?

Touche. But I think you know I meant that I haven't seen a good enough reason to change. And for what it's worth, I moved from glide to dep to go mod. So yes, I do pay attention to what is out there. I just don't think it's worth moving away from vendoring as a default behavior -- yet.

The first clone is the one we do ~10,000 times a day in CI

I thought there was some level of caching in the Prow clusters to reduce the overhead of this?

That's not true at all. Nothing can completely get rid of them

Again, fair. Please allow me to restate that it reduces the number of things that are able to flake.

For binary artifacts I'm all for having dedicated artifact management -- Git and other version control systems designed for plain-text files are terrible at managing binary dependencies. For textual dependencies, Golang has this unique feature that I think adds to the value of the language.

I suppose I just don't see a lot of difference between a Go module service and a shallow git clone. Sure, one is more straight-forward and simpler in implementation and execution -- but in the end they both just add points-of-failure.

With vendor, if you can successfully obtain the project sources you want to build, you can build the project. With go mod download and a Go module service / git, just because you have the project sources doesn't mean you will be able to build the project.

[akutz]

Hi @BenTheElder / @cblecker,

By the way, I don't think vendor or proxy.golang.org must be mutually exclusive either. I just wanted to state that explicitly in case my arguments made it appear as if I'm in blanket opposition to the latter.

[cblecker]

By the way, I don't think vendor or proxy.golang.org must be mutually exclusive either

Agreed. For something like k/k where I don't think we're ready to get rid of vendor, using the proxy still helps for those vendor verification steps.

That said, in go 1.13, we would have to manually force the use of vendor as it will be ignored by default. /shrug

[akutz]

That said, in go 1.13, we would have to manually force the use of vendor as it will be ignored by default.

And this makes me incredibly sad. It's such a useful feature. And if it's being switched to "opt-in," then it's only a matter of time before it is deprecated and removed entirely.

sigh

[BenTheElder]

I thought there was some level of caching in the Prow clusters to reduce the overhead of this?

None. There's bazel build output caching, and GitHub API call caching (for prow itself) but those are orthogonal.

Again, fair. Please allow me to restate that it reduces the number of things that are able to flake.

It only reduces by one versus the module proxy, and the module proxy so far has been more reliable than github .. it has a different workload.

For binary artifacts I'm all for having dedicated artifact management -- Git and other version control systems designed for plain-text files are terrible at managing binary dependencies. For textual dependencies, Golang has this unique feature that I think adds to the value of the language.

Many of our dependencies contain large binary artifacts (EG compiled proto, gobindata) packed into source code, so it's not really either-or.

I suppose I just don't see a lot of difference between a Go module service and a shallow git clone.

Shallow clones are pretty much off the table because git looks up tags by walking the history. You pretty much have to stop using git tags to use shallow clones, which doesn't really sound realistic or feasible for most of our projects.

Sure, one is more straight-forward and simpler in implementation and execution -- but in the end they both just add points-of-failure.

The module proxy is a single additional point over your primary source hosting, and unlike your original source hosting it can be trivially swapped out with strong guarantees that you'll get the same results. There are multiple public module proxies.

Also FWIW, what the module system actually does:

• versions are looked up at an index endpoint to avoid expensive calls against source control
• sources for versions are downloaded as archives managed by the proxy server (IE it exports an archive of the sources at a particular version and caches that)
• hashes of known versions are served so you can verify them

So in effect it's actually even cheaper than a shallow clone as you download a pre-computed compressed archive served from ~a hashmap of (dep, version) -> archive . Additionally if you already happen to have this dependency in your local system module cache from another project you get to skip the download.

That last point is especially beneficial to an ecosystem of projects sharing libraries.

For some operations like updating / verifying dependencies it is significantly less points of failure.

With vendor, if you can successfully obtain the project sources you want to build, you can build the project. With go mod download and a Go module service / git, just because you have the project sources doesn't mean you will be able to build the project.

True, however you have many ways to obtain these including multiple public proxies, a corporate / organizational proxy, and your local cache which de-dupes downloading these.

For local development you should be fine with the local cache except when deps change, and for CI we already depend on the internet and can switch caches (thanks to those gosum hashes).

By the way, I don't think vendor or proxy.golang.org must be mutually exclusive either

Right.

---

That said, in go 1.13, we would have to manually force the use of vendor as it will be ignored by default. /shrug

This. 1.13 releases this fall. Early results suggest this is going to work very well.

[akutz]

Hi @BenTheElder / @cblecker,

So perhaps this issue should be framed as "when" versus "if". I do think they're two different discussions, but if the sun is setting on vendor, then that changes things. The conversation is no longer Should we use a Go module proxy but rather:

• What is required to migrate to the proxy?
• What challenges, if any, are expected?
• What is the timeline? Does it match Go's timeline with respect to vendor?

[cblecker]

If we opt to not roll our own proxy (based on the fact that the official proxy is already google hosted, I agree that we don't need to roll our own for prow.k8s.io), then it could be as easy as injecting the variable:
GOPROXY=https://proxy.golang.org

Then any go module operation will consume that.

[akutz]

Hi @cblecker,

Agreed. I'm saying though that perhaps this conversation should be about the work required to figure out the work required to move to the use of a GOPROXY. As you and @BenTheElder have stated, once k/k projects are using GOPROXY, changing to where that points is the more trivial piece of work in the process (setting up a Go module proxy aside).

[BenTheElder]

GOPROXY is always used when you use go modules, however before go 1.13 it is direct instead of an actual proxy. When we upgrade to go 1.13 an official proxy will be enabled by default (along with modules IIUC).

So.. shouldn't require any work except:

• start using go modules for projects (we already have, and this is going to be default on in go 1.13?)
• upgrade to go 1.13 (which we'll do eventually anyhow for other reasons?)
• opt-in early to GOPROXY=https://proxy.golang.org if we want since it is already available for projects that want to start using it before we get go 1.13
• don't opt out unless we find out we need to for some reason ...

[akutz]

Hi @cblecker,

What did you just read? Because I just read @BenTheElder volunteering to lead the GOPROXY effort for k/k :)

[cblecker]

#13648
/shrug

[BenTheElder]

we already switched k/k to modules and aren't opting out of this, but we'll need to updated a few places to make sure it propagates to the build containers etc. should be straightforward :-)

[spiffxp]

/unshrug
/sig testing
/milestone v1.16
/area jobs

[k8s-ci-robot]

@spiffxp: ¯\_(ツ)_/¯

In response to this:

/unshrug
/sig testing
/milestone v1.16
/area jobs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.