golang: migrate to Godep alternative?

[jonboulle]

We currently recommend godep as our standard solution for vendoring dependencies in CoreOS golang projects, but there are massive inconsistencies in how it's used (e.g. using the vendor experiment or not, rewriting or not, using scripts for management), and some projects don't use it at all. And of course it provides a lot of difficulty for many developers trying to upgrade dependencies (not even going to bother to provide references for this point).

Given the number of developer hours we still lose to fiddling with Godep, and the general community sentiment towards it (e.g. projects like Kubernetes considering moving), it would be nice if we could agree on a sensible replacement with a more usable workflow, and recover a semblance of standardisation across CoreOS projects whereever possible.

The primary contender at the moment (ancedotally, see below) is glide.

Assorted non-exhaustive references:
rkt/rkt#1933
rkt/rkt#1424
coreos/coreos-baremetal#82
containers/build#186
Masterminds/glide#303 (Kubernetes + others investigating migrating)

[crawford]

"Are you used to tools such as Cargo..." LGTM

[peterbourgon]

See also etcd-io/etcd#4913 which is causing me quite some consternation.

+1 to moving away from godep and tentative +1 to glide. Please don't check in vendored dependencies for any repo which may conceivably be imported from other packages. /cc @sdboyer

[ericchiang]

As a POC I tried to move dex over to glide (with some help from @chancez).

Moving from godep

Since we'd like dex to be buildable with only a Go installation, and dex is never imported, I attempted to check in a vendor directory using glide. By default, glide assumes you don't commit this directory, so some special configuration is required.

The command to move dex from godep to glide was:

$ glide init

This created a glide.yaml file from Godeps/Godeps.json with a list of all packages and the version. As a major improvement from godep, the top level repo is versioned, and the subpackages dex depends on simply listed.

package: github.com/coreos/dex
import:
- package: github.com/coreos/go-oidc
  version: 6039032c0b15517897116d333ead8edf38792437
  subpackages:
  - http
  - jose
  - key
  - oauth2
  - oidc

To (re)populate the vendor directory, use glide's update:

$ glide update --strip-vendor --strip-vcs --update-vendored

This creates a glide.lock, which holds similar info to glide.yaml. As a semantic point, it seems that glide.yaml represents the desired state of vendor while glide.lock represents the actual state. The flags do the following:

• --strip-vendor: Remove nested vendor directories.
• --strip-vcs: Remove version control information (by default glide uses git submodules).
• --update-vendored: Actually update the vendor directory, rather than just the glide.lock file.

All of these flags are mandatory. For instance, I omitted the --update-vendored flag and glide wasn't able to figure out the correct versions for glide.lock due to --strip-vsc.

Glide downloaded the packages at the correct version and everything worked fine. It leaves a bunch of junk in the vendor directory that other tools strip out: test data, unused packages, etc. Oh well.

Updating a dependency

As a test, I tried to update a package dex depends on. To do this, I looked up the latest commit of github.com/lib/pq on GitHub, edited the glide.yaml file, and ran the same update command.

diff --git a/glide.yaml b/glide.yaml
index 905dcc9..4bdb943 100644
--- a/glide.yaml
+++ b/glide.yaml
@@ -36,7 +36,7 @@ import:
   - diff
   - pretty
 - package: github.com/lib/pq
-  version: 7175accbed18058468c07811f76440d6e8d7cf19
+  version: 3cd0097429be7d611bb644ef85b42bfb102ceea4
 - package: github.com/mailgun/mailgun-go
   version: 9578dc67692294bb7e2a6f4b15dd18c97af19440
 - package: github.com/mattn/go-sqlite3

$ glide update --strip-vendor --strip-vcs --update-vendored github.com/lib/pq

This downloads all of dex's dependencies before updating lib/pq. It updated fine even though I didn't have lib/pq in my GOPATH.

Misc notes

• Dex needs to vendor a main (like rkt's vendoredApps). glide does this just fine.
• Updating any dependency requires downloading all the dependencies if you strip .git directories. I don't know what happens if something you depend on goes away.
• Glide update does NOT depend on the state of your GOPATH. No godep restore dance.
• The update command can be shortened to glide up -s -v -u.

[chancez]

I'm 👍 for glide. I think commiting our vendored deps also needs to be potentially re-considered, or re-thought.

We commit vendor on pretty much all products for 3 main reasons that I can see:

1. We want to have reproducible builds
2. Dependency versions are not always immutable, and can disappear, or change upstream.
3. Packages can disappear

For 1 we can solve this by using some sort of tool that has a lockfile for retrieving all packages at a specified revision, glide supports this already.

For 2 and 3 we have a few choices:

One option is to continue committing the vendor dir into our repositories, I'm not a huge fan of this, as it causes problems with downstream users of our packages if they do not remove nested vendor directories. Glide will allow us to do this, and supports stripping most anything we need.

Another option is to fork upstream projects and/or re-host/mirror upstream dependencies, and use repository aliasing (a glide feature) to tell Glide where to pull upstream packages from, without changing the actual import path used in code. This adds more burden on us to maintain these copies, but that might be worth while. To fix the issue with any hosting location potentially going down, or disappearing, we should always maintain a copy somewhere.

A final option I've considered is a mixture of the above. Instead of saving dependencies into vendor, we save them to another directory which isn't special to go, perhaps vendor.dist, and then we would add vendor/* to .gitignore. This means downstream dependencies don't have to worry about us vendoring anything, but also means we have a cached copy of dependencies in our repositories. If a developer wants to skip downloading all the dependencies they can mv vendor.dist vendor and do work. But otherwise developers can use glide as usual.

[sdboyer]

Great to hear you folks are trying glide!

All of these flags are mandatory. For instance, I omitted the --update-vendored flag and glide wasn't able to figure out the correct versions for glide.lock

It's worth noting that we are actively working on getting rid of both the --strip-vcs and --strip-vendor options. Not getting rid of them in the sense of, "it's a feature and we're removing it," but getting rid of them in the sense that "stripping vcs and vendor is so much so the only sane way of doing things that we'll just always do it."

That'll also mean the --update-vendored flag goes away, too. (Or at least, you won't need to set it). So, glide update github.com/lib/pq will be all you need to do for your desired outcome.

It'll probably still be a few weeks - I have to get the new engine shipshape - but the ball is rolling, so it's just a matter of time.

[chancez]

From @ericchiang

Glide downloaded the packages at the correct version and everything worked fine. It leaves a bunch of junk in the vendor directory that other tools strip out: test data, unused packages, etc. Oh well.

Glide is planning on adding native functionality to strip packages which are not directly used IIRC, but https://github.com/sgotti/glide-vc/ does support removing the packages we dont directly import, and a few other things, which we can use to address that concern.

[bobbyrullo]

Glide update does NOT depend on the state of your GOPATH

Thats AWESOME - means much better experience with tools like gvm

[sdboyer]

🎉 all vendor, all the time 🎉

That said, the kubernetes folks - being that they're stuck on Go 1.4 due to performance regressions - did put in a request (Masterminds/glide#374) to allow godep-like GOPATH manipulating behavior. It's a low, loooow priority for us, but some specific variant of it could happen (dumping code into a separate location, /path/to/blah, and adding that GOPATH=/path/to/blah:$GOPATH)

...though again, that's only after the new engine comes in, if at all.

[mattfarina]

Folks, I thought I'd jump in here and say hi. I'm one of the authors of Glide. Along with others, like @sdboyer, we're working to make a better overall experience. I'm happy to try and answer any of your questions if you have them.