Give an option to install binaries from vendored dependencies

[hectorj]

Hi :) ! Small feature request:

I could use something similar to what Composer does: https://getcomposer.org/doc/06-config.md#bin-dir

The ability to install binaries from vendored dependencies.

it could look something like that:

package: my-package
bin-dir: ./bin/
install:
- package: github.com/tinylib/msgp
  version: ~1.0-beta
  #[same attributes as for an import: repo, vcs, etc.]

To clarify the use case, I would use this for pinpointing linters and code generators. Me and my colleagues have to use the same versions of these tools.

If I get some free time I'll see if I can make a PR implementing this, just created the issue to start the discussion.

[sdboyer]

I'm not sure where @mattfarina and @technosophos stand, but I view this as a major (though not necessarily bad) additional class of behavior. In the terminology of my writeup, adding this totters on the difference between having a PDM and an LPM.

The biggest issue I see with it is that it's actually got nothing to do with the depgraph of the project itself, and conflating it in could create unnecessary problems. For example: say that your actual source code depends on github.com/foo/somelib, and that one of the binaries you want also depends on that lib. If the deps are all treated as part of the same vendor - which seems to be what you're suggesting - then your project and the binary you want to install have to agree on a version of github.com/foo/somelib to use, even though, by design, they'll never have to be compiled at the same time. In other words, it's a conflict where there needn't be one.

A better design is to keep separate the dependencies for your project and any supporting binaries. That's mostly feasible, except that if you want to truly pin them, you still need to have some kind of locking in place, because it's not (yet!) safe to trust the Go ecosystem to have done that. So...there's still a need for some level of control from within your manifest, I guess.

⛅ ⛅

I need to ponder this one...

[hectorj]

I'd totally understand if this is judged out-of-scope, in which case I'll see about developing a complementary tool or just forking this one.

your project and the binary you want to install have to agree on a version

I do not see a use-case where I would want a binary and a library from a same repo but at different versions.

As I said, my use-case is mostly for code generators, and it is usually important to use the same version of the lib and the generator.

Right now I am using glide to vendor the repo, and a Makefile to install the bin from vendor.
Now that I think about it, I could indeed make a simple separate tool to install a list of binaries from the vendor dir...

The only issue I would have left is that Glide does not seem to pick-up packages imported in main packages (which totally makes sense if you don't plan on installing the binary)

[sdboyer]

I do not see a use-case where I would want a binary and a library from a same repo but at different versions.

It's just the next step here - say, wanting to have gometalinter or go-bindata. There's no reason I can think of that either of those particular tools should need a shared dep tree with your source - but maybe I've missed one?

As I said, my use-case is mostly for code generators, and it is usually important to use the same version of the lib and the generator.

Rereading the original issue, you listed linters and code generators, but I was focused on the former to the exclusion of the latter - sorry. Yes, where there is a joint binary and library, as in tinylib/msgp, they of course should be at the same version, and so should be part of your project's depgraph proper.

OTOH, tinylib/msgp is the only lib I happen to be aware of (just coincidence there) that works quite that way. As I mentioned above, I can think of no such need for go-metalinter or go-bindata. Now, that's not a reason not to support this pattern - just noting that, really, there are two distinct cases here.

The only issue I would have left is that Glide does not seem to pick-up packages imported in main packages (which totally makes sense if you don't plan on installing the binary)

This may be the rub, more than anything else. The only reasonable general assumption glide can make is that it should follow the import graph from the current project when deciding what dependencies to bring in - and that necessarily excludes any main packages contained in deps.

I suspect there's probably an elegant solution here, one that covers both use cases: glide could do the static analysis to determine whether the main project and supporting binary's depgraphs should be conjoined or not. Or, even if it's explicit/not based on analysis, a simple configuration option that lets glide then take care of the rest.

I think that'd probably end up being tricky to do everything sanely, though - not surprising, given that this starts bridging the gap between LPM and PDM. IMO, if you want to try for a PR, go for it - but for now, the best approach may be a complementary tool that works for at least some cases, helps map out the problem space a little more, and can then later be merged into glide once the implications are clearer.

Then again, it's not up to me :)

[hectorj]

Other example which behaves like tinylib/msgp: https://github.com/mailru/easyjson

[xeger]

I'd really appreciate this feature too! I don't think glide needs to grok a subpackages stanza as currently represented in glide.yaml, but it's crucial to me that I be able to install a binary from vendored source. Otherwise, Glide cannot version packages that contain a code-generator tool!

Let me provide a workaround for the people with this issue, plus illustrate the situation I'm talking about. My app uses the goa HTTP framework which includes a code-generating tool, goagen. I use make with my go project, which enables me to do this:

$(GOPATH)/bin/goagen: $(shell find vendor/github.com/goadesign/goa/goagen -name \*.go)
        cd vendor/github.com/goadesign/goa/goagen && \
        go get `glide list 2> /dev/null | grep -A100 MISSING | grep -v MISSING | awk '{$$1=$$1};1'` && \
        go build -o $(GOPATH)/bin/goagen

This helps ensure that goagen will generate code that targets the vendored goa that my app uses. However, it feels like Glide could make this substantially easier on me.

If glide could detect which of my packages/subpackages contained a main, track their dependencies, and build binaries on request, that would be superb!

I rather suspect that @mattfarina and @technosophos feel this to be mission creep, but for the code-gen case it's really necessary. As my workaround shows, it's not mission creep for glide; the fundamentals are already built into the tool. I'm happy to work on a PR if I've got buy-in on the idea.

[sdboyer]

@xeger idk what they think about scope creep, but the good news is, i'm probably gonna end up adding support for optionally incorporating dependencies' main packages into the depgraph as part of sdboyer/gps#36.

That won't be available to glide until #384 is in, though, and even then, glide will have to decide if/how to incorporate that option into its manifest.

[sdboyer]

Updating to note - I didn't actually end up including support for this in sdboyer/gps#36. However, I did open a separate issue - sdboyer/gps#42 . That doesn't mention this use case explicitly (it's focused instead on facilitating e.g. glide get), but nevertheless, it's the mechanism by which a tool would be able to easily support it.

[alienspaces]

I would also appreciate this feature! Have an application using https://github.com/mattes/migrate and now need to install the package specifically instead of being able to depend on the glide tool alone to manage dependencies.

Otherwise glide is awesome!

[roboll]

I wrote a tool to install binaries from vendor deps - https://github.com/roboll/go-vendorinstall.