"dep ensure pkg@^ver" causing "dep prune" to be undone and corrupts the lock file

[ahmetb]

So currently I am in a project where I successfully vended many dependencies and ran dep prune to eliminate unused packages under vendor/.

Now I am trying to add github.com/pkg/errors dependency, so I run:

dep ensure github.com/pkg/errors@^0.8.0

Once this command executes, all unused packages of my already saved dependencies come back, effectively undoing what dep prune did.

At this point I cannot execute "dep prune" anymore as it appears like the lock file is corrupted:

$ dep prune
lock hash doesn't match

At least it did vendor github.com/pkg/errors successfully but millions of lines of code I did not use came back, plus I can't run dep prune anymore. So I rm -rf Gopkg.* vendor/ and start over with dep init.

[sdboyer]

Ahh, that's a frustrating series of occurrences, I'm sorry 😞. You're hitting on a cluster of several ongoing issues, nearly all of which we have work in-flight to fix.

Once this command executes, all unused packages of my already saved dependencies come back, effectively undoing what dep prune did.

Right, this is by design, and much of why folks want prune to be absorbed into ensure.

dep treats vendor as volatile, discarding and recreating it in its entirety with each dep ensure run. And it does so the same way it always does - from the information in Gopkg.lock. It doesn't know that dep prune is part of what you want, because prune is currently a tacked-on manual step that's not part of the normal flow. So, for now, you have to rerun it each time.

(In the future we'll be able to avoid fully regenerating vendor/, but because we currently lack a vendor/ verification mechanism - #121 - safety demands that we must do the whole thing).

Now I am trying to add github.com/pkg/errors dependency

Sorta side note, but adding a new dependency is a bit of a confusing area; see #303. We do have an in-process plan (#277, #489) for making it clearer and more orthogonal.

At this point I cannot execute "dep prune" anymore as it appears like the lock file is corrupted:

It's not actually corrupted, that's just a horribly confusing error message 😢

The memo value in Gopkg.lock is a hash digest of all the inputs to the version solving process that produced that lock. When there's a mismatch between what's inGopkg.lock and what we've read from your import statements + Gopkg.toml, then we know the inputs have changed, and so things are out of sync, and you need to re-run dep ensure to get them back in alignment.

This is hard to see right now, though, because dep status is in terrible shape (#613 is the first of many steps to improve this), and, again, that error message is terrible. A little hidden easter egg is that you can run dep hash-inputs to see the inputs. Squint at that output a bit (there's little headers to break up the sections) and you should be able to get a good sense of what's going on there.

I am a little surprised you had a hash mismatch immediately after that dep ensure, though. AFTER #277 is in, it'll be possible for that to happen because of the trick with an ephemeral "required" described in #36, but it shouldn't be possible now. Did you change anything after running dep ensure github.com/pkg/errors@^0.8.0, but before dep prune?

[ahmetb]

Thanks for the detailed explanation. It just sounds interesting that dep ensure <pkg> would still impact all other packages vendored.

Did you change anything after running dep ensure github.com/pkg/errors@^0.8.0, but before dep prune?

Nope.

[sdboyer]

It just sounds interesting that dep ensure would still impact all other packages vendored.

This is the fundamental nature of the problem - it's not possible to know a priori whether any given dependency, at any given version, will (with its dependencies) constitute a disjoint subgraph, or if it will have shared dependencies that require us to reconcile with the existing depgraph and search harder for a solution. That unpredictability is one of the crucial reasons that the search - or at least, our current approach to it - is NP-hard.

Nope.

Very odd, then. Suggests we have a bug, though I'm a little loathe to devote too much energy to chasing it down when we're about to change the whole way ensure works.

[sdboyer]

Closing this in favor of #944