cmd/go: graph pruning not applied during 'go get'

[bcmills]

(Found while investigating anacrolix/torrent#776.)

$ go version
go version devel go1.20-8d3631fcef Thu Sep 29 09:05:07 2022 +0000 linux/amd64

If I set GOPRIVATE=github.com/anacrolix and run go get github.com/anacrolix/dht/[email protected], I get an invalid-version error for a pseudo-version that no longer exists (the commit may have been removed by a force-push).

$ export GOPRIVATE=github.com/anacrolix

$ go mod init example
go: creating new go.mod: module example

$ go get github.com/anacrolix/dht/[email protected]
go: downloading github.com/anacrolix/dht/v2 v2.19.0
go: downloading github.com/anacrolix/log v0.13.2-0.20220426014722-7b7d13a55d55
go: downloading github.com/anacrolix/missinggo/v2 v2.7.0
go: downloading github.com/anacrolix/chansync v0.3.0
go: downloading github.com/anacrolix/sync v0.4.0
go: downloading github.com/anacrolix/torrent v1.46.0
go: downloading github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8
go: downloading github.com/rs/dnscache v0.0.0-20210201191234-295bba877686
go: downloading golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
go: downloading github.com/anacrolix/multiless v0.2.1-0.20211218050420-533661eef5dc
go: downloading github.com/benbjohnson/immutable v0.3.0
go: downloading github.com/anacrolix/missinggo v1.3.0
go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
go: downloading github.com/anacrolix/stm v0.3.0
go: downloading github.com/anacrolix/missinggo/perf v1.0.0
go: downloading github.com/edsrzf/mmap-go v1.1.0
go: downloading golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e
go: downloading github.com/huandu/xstrings v1.3.2
go: github.com/anacrolix/dht/[email protected]: invalid version: unknown revision 0538803801cb

However, if I run the same command using proxy.golang.org, the operation succeeds. That isn't surprising in and of itself, because the proxy has the pseudo-version cached. What is surprising is that the invalid version is not actually in the final module graph:

$ unset GOPRIVATE

$ go get github.com/anacrolix/dht/[email protected]
go: added github.com/anacrolix/chansync v0.3.0
go: added github.com/anacrolix/dht/v2 v2.19.0
go: added github.com/anacrolix/log v0.13.2-0.20220426014722-7b7d13a55d55
go: added github.com/anacrolix/missinggo v1.3.0
go: added github.com/anacrolix/missinggo/perf v1.0.0
go: added github.com/anacrolix/missinggo/v2 v2.7.0
go: added github.com/anacrolix/multiless v0.2.1-0.20211218050420-533661eef5dc
go: added github.com/anacrolix/stm v0.3.0
go: added github.com/anacrolix/sync v0.4.0
go: added github.com/anacrolix/torrent v1.46.0
go: added github.com/benbjohnson/immutable v0.3.0
go: added github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8
go: added github.com/edsrzf/mmap-go v1.1.0
go: added github.com/huandu/xstrings v1.3.2
go: added github.com/rs/dnscache v0.0.0-20210201191234-295bba877686
go: added golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
go: added golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e
go: added golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac

$ go mod graph | grep -o '[^ ]*dht[^ ]*' | sort -u
github.com/anacrolix/dht/[email protected]
github.com/anacrolix/dht/[email protected]

If I clone the repo, check out v2.19.0, and run go mod tidy, it appears to be tidy and indeed does not depend in any way on v2.15.2-0.20220123034220-0538803801cb.

This suggests that there is a bug somewhere in go get that is causing it to follow dependency edges that should actually be pruned out.

[Moniseeta]

Hello

faced the same issue and it is really difficult when we have multiple dependent packages using same modules.
Any plans to pick this up or any workaround?

[anacrolix]

I'm bumping into this weekly. I don't know if it's also the cause, but go get becomes incredibly slow, taking 5-10 minutes or more sometimes.

[bcmills]

Yep. I'm still actively working on it. Unfortunately go get is very complicated and progress is slower than I'd like.

[anacrolix]

Does this same thing (the pruning issue) occur for go mod tidy and go mod why? I am able to reliably crash my Macbook for one of my projects now, and I can only think of this issue. CPU use goes up to 300%, memory slowly climbs to 500MB and then suddenly my laptop stops responding and reboots a few minutes later. I have no idea how that's possible. It only occurs when I have Go workspaces turned on and in use: If I disable Go workspaces with GOWORK=off, it's slow but doesn't crash.

[bcmills]

@anacrolix, only go get is affected by this particular bug. If you're having trouble with go mod tidy or go mod why, please file a separate issue for that because the root cause is guaranteed to be different. 😅

[bcmills]

I've mailed CL 471595 with a proposed fix. Folks who are affected by this bug: please try out a build with that change (either now or after it is merged) and let me know if any problems remain.

That change may also fix some of the “infinite hang” problems, since it fixes a source of errors in the build list. (go get iterates until the build list stabilizes, and I observed some infinite loops in testing caused by pruned dependencies being reintroduced or repeatedly re-eliminated.)

[gopherbot]

Change https://go.dev/cl/471595 mentions this issue: cmd/go: prune more dependencies in 'go get'

[anacrolix]

@bcmills The CL fixes this issue, and all the issues I've linked from here.

[ryancurrah]

@bcmills will this fix be back ported to Go 1.19?

[bcmills]

will this fix be back ported to Go 1.19?

That's not clear to me at this time.

In principle it could be (I don't expect significant conflicts in the backport change), but it's a very large change. On the other hand, almost all of the change is isolated within go get, which is increasingly broken without it.

[bcmills]

@gopherbot, please backport to Go 1.19 and 1.20. This bug causes go get to fail in very confusing ways; the fix is very invasive but mostly only affects 'go get'. (It isn't clear to me whether we should approve these backports, but I'm filing them for discussion.)

[gopherbot]

Backport issue(s) opened: #60159 (for 1.19), #60160 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.