cmd/go: varying GOROOT_FINAL invalidates precompiled C dependencies of "net"

[lyderic]

What version of Go are you using (go version)?

$ ~/go/bin/go1.18beta1 version
go version go1.18beta1 linux/amd64

Does this issue reproduce with the latest release?

It reproduces with the latest beta of 1.18. However, it is fine with go 1.17.5!

What operating system and processor architecture are you using (go env)?

go env Output

$ ~/go/bin/go1.18beta1 version
go version go1.18beta1 linux/amd64
I [unix@go118 foo]$ ~/go/bin/go1.18beta1 env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/unix/.cache/go-build"
GOENV="/home/unix/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/unix/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/unix/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/unix/sdk/go1.18beta1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/unix/sdk/go1.18beta1/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18beta1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/unix/foo/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2623523050=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to compile this program:

$ cat main.go 
package main

import (
        t "github.com/lyderic/tools"
)

func main() {
        t.Magentaln("FOO")
}

What did you expect to see?

I expected it to compile with go 1.18 as it compiles with go 1.17

What did you see instead?

I got the following error:

$ ~/go/bin/go1.18beta1 build
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

[cherrymui]

Does you program use cgo anywhere? Is there any build tag that could cause different code to be built with Go 1.17 and Go 1.18? Thanks.

If you don't have a C compiler installed, maybe try setting CGO_ENABLED=0?

[lyderic]

I confirm that I don't have a C compiler installed. I never needed one, don't need one for go 1.17. I confirm also that I don't use cgo anywhere.

I can work around the problem indeed as follows:

$ CGO_ENABLED=0 ~/go/bin/go1.18beta1 build

However, I would say this is a regression problem as setting CGO_ENABLED is not needed when using go 1.17.

[seankhliao]

How was go1.18beta1 installed?

[lyderic]

I installed go 1.17 on Arch with pacman and I ran the following commands:

$ sudo pacman -Syyu go go-tools
$ go install golang.org/dl/go1.18beta1@latest
$ ~/go/bin/go1.18beta1 download
$ ~/go/bin/go1.18beta1 version
go version go1.18beta1 linux/amd64

[seankhliao]

Seems reproducible with just: (tried with both archlinux:base and debian container images)

package main

import _ "net/http"

func main() {}

[lyderic]

Please note that we had the same regression issue when we moved from 1.16 to 1.17: #47215

It had been fixed then.

[cherrymui]

cc @bcmills as this is perhaps related to #47215

Also cc @golang/release for if there is anything related to building the release.

Marking as release blocker as it is a regression and was previously a release blocker.

[heschi]

Ping: we think this is an RC blocker.

[bcmills]

I can reproduce this without even using a container by using a go1.18beta1 installed via go install and paring back $PATH to not include a C compiler.

I think this is specific to the way that golang.org/dl/go1.18beta1 distributes itself. Notably, it does not occur when using gotip in an identical environment:

$ go1.18beta1 build -o /dev/null .
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

$ gotip version
go version devel go1.18-d15481b8c Fri Jan 21 01:14:28 2022 +0000 linux/amd64

Still investigating the root cause.

[bcmills]

This notably does not reproduce with a go1.18beta1 built from source using make.bash:

$ go1.18beta1 build -o /dev/null .
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

$ ~/go1.18beta1/bin/go build -o /dev/null .

[bcmills]

Ah. This is probably from https://go.dev/cl/353352 (which fixed #48319).

When -trimpath is not set, the value of GOROOT_FINAL ends up in the compiled C dependencies. Because of that, we now include GOROOT_FINAL (which defaults to GOROOT) in the cache key for those dependencies.

x/build/cmd/release explicitly sets GOROOT_FINAL to /usr/local/go, but go1.18beta1 doesn't actually end up installing itself there: instead, it installs to $HOME/sdk/go1.18beta1. When you run go1.18beta1 env GOROOT it notices the mismatch and reports the real GOROOT instead, but the built C artifacts are stale because they have embedded the path /usr/local/go instead of $HOME/sdk/go1.18beta1.

So when you run go build, it tries to rebuild the C dependencies of the net package (see #47257 (comment)), and because CGO_ENABLED=1 it requires gcc to do so.

Indeed, setting GOROOT_FINAL to match the value set by x/build/cmd/release causes the precompiled .a file to no longer be stale and allows the build to succeed:

$ GOROOT_FINAL=/usr/local/go go1.18beta1 build -o /dev/null .

[bcmills]

One possible resolution might be to build the release with -trimpath, so that users could themselves build with -trimpath to avoid staleness. (That change would be up to @golang/release to decide.)

Another possible resolution is to declare that you need a C compiler in order to build with CGO_ENABLED=1. (That seems eminently reasonable to me, but it would be a change in policy.)

A third possible resolution is to declare that if you want to use a precompiled Go distribution without a C compiler, you must either set GOROOT_FINAL explicitly to the path used when building that distribution, or install that precompiled distribution to exactly its preordained GOROOT_FINAL location.

[bcmills]

As a more immediate workaround (but IMO clearly a hack), I suppose we could change golang.org/dl/internal/version to explicitly set GOROOT_FINAL itself before invoking the go command. 😅

(@golang/release: how do y'all feel about that?)

[ianlancetaylor]

When -trimpath is not set, the value of GOROOT_FINAL ends up in the compiled C dependencies. Because of that, we now include GOROOT_FINAL (which defaults to GOROOT) in the cache key for those dependencies.

Looking at #48319 the issues to be the DW_AT_comp_dir debugging entry which describes the compilation directory. But I don't understand why we can't use -fdebug-prefix-map to avoid this.

[bcmills]

I think that would be more-or-less equivalent to always using the -trimpath behavior for C dependencies in the standard library. That may be an option , although it would lose some debugging information that one might expect to be preserved when -trimpath isn't set.

[ianlancetaylor]

Seems that we already do the equivalent for the Go code in the standard library, at least in the release.

[bcmills]

My understanding is that the Go linker expands the elided GOROOT at link time to produce absolute paths in the resulting binaries:
https://cs.opensource.google/go/go/+/master:src/cmd/link/internal/ld/dwarf.go;l=1138-1168;drc=587b3c1192397393afb0ec5acd608e3dfe9f2116

I don't know whether it does that only for Go libraries, or for C dependencies of Go libraries as well. (Or do we always use the C linker if the binary includes C dependencies?)

[cherrymui]

I think that expansion only applies to Go code, as it applies to files from the FileTable from the CompilationUnit, which seems specific to Go object files.

[gopherbot]

Change https://golang.org/cl/380503 mentions this issue: internal/version: set GOROOT_FINAL and PWD in environments

[bcmills]

Absent a clear alternative, I'm going to go ahead and mail the GOROOT_FINAL change in golang.org/dl/internal/version.
It's at least a strict improvement over the status quo.

[dmitshur]

If I understand correctly, once the accepted proposal #47257 is implemented, this issue (and others like it) will stop being relevant. So that is the long-term fix for this problem, and we just need a short-term one here.

One possible resolution might be to build the release with -trimpath, so that users could themselves build with -trimpath to avoid staleness. (That change would be up to @golang/release to decide.)

It seems that it's not tenable to release with -trimpath unless that becomes the default in cmd/go as well, since we want to avoid staleness with the most common configuration, which is using defaults. Either way, this isn't relevant post-#47257, and there are better short-term alternatives below.

Another possible resolution is to declare that you need a C compiler in order to build with CGO_ENABLED=1. (That seems eminently reasonable to me, but it would be a change in policy.)

Needing a C compiler to build with CGO_ENABLED=1 seems reasonable to me too, and I don't see how things can work otherwise. This seems inevitable post-#47257, and before then, there are better short-term alternatives below.

A third possible resolution is to declare that if you want to use a precompiled Go distribution without a C compiler, you must either set GOROOT_FINAL explicitly to the path used when building that distribution, or install that precompiled distribution to exactly its preordained GOROOT_FINAL location.

As a more immediate workaround (but IMO clearly a hack), I suppose we could change golang.org/dl/internal/version to explicitly set GOROOT_FINAL itself before invoking the go command. 😅

(@golang/release: how do y'all feel about that?)

This seems like the best option to improve things before #47257 is resolved. Its only downside is it increases complexity in golang.org/dl/internal/version, but oh well.

Now that os.Executable() exists and cmd/go finds itself and infers its GOROOT from that, I wonder if GOROOT_FINAL has outlived its usefulness and can be removed, which would help remove the complexity/issues it adds. If so, this workaround in golang.org/dl/internal/version can eventually go away.

[dmitshur]

Based on analysis in #50183 (comment) and following discussion, it sounds like it's determined that cmd/go itself is working as intended for 1.18, and there are no changes planned to it.
CL 380503 will improve the behavior of the golang.org/dl/... commands.
Is that all that's needed here for 1.18, or is there anything more?

[bcmills]

Is that all that's needed here for 1.18, or is there anything more?

I'm going to run it by @rsc and @matloob later today, but my opinion at the moment is that CL 380503 is sufficient to address this issue for 1.18.

[bcmills]

@rsc pointed out that setting GOROOT_FINAL will also affect the GOROOT stamped into any user-built executables, which would be unfortunate.

We're going to see if the -fdebug-prefix-map approach is simple enough for 1.18, since that's likely equivalent to what we'll do for #47257 anyway. (Then we could remove GOROOT_FINAL from the cache key for libraries.)

[gopherbot]

Change https://golang.org/cl/380915 mentions this issue: cmd/go: avoid recording GOROOT_FINAL in precompiled C archives

[gopherbot]

Change https://golang.org/cl/380914 mentions this issue: cmd/go: refactor TestScript/build_issue48319 to check a more general property

[gopherbot]

Change https://golang.org/cl/380916 mentions this issue: cmd/go: add mv and support "! cmp" in script tests

[lyderic]

The issue is fixed for me when using version 1.18beta2 instead of 1.18beta1.

Many thanks!