cmd/go: build fails requiring C compiler for runtime/cgo

[lyderic]

What version of Go are you using (go version)?

I am running go 1.16.5, but I am also testing go 1.17rc1:

$ go1.17rc1 version
go version go1.17rc1 linux/amd64
$ go version
go version go1.16.5 linux/amd64

Does this issue reproduce with the latest release?

It's ok with the latest stable release (1.16.5), not with 1.17rc1

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="linux"
I am running Archlinux, fully up to date as of July 15 2021, 10:15 BST. I am running in an LXD container (LXD 4.16)

go env Output

$ go1.17rc1 env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/unix/.cache/go-build"
GOENV="/home/unix/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/unix/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/unix/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/unix/sdk/go1.17rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/unix/sdk/go1.17rc1/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17rc1"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build640958217=/tmp/go-build -gno-record-gcc-switches"

What did you do?

This is the full script of what I ran:

I [unix@go117 foo]$ go1.17rc1 mod init foo
go: creating new go.mod: module foo
I [unix@go117 foo]$ ed main.go
main.go: No such file or directory
a
package main

import ( 
  . "github.com/lyderic/tools"
)

func main(){
  Cyanln("Hello, World!")
}
.
wq
98
I [unix@go117 foo]$ go1.17rc1 mod tidy
go: finding module for package github.com/lyderic/tools
go: downloading github.com/lyderic/tools v0.3.4
go: found github.com/lyderic/tools in github.com/lyderic/tools v0.3.4
go: downloading github.com/fatih/color v1.10.0
go: downloading github.com/sendgrid/sendgrid-go v3.10.0+incompatible
go: downloading github.com/wayneashleyberry/terminal-dimensions v1.0.0
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/sendgrid/rest v2.6.4+incompatible
go: downloading github.com/stretchr/testify v1.7.0
go: downloading golang.org/x/sys v0.0.0-20210423082822-04245dca01da
go: downloading golang.org/x/net v0.0.0-20210614182718-04defd469f4e
go: downloading github.com/davecgh/go-spew v1.1.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
I [unix@go117 foo]$ go1.17rc1 build
\# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH
I [unix@go117 foo]$ go build
I [unix@go117 foo]$ ./foo
Hello, World!

What did you expect to see?

With go 1.16.5, the above program compiles with no error. I use only one external lib: github.com/lyderic/tools

What did you see instead?

With go 1.17rc1, I get the following error:

$ go1.17rc1 build
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

Note: if I install gcc, the error (obviously) goes away. However, the program compiles with no gcc installed if I compile with go 1.16.5, so there might be a bug.

Thanks
L.

[seankhliao]

How did you install 1.16.5 and 1.17rc1 ?

[lyderic]

I installed go 1.16.5 as follows:

$ sudo pacman -S go

Then I installed go 1.17rc1 as follows:

$ go get golang.org/dl/go1.17rc1
$ go1.17rc1 download

Then I set up my PATH to include ${HOME}/go/bin

[lyderic]

To rule out any Archlinux / Pacman effect, I did a manual install of go 1.16.6 :

$ curl -LOs https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
$ sudo tar xzf go1.16.6.linux-amd64.tar.gz -C /usr/local
$ export PATH=${PATH}:/usr/local/go/bin
$ go version
go version go1.16.6 linux/amd64
$ go get golang.org/dl/go1.17rc1
go: downloading golang.org/dl v0.0.0-20210713194856-38ddc79c2163
$ ~/go/bin/go1.17rc1 download
Downloaded   0.0% (     3273 / 134784935 bytes) ...
[...]
Downloaded  91.8% (123780176 / 134784935 bytes) ...
Downloaded  93.4% (125844544 / 134784935 bytes) ...
Downloaded 100.0% (134740992 / 134784935 bytes) ...
Downloaded 100.0% (134784935 / 134784935 bytes)
Unpacking /home/unix/sdk/go1.17rc1/go1.17rc1.linux-amd64.tar.gz ...
Success. You may now run 'go1.17rc1'
$ export PATH=${PATH}:${HOME}/go/bin
$ go1.17rc1 version
go version go1.17rc1 linux/amd64

It didn't fix the problem. The most notable thing is that the simple example program I use compiles fine with 1.16 but not with go 1.17. So there is a regression issue, I guess.

Many thanks

L.

[lyderic]

I tried the above manual install on a fresh ubuntu 20.04 LXC container: same error:

$ go1.17rc1 build
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

However:

$ go version
go version go1.16.6 linux/amd64
$ go build
$ ./foo 
Hello, World!

[seankhliao]

It looks like go doesn't skip runtime/cgo when a C compiler isn't available and CGO_ENABLED isn't explicitly disabled

[root@1ba9e2afedf2 x]# go1.16.5 build -x runtime/cgo
WORK=/tmp/go-build3995079167
[root@1ba9e2afedf2 x]# go1.17rc1 build -x runtime/cgo
WORK=/tmp/go-build3635095243
mkdir -p $WORK/b001/
cd /root/sdk/go1.17rc1/src/runtime/cgo
TERM='dumb' CGO_LDFLAGS='"-g" "-O2" "-lpthread"' /root/sdk/go1.17rc1/pkg/tool/linux_amd64/cgo -objdir $WORK/b001/ -importpath runtime/cgo -import_runtime_cgo=false -import_syscall=false -- -I $WORK/b001/ -g -O2 -Wall -Werror ./cgo.go
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

cc @bcmills @jayconrod @matloob

[lyderic]

Yes indeed. If I disable CGO_ENABLED, it works:

$ go1.17rc1 build
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH
$ CGO_ENABLED=0 go1.17rc1 build
$ ./foo
Hello, World!

[jayconrod]

Closing as a duplicate of #33840. #37158, #27303, #26988 are also related.

We need a much better diagnostic here, and we should reconsider whether CGO_ENABLED should default to 1 when no C compiler is installed.

[lyderic]

The issue is that compilation works independently of how CGO_ENABLED is set when the same program is compiled with 1.16. So it looks like a regression problem to me...

$ CGO_ENABLED=0 go build
$ CGO_ENABLED=1 go build
$ CGO_ENABLED=0 go1.17rc1 build
$ CGO_ENABLED=1 go1.17rc1 build
# runtime/cgo
cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

[jayconrod]

@lyderic I expect this is because of the build cache. If you run go clean -cache then CGO_ENABLED=1 go build, does it still succeed with Go 1.16 and gcc not installed?

[lyderic]

Yes it does:

$ go clean -cache
$ CGO_ENABLED=1 go build

[jayconrod]

Interesting. Reopening to investigate this more. I wonder if we changed how the cache key is calculated?

[jayconrod]

We did change how the cache key is calculated.

I've opened #47251 with an explanation and possible solution.

[bcmills]

Ok, so here's an idea for 1.18 (if we don't do #47257 or #47251 instead).

What if we bake the C compiler ID used to build the standard library into cmd/go itself, and use that as the default ID for the default CC if exec.LookPath fails to locate a matching binary?

I guess that would allow CGO_ENABLED=1 builds to succeed, but only under very narrow circumstances. They would succeed if (and only if) both:

1. the user doesn't set any other flags that would invalidate cache entries, and
2. the user's code doesn't depend on any third-party packages with cgo build constraints, which would presumably fail to build anyway if no C compiler is present.

But that seems too fragile to me. I prefer #47251 (don't default to CGO_ENABLED=1 without a C compiler) and/or #47257 (don't ship precompiled .a files; always require a C compiler for builds that depend on cgo code).

[jayconrod]

Verified that the workaround in CL 335409 fixes this.

Discussion continues in #47257. If that's approved and implemented, it would still be an error to build a package that needs cgo (like runtime/cgo) if no C compiler is installed. However, net and os/user would not need cgo.

[gopherbot]

Change https://go.dev/cl/452677 mentions this issue: cmd/go: remove special case for prebuilt cgo library cache keys