cmd/go: default to CGO_ENABLED=0 when CC not set and default compiler not found

[jayconrod]

When we fixed #40042, we started including version information about the C compiler in cache action keys that use it. It looks like this:

HASH[build runtime/cgo]: "CC ID=\"gcc version 8.3.0 (Debian 8.3.0-6) \"\n"

This means that if the C compiler changes, Go packages in the cache and in GOROOT/pkg built with the old version will not be used.

Consequently, runtime/cgo, net, os/user, and other packages in std that use cgo by default are stale if the user's C compiler is different than the one used to build the Go binary distribution. It almost always is.

If the user does not have a C compiler installed and has not explicitly set CGO_ENABLED=0, go build will fail to build any package that depends on these packages, even if nothing else uses cgo. This is common when building inside a Docker container and in CI. I'm worried Go 1.17 will break a lot of builds without some mitigation. For example #47215 is caused by this.

One possibility is to default CGO_ENABLED to 0 if CC is not explicitly set, and the default C compiler (usually gcc) is not in PATH. std packages would be stale, but everything could be rebuilt without a C compiler.

This may cause subtle changes in behavior. The cgo versions of net and os/user use libc code to resolve DNS queries and find users. The pure Go versions don't use libc. Also, any package, even a pure Go package may use the cgo build tag for conditional compilation, and that would change.

This is a big behavioral change late in the release cycle, though I expect the diff will be small.

cc @rsc @ianlancetaylor @bcmills @matloob

[nishanths]

One possibility is to [...] std packages would be stale, but everything could be rebuilt without a C compiler.

With this approach, does this mean that if one makes a local changes to package net (say, to debug an issue), and then builds a binary that imports net, the binary would not include the local changes to package net?

If so, this sounds bad, since it leads to programs being built that don't match the user's expectation, and the user has no way of knowing this is happening.

[ianlancetaylor]

Interesting. I think that at this point in the release cycle we have to roll back the fix for #40042.

[jayconrod]

With this approach, does this mean that if one makes a local changes to package net (say, to debug an issue), and then builds a binary that imports net, the binary would not include the local changes to package net?

@nishanths If there are local changes to net or any other package, go build would definitely still pick those up, regardless of this issue. The SHA-256 sum of each source file is part of a package's cache key, so if anything changes, the cache key will change, and the .a file compiled earlier (whether in $GOROOT/pkg or in $GOCACHE) won't be used.

[bcmills]

Rolling back the fix for #40042 would be unfortunate, since it can lead to subtle problems in other ways (for example, linking against cached packages that were affected by subsequently-fixed bugs, or benchmarking cgo calls that do not accurately reflect the current C compiler).

As a middle ground, would it make sense to omit the CC ID part of the build ID only for packages marked as Standard for Go 1.17? That would allow the build to continue to link against the bundled standard-library packages, but still correctly rebuild stale user packages when the C compiler changes.

(Then we could consider a more holistic fix, such as the one proposed in this issue, for Go 1.18.)

[jayconrod]

Interesting. I think that at this point in the release cycle we have to roll back the fix for #40042.

@ianlancetaylor That may be the right thing to do, though @bcmills' suggestion sounds like a good alternative, keeping most of the benefit of that fix.

We've been considering removing pre-compiled .a files from the distribution altogether. I'm planning to file a proposal for that in the next day or two. If we decide to do that, perhaps we should change CGO_ENABLED at the same time, since that change would cause the same problem.

[ianlancetaylor]

Skipping the compiler ID for standard library packages occurred to me also. Let's do that for 1.17.

[gopherbot]

Change https://golang.org/cl/335409 mentions this issue: cmd/go: don't add C compiler ID to hash for standard library

[jayconrod]

After some internal discussion, I'd like to withdraw this issue.

Changing the default of CGO_ENABLED based on whether a C compiler is in PATH is too magical. This can result in a subtle but substantial change to how our users' code is built, so we need to be extremely cautious with a change like this.

Two issues motivated this:

• In cmd/go: build fails requiring C compiler for runtime/cgo #47215, we found that including the C compiler ID in the cache key required runtime/cgo and anything that depends on it (including net) to be rebuilt on most systems. CL 335409 seems like a good mitigation for this, so there's no longer any urgency here.
• In cmd/dist: remove precompiled .a files from binary distributions [freeze exception] #47257, we're looking for a way to remove precompiled .a files from the Go distribution. We'd like to build net without a C compiler installed without loss of functionality. That needs more investigation, but it seems like we can link and call C code via assembly trampolines, so we might not need actually cgo.

[bcmills]

This was implemented after all in CL 450739.