cmd/go: hub+kubernetes has random cgo_gotypes build path

[bmwiedemann]

What version of Go are you using (go version)?

$ go version
go version go1.21.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/abuild/.cache/go-build'
GOENV='/home/abuild/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/abuild/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/abuild/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib64/go/1.21'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib64/go/1.21/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2297302055=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I did a double-build of hub and kubernetes1.27 in openSUSE.

# on Debian or openSUSE
osc co openSUSE:Factory/hub ; cd $_
for N in 1 2 ; do
    osc build --clean --nopreinstallimage --keep-pkg=RPMS.$N --noservice standard
    unrpm RPMS.$N/hub-2*.x86_64.rpm
    strings usr/bin/hub > $N.strings
done
diff -u {1,2}.strings

What did you expect to see?

two identical results

What did you see instead?

strings of the produced hub binary varied thusly:

 /usr/lib64/go/1.21/src/net/tcpsockopt_unix.go
 /usr/lib64/go/1.21/src/net/udpsock_posix.go
 /usr/lib64/go/1.21/src/net/unixsock_posix.go
-/tmp/go-build1489893222/b079/_cgo_gotypes.go
+/tmp/go-build4283137480/b079/_cgo_gotypes.go
 /usr/lib64/go/1.21/src/net/cgo_unix_cgo.go
 /usr/lib64/go/1.21/src/crypto/des/block.go

This was initially reported in kubernetes/kubernetes#110928

[bcmills]

osc build is not part of the Go project. Can you reproduce this problem using go build?

[bmwiedemann]

I made a more simplified reproducer with hub-2.14.2:

for i in 1 2 ; do 
  rm -rf ~/.cache/go-build
  go build -mod=vendor -gcflags 'all=-trimpath' -o bin/hub
  md5sum bin/hub
done

Both the -trimpath and removal of ~/.cache/go-build are required to trigger the issue.

[seankhliao]

Why are you using -gcflags=all=-trimpath instead of the -trimpath build flag?

[bmwiedemann]

It comes from https://github.com/mislav/hub/blob/5c547ed804368763064e51f3990851e267e88edd/Makefile#L11
from mislav/hub@39e1523 referencing #16860

Also https://github.com/kubernetes/kubernetes/blob/5ce0bd95cc6e6cf0333f6e58623344d9562ba7e2/hack/lib/golang.sh#L877

[bmwiedemann]

also kubernetes/kubernetes@bf27cad has some insight.

[bmwiedemann]

I did a quick re-test and both

for i in 1 2 ; do rm -rf ~/.cache/go-build ; go build -mod=vendor -gcflags '-trimpath' -o bin/hub && md5sum bin/hub ; done

and

for i in 1 2 ; do rm -rf ~/.cache/go-build ; go build -mod=vendor '-trimpath' -o bin/hub && md5sum bin/hub ; done

would produce bit-identical results.

What does the all= do to not cover the random /tmp/go-build4283137480/b079/_cgo_gotypes.go path?

[bcmills]

@bmwiedemann, the -gcflags flag specifies flags to be passed to the compiler (cmd/compile, formerly known as gc).

Those flags necessarily do not affect the output of the C compiler or linker or the cgo command. One of those is likely where that path is coming from.

In general the -gcflags flags are “you break it, you bought it”. To build reproducible binaries using go build, you need to pass the -trimpath flag to the go command instead.

[bmwiedemann]

Those flags necessarily do not affect the output of the C compiler or linker or the cgo command. One of those is likely where that path is coming from.

For me, this does not yet explain, why this helps for reproducibility:

- -gcflags 'all=-trimpath'
+ -gcflags '-trimpath'

We are still only passing it to the go-compiler, so this issue should not be from the C compiler or linker?

If we assume, only go -trimpath is the right way,
for hub the patch is simple, but for kubernetes rather not so much. Maybe you could work with them on kubernetes/kubernetes#110928?

[bmwiedemann]

Another observation: just by dropping the all=-trimpath it becomes reproducible (when re-building in the same path).
This seems to contradict your explanation.

[seankhliao]

closing as not an issue in the go toolchain

[bcmills]

@bmwiedemann, the -trimpath flag supported by cmd/compile requires a prefix argument (see https://pkg.go.dev/cmd/compile). So probably setting -gcflags 'all=-trimpath' is actually undoing whatever -trimpath effects cmd/go itself is actually setting on the command line for the go build invocation.

At any rate, the conclusion here is the same either way: if you want reproducible binaries, you need to pass the -trimpath flag to cmd/go itself, not to the compiler via -gcflags.

[thockin]

I started digging here, and it seems like -trimpath should be much simpler BUT... the semantic is different.

-gcflags 'all=-trimpath=$(pwd)' converts "/home/thockin/src/kube/pkg/foo/bar.go" into "pkg/foo/bar.go", but -trimpath produces "k8s.io/kubernetes/pkg/foo/bar.go". One of these is file path, one is a module. So now there are tools which fail to find the file.

Is there a simple way to convert a module path to a file path ? It looks like -trimpath doesn't take an arg.

[bcmills]

@thockin, binaries built with -trimpath from modules fundamentally cannot embed file paths for modules. The file paths would have to be relative, but there is no one consistent thing they could all be relative to. (It can't be “the module's root directory”, because not all inputs to the build are found in the same module — some of them are from $(go env GOROOT)/src, and in the general case some are from various subdirectories of $(go env GOMODCACHE).)

Instead, -trimpath records the paths as the package import path followed by the filename within the package directory.

You can get the file path for a give package import path using go list -json=ImportPath,Dir, although bear in mind that the ImportPath field is unfortunately overloaded to include more than just the import path. (I can't find a proposal to fix that at the moment, so I'm filing a new one now.)

[thockin]

It's unfortunate - we vendor everything, so we COULD resolve to file paths. It works the way we want (mostly) with -gcflags.

So right now I am hunting down the places where we get filenames which (after this change) are not actual paths, and seeing if/how to fix those.