cmd/go: locating GOROOT fails when the go command is run from the cross-compiled bin subdirectory

[tspearconquest]

What version of Go are you using (go version)?

$ /usr/local/go/bin/go version
go version go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ /usr/local/go/bin/go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/go/.cache/go-build'
GOENV='/go/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='[https://proxy.golang.org,direct'](https://proxy.golang.org%2Cdirect%27/)
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2492856274=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I'm trying to build an internal golang image based originally on the docker-library dockerfile for golang 1.20, but customized in the following ways:

1. Retrieve the sources for 1.21 instead of 1.20
2. Build a multi-architecture (AMD64 and ARM64) image as a single image with Buildkit
3. Use a multi-stage build process to copy only the relevant files from the build stage to the final stage; the final stage runs only busybox and has no package manager
4. Do the whole build without root privileges except for the package installs and adding a non-root user to the base image
5. Do a simple smoke test of the go binary before publishing the image to our internal registry (this is where the build is failing now)
6. Do some basic setup of GOPATH to set strict permissions on it while allowing the non-root user to write where it needs to for our developers' pipelines

The full dockerfile with some redaction of internal repo paths:

# syntax=docker/dockerfile:1

FROM --platform=${BUILDPLATFORM} registry.gitlab.com/my-corp/images/alpine-image/alpine:stable as builder
USER 0

ARG IMAGE_VERSION
ENV GOLANG_VERSION=${IMAGE_VERSION}
ENV PATH=/usr/local/go/bin:${PATH}

ARG BUILDARCH
RUN --mount=type=cache,id=apk-${BUILDARCH},sharing=locked,target=/var/cache/apk \
    ln -vsf /var/cache/apk /etc/apk/cache \
 && apk --update add \
    bash==5.2.15-r5 \
    gcc==12.2.1_git20220924-r10 \
    git==2.40.1-r0 \
    go==1.20.7-r0 \
    musl-dev==1.2.4-r1

USER 65532:65532
ARG TARGETARCH
WORKDIR /usr/local/go
RUN git clone -b "go${GOLANG_VERSION}" --depth 1 https://github.com/golang/go.git . \
 && export GOCACHE='/tmp/gocache' \
 && ( \
      cd /usr/local/go/src \
   && export  \
      GOAMD64='v1' \
      GOARCH="${TARGETARCH}" \
      GOOS='linux' \
      GOROOT_BOOTSTRAP="$(go env GOROOT)" \
      GOHOSTOS='linux' \
      GOHOSTARCH="${BUILDARCH}" \
   && ./make.bash \
    ) \
 && rm -rf \
    /usr/local/go/pkg/*/cmd \
    /usr/local/go/pkg/bootstrap \
    /usr/local/go/pkg/obj \
    /usr/local/go/pkg/tool/*/api \
    /usr/local/go/pkg/tool/*/go_bootstrap \
    /usr/local/go/src/cmd/dist/dist \
    "${GOCACHE}"

USER 0
RUN addgroup -S -g 65532 golang \
 && adduser -S -D -H -h /go -u 65532 -G golang golang

USER 65532:65532

FROM registry.gitlab.com/my-corp/images/busybox-image/busybox:stable as image
USER 0

ARG IMAGE_VERSION
ENV GOCACHE=/go/.cache/go-build
ENV GOENV=/go/.config/go/env
ENV GOLANG_VERSION=${IMAGE_VERSION}
ENV GOPATH=/go
# Added this and things work, so just want to confirm if this is the correct fix
# ENV GOROOT=/go
ARG TARGETARCH
ENV PATH=${GOPATH}/bin:/usr/local/go/bin/linux_${TARGETARCH}:/usr/local/go/bin:${PATH}

COPY --link --from=builder /etc/passwd /etc/passwd
COPY --link --from=builder /etc/group /etc/group
COPY --link --from=builder --chown=0:0 /usr/local/go /usr/local/go
RUN go version \ ## failing here
 && mkdir ${GOPATH} \
 && mkdir -p ${GOPATH}/.cache ${GOPATH}/.config/go ${GOPATH}/bin ${GOPATH}/pkg/mod ${GOPATH}/pkg/sumdb ${GOPATH}/src \
 && touch ${GOPATH}/.netrc \
 && chmod g+w ${GOPATH}/.netrc \
 && chmod 0755 ${GOPATH} \
 && chmod 1775 ${GOPATH}/.cache ${GOPATH}/.config/go ${GOPATH}/bin ${GOPATH}/pkg/mod ${GOPATH}/pkg/sumdb ${GOPATH}/src \
 && chgrp golang ${GOPATH}/.cache ${GOPATH}/.config/go ${GOPATH}/.netrc ${GOPATH}/bin ${GOPATH}/pkg/mod ${GOPATH}/pkg/sumdb ${GOPATH}/src

WORKDIR ${GOPATH}

USER 65532:65532

ENTRYPOINT [ "go" ]
CMD [ "version" ]

What did you expect to see?

go version output the version

What did you see instead?

The same error as in #61921

[dmitshur]

CC @bcmills, @golang/release.

[bcmills]

The cross-compiling configuration seems to be the key here. I can reproduce the failure mode by running make.bash with GOARCH=386 GOHOSTARCH=amd64 and then running the cross-compiled go command explicitly:

~/go/src$ go install -trimpath cmd

~/go/src$ ../bin/linux_386/go env GOROOT
go: cannot find GOROOT directory: 'go' binary is trimmed and GOROOT is not set

[tspearconquest]

Is this a legitimate issue or something I need to fix on my end; and if it's the latter, what's the proper way to handle a cross compile?

[bcmills]

It's a bug in cmd/go, I think, but kind of an obscure one. I don't really understand how multi-architecture images work, so it's hard to say beyond that. 😅

[tspearconquest]

Thanks! If we can help troubleshoot in any way, please let me know. Our infrastructure and my mac both run Intel/AMD but one of my colleagues (cc: @waterfoul) has an M2 mac, so we have the ability to test on either architecture without affecting anything.

[gopherbot]

Change https://go.dev/cl/521015 mentions this issue: cmd/go: find GOROOT using os.Executable when installed to GOROOT/bin/GOOS_GOARCH

[bcmills]

@gopherbot, please backport to Go 1.21. This interacts badly with the change to build cmd/go with -trimpath by default, causing cross-compiled go commands to fail to locate GOROOT if built at a release version.

[gopherbot]

Backport issue(s) opened: #62144 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[bcmills]

@tspearconquest, this should be fixed by https://go.dev/cl/521015 when that is merged.

In the meantime, you could potentially work around it by deleting the $GOROOT/VERSION file before building the toolchain, which will cause GOROOT to once again be embedded in the binaries.

[ardnew]

Just got bit by this using a GOOS=windows GOARCH=amd64 cross-compiled build (hosted on WSL) with some commit on the go1.21 branch. The error appeared with VS Code failing to update gopls by claiming to not have GOROOT defined:

# in WSL
cd $GOROOT; GOOS=windows GOARCH=amd64 ./src/make.bash

Then VS Code on Windows:

Installing 1 tool at C:\andrew\go\bin in module mode.
  gopls

Installing golang.org/x/tools/gopls@latest FAILED
{
 "code": 2,
 "killed": false,
 "signal": null,
 "cmd": "C:\\andrew\\go\\dev\\bin\\windows_amd64\\go.exe install -v golang.org/x/tools/gopls@latest",
 "stdout": "",
 "stderr": "go: cannot find GOROOT directory: 'go' binary is trimmed and GOROOT is not set\n"
}

1 tools failed to install.

gopls: failed to install gopls(golang.org/x/tools/gopls@latest): Error: Command failed: C:\andrew\go\dev\bin\windows_amd64\go.exe install -v golang.org/x/tools/gopls@latest
go: cannot find GOROOT directory: 'go' binary is trimmed and GOROOT is not set

I've deleted $GOROOT/VERSION and rebuilt, but not seeing any difference in behavior; receiving the same error messages.

[gopherbot]

Change https://go.dev/cl/521695 mentions this issue: [release-branch.go1.21] cmd/go: find GOROOT using os.Executable when installed to GOROOT/bin/GOOS_GOARCH