x/build/cmd/gomote: default SSH environment is still not usable on Windows

[bcmills]

What version of Go are you using (go version)?

~/go/src$ go version
go version devel go1.20-b150c629ab Mon Oct 3 15:00:34 2022 -0400 linux/amd64

~/go/src$ go version -m $(which gomote)

/usr/local/google/home/bcmills/bin/gomote: devel go1.20-b150c629ab Mon Oct 3 15:00:34 2022 -0400
        path    golang.org/x/build/cmd/gomote
        mod     golang.org/x/build      v0.0.0-20220930212151-6d81d672f5d8      h1:n9vXkRm+rbxzB4YPd3iUolVmrIu3MN3ZjkMM9siIrIw=
        dep     cloud.google.com/go/compute     v1.3.0  h1:mPL/MzDDYHsh5tHRS9mhmhWlcgClCrCa6ApQCU6wnHI=
        dep     github.com/aws/aws-sdk-go       v1.30.15        h1:Sd8QDVzzE8Sl+xNccmdj0HwMrFowv6uVUx9tGsCE1ZE=
        dep     github.com/golang/groupcache    v0.0.0-20200121045136-8c9f03a8e57e      h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
        dep     github.com/golang/protobuf      v1.5.2  h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
        dep     github.com/googleapis/gax-go/v2 v2.1.1  h1:dp3bWCh+PPO1zjRRiCSczJav13sBvG4UhNyVTa1KqdU=
        dep     github.com/jmespath/go-jmespath v0.4.0  h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
        dep     go.opencensus.io        v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
        dep     golang.org/x/net        v0.0.0-20220127200216-cd36cc0744dd      h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
        dep     golang.org/x/oauth2     v0.0.0-20211104180415-d3ed0bb246c8      h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
        dep     golang.org/x/sync       v0.0.0-20210220032951-036812b2e83c      h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
        dep     golang.org/x/sys        v0.0.0-20220722155257-8c9f86f7a55f      h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
        dep     golang.org/x/text       v0.3.7  h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
        dep     golang.org/x/time       v0.0.0-20210723032227-1f47c861a9ac      h1:7zkz7BUtwNFFqcowJ+RIgu2MaV/MapERkDIy+mwPyjs=
        dep     google.golang.org/api   v0.70.0 h1:67zQnAE0T2rB0A3CwLSas0K+SbVzSxP+zTLkQLexeiw=
        dep     google.golang.org/genproto      v0.0.0-20220222213610-43724f9ea8cf      h1:SVYXkUz2yZS9FWb2Gm8ivSlbNQzL2Z/NpPKE3RG2jWk=
        dep     google.golang.org/grpc  v1.44.0 h1:weqSxi/TMs1SqFRMHCtBgXRs8k3X39QIDEZ0pRcttUg=
        dep     google.golang.org/protobuf      v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
        build   -buildmode=exe
        build   -compiler=gc
        build   CGO_ENABLED=1
        build   CGO_CFLAGS=
        build   CGO_CPPFLAGS=
        build   CGO_CXXFLAGS=
        build   CGO_LDFLAGS=
        build   GOARCH=amd64
        build   GOOS=linux
        build   GOAMD64=v1

What did you do?

Use gomote create to create a Windows instance, then gomote putbootstrap and gomote push to populate it.

Then, use gomote ssh to connect to it.

What did you expect to see?

Since #32430 was closed, I expected to see an environment in which I could run make.bat without additional configuration.

What did you see instead?

I still had to discover and set the correct values for GOROOT_BOOTSTRAP and PATH manually, and even then I suspect that my PATH may be missing some things.

~/go/src$ mote ssh
$ /usr/bin/ssh -o CertificateFile=/tmp/.gomote/id_ed25519-162749511-cert.pub -i /usr/local/google/home/bcmills/.config/gomote/.ssh/id_ed25519 -p 2222 [email protected]
Enter passphrase for key '/usr/local/google/home/bcmills/.config/gomote/.ssh/id_ed25519':
# Welcome to the gomote ssh proxy.
# Connecting to/starting remote ssh...
#
# `gomote push` and the builders use:
# - workdir: C:\workdir
# - GOROOT: C:\workdir/go
# - GOPATH: C:\workdir/gopath
# - env: GO_BUILDER_NAME=windows-amd64-2016 GOARCH=amd64 GOHOSTARCH=amd64 GO_TEST_TIMEOUT_SCALE=2
# Happy debugging.
set GO_BUILDER_NAME=windows-amd64-2016
set GOARCH=amd64
set GOHOSTARCH=amd64
set GO_TEST_TIMEOUT_SCALE=2
set GOPATH=C:\workdir\gopath
set PATH=$PATH;C:\workdir\go\bin
cd C:\workdir\go\src
Warning: Permanently added '[localhost]:44503' (ECDSA) to the list of known hosts.
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

gopher@SERVER-2016-V7- C:\Users\gopher>set GO_BUILDER_NAME=windows-amd64-2016

gopher@SERVER-2016-V7- C:\Users\gopher>set GOARCH=amd64

gopher@SERVER-2016-V7- C:\Users\gopher>set GOHOSTARCH=amd64

gopher@SERVER-2016-V7- C:\Users\gopher>set GO_TEST_TIMEOUT_SCALE=2

gopher@SERVER-2016-V7- C:\Users\gopher>set GOPATH=C:\workdir\gopath

gopher@SERVER-2016-V7- C:\Users\gopher>set PATH=$PATH;C:\workdir\go\bin

gopher@SERVER-2016-V7- C:\Users\gopher>cd C:\workdir\go\src

gopher@SERVER-2016-V7- C:\workdir\go\src>.\make.bat
ERROR: Cannot find C:\Users\gopher\Go1.4\bin\go.exe
Set GOROOT_BOOTSTRAP to a working Go tree >= Go 1.17.

gopher@SERVER-2016-V7- C:\workdir\go\src>set GOROOT_BOOTSTRAP=C:\workdir\go1.4

gopher@SERVER-2016-V7- C:\workdir\go\src>.\make.bat
Building Go cmd/dist using C:\workdir\go1.4. (go1.17.13 windows/amd64)
go tool dist: cannot invoke C compiler ["gcc"]: exec: "gcc": executable file not found in %PATH%

Go needs a system C compiler for use with cgo.
To set a C compiler, set CC=the-compiler.
To disable cgo, set CGO_ENABLED=0.

gopher@SERVER-2016-V7- C:\workdir\go\src>set PATH=%PATH%;C:\godep\gcc64\bin

gopher@SERVER-2016-V7- C:\workdir\go\src>.\make.bat
Building Go cmd/dist using C:\workdir\go1.4. (go1.17.13 windows/amd64)
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.
---
Installed Go for windows/amd64 in C:\workdir\go
Installed commands in C:\workdir\go\bin

[bcmills]

(CC @ianlancetaylor @mknyszek from #32430)

[bcmills]

The PATH issue seems to be invalid escaping somewhere:

gopher@SERVER-2016-V7- C:\workdir\go\src>echo %PATH%
$PATH;C:\workdir\go\bin

Note that %PATH% contains the literal string $PATH, as if it were trying to expand a variable on Unix instead of Windows.

That comes from here:
https://cs.opensource.google/go/x/build/+/master:internal/coordinator/remote/ssh.go;l=323;drc=bdfd194e7635d156821351a49bba2bc00a3211ac

[gopherbot]

Change https://go.dev/cl/437817 mentions this issue: internal/coordinator/remote: use correct syntax to expand PATH on Windows

[bcmills]

https://go.dev/cl/437817 is necessary and perhaps sufficient to fix this. (I'll double-check once a coordinator is deployed with that fix.)

[bcmills]

(I'll double-check once a coordinator is deployed with that fix.)

With the most recent coordinator push, it appears that:

• on many (all?) platforms we're missing either a GOROOT_BOOTSTRAP environment variable or a PATH entry that includes the default bootstrap directory
• on Windows, we're also missing a PATH entry for the C compiler.

From what I can tell, the problem is that these are set in cmd/buildlet.baseEnv instead of provided by the coordinator.

I can think of a few possible solutions:

1. Move the handling for those environment variables from the buildlet to the coordinator.

   • However, the buildlet logic for GOROOT_BOOTSTRAP is a bit subtle, so it's not obvious to me whether this is feasible.

2. Have the coordinator execute a script on the buildlet to identify the actual environment and then use that.

   • This seems like the simplest approach to me, but perhaps not the cleanest?

3. Somehow change the way the ssh daemon is started so that it inherits buildlet environment variables.

   • This seems like the most hermetic approach, but I don't know how to implement it.

I don't plan to follow up on any of these myself at the moment, although approach (2) seems straightforward enough to be shovel-ready when someone gets the time.

[cagedmantis]

I believe these issues have been fixed. The GOROOT_BOOTSTRAP environment variable is now set properly. And the PATH on windows includes an entry for the C compiler. These are now set by golangbuild upon the creation of a gomote instance. The build environment is identical to the gomote environment on LUCI. Please feel free to reopen this issue if you believe it was closed in error.