x/build/cmd/gomote: make environment complete enough to be usable

[bcmills]

Every time I gomote ssh into an windows-amd64-2016 instance, I end up needing to run a bunch of boilerplate just to get make.bat to work:

~/go/src$ mote create windows-amd64-2016

~/go/src$ mote put14

~/go/src$ mote push
2019/06/04 11:55:19 Remote doesn't have "src/cmd/internal/obj/addrtype_string.go"
2019/06/04 11:55:19 Remote doesn't have "src/cmd/vendor/golang.org/x/sys/unix/syscall_linux_amd64.go"
2019/06/04 11:55:19 Remote doesn't have "test/fixedbugs/bug145.go"
2019/06/04 11:55:19 Remote doesn't have "test/fixedbugs/issue18725.go"
2019/06/04 11:55:19 Remote doesn't have "test/fixedbugs/issue4518.go"
2019/06/04 11:55:19 Remote doesn't have 8590 files (only showed 5).
2019/06/04 11:55:19 Remote lacks a VERSION file; sending a fake one
2019/06/04 11:55:22 Uploading 8591 new/changed files; 21554175 byte .tar.gz

~/go/src$ mote ssh
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

gopher@SERVER-2016-V7- C:\Users\gopher>cd C:\workdir\go\src

gopher@SERVER-2016-V7- C:\workdir\go\src>set CGO_ENABLED=0

gopher@SERVER-2016-V7- C:\workdir\go\src>set GOROOT_BOOTSTRAP=C:\workdir\go1.4

gopher@SERVER-2016-V7- C:\workdir\go\src>make.bat
Building Go cmd/dist using C:\workdir\go1.4
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.
---
Installed Go for windows/amd64 in C:\workdir\go
Installed commands in C:\workdir\go\bin

gopher@SERVER-2016-V7- C:\workdir\go\src>

That's five commands for what is logically only one operation: “give me an instance of windows-amd64-2016 set up to test my current GOROOT”.

On top of that, I end up needing to run that sequence repeatedly because the instance keeps rotting away out from under me (#28365).

That makes it really difficult to debug fixes, and produces a strong incentive to just run TryBots instead, wasting resources.

CC @golang/osp-team @aclements

[bradfitz]

There are elements of Windows-specific stuff to this issue, but the more general issue (if I risk overloading this one) is that each gomote environment should be audited to be usable. Perhaps more code should be shared so that the gomote environments can't rot, or as easily.

Also, IIRC, gomote to BSDs ssh in as "gopher" or some non-root user with pwd in /home/gopher but the buildlets run as root, in /work ($WORKDIR). That should also be made consistent, one way or another.

Tests would be nice too, even if they were manual integration tests we run occasionally by hand.

[gopherbot]

Change https://golang.org/cl/181542 mentions this issue: cmd/go: in TestScript, set $GOEXE instead of $exe

[gopherbot]

Change https://golang.org/cl/207459 mentions this issue: dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers

[bradfitz]

Related: I filed #35629 to make the Windows environments have their GUIs enabled, so RDP into them is more useful than a black cmd.exe on a black background.

[prattmic]

On a freebsd-amd64-12_2 gomote, the workdir (/tmp/workdir) is owned by and only writable by root, but gomote ssh logs in as gopher. I have to switch to root just to write files in the workdir.

[bcmills]

An experience report from debugging #50892 via gomote.

1. When I gomote ssh, I get a greeting message listing an environment, but that environment isn't actually set for me:

# `gomote push` and the builders use:
# - workdir: /Users/gopher/workdir
# - GOROOT: /Users/gopher/workdir/go
# - GOPATH: /Users/gopher/workdir/gopath
# - env: GO_BUILDER_NAME=darwin-amd64-12_0 GOROOT_BOOTSTRAP=/Users/gopher/goboot
…
gopher@gophers-Mac src % ./make.bash
ERROR: Cannot find /Users/gopher/go1.4/bin/go.

2. The terminal mapping in gomote ssh from Linux is completely broken. When I press the ⌫ key on my keyboard, the cursor advances ahead a space, but appears to maybe also delete a character from its internal buffer? When I press the left and right arrow keys the cursor doesn't move. When I press the ⌦ key on my keyboard, the console prints ~ instead of deleting a character. Maybe something in the $TERM settings is off?

3. When I try to work around (2) using gomote run, it apparently has an empty $PATH, even though gomote run (without other arguments) claims that for the -path flag “The empty string (default) does not modify the $PATH”.

4. gomote run gives an inscrutable error when I try to give it a (possibly-not-found?) relative directory:

~/go/src$ gomote run $(<~/.gomote) /bin/pwd
/Users/gopher/workdir

~/go/src$ gomote run $(<~/.gomote) ../bin/go
Error running run: Error trying to execute ../bin/go: buildlet: HTTP status 400 Bad Request: requires 'cmd' parameter

[aclements]

@bcmills , for (2), what's your local TERM and what TERM do you get once sshed? The output of stty -a on both sides may also be relevant (not that I understand most of that output).

[bcmills]

for (2), what's your local TERM and what TERM do you get once sshed?

Local $TERM is tmux-256color. Remote $TERM is also tmux-256color, which is likely related to the problem.

stty -a

Local:

~$ stty -a
speed 38400 baud; rows 105; columns 100; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>;
swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O;
min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel
iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho
-extproc

gomote:

gopher@gophers-Mac-74 ~ % stty -a
speed 38400 baud; 105 rows; 100 columns;
lflags: icanon isig iexten echo echoe echok echoke -echonl echoctl
        -echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
        -extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff -ixany -imaxbel -iutf8
        -ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
        -dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = ^@; eol2 = ^@;
        erase = ^?; intr = ^C; kill = ^U; lnext = ^V; min = 1; quit = ^\;
        reprint = ^R; start = ^Q; status = ^T; stop = ^S; susp = ^Z;
        time = 0; werase = ^W;

[bcmills]

~/go-review$ gomote run $(<~/.gomote) /tmp/buildlet/go/src/make.bash
make.bash must be run from $GOROOT/src
Error running run: exit status 1

~/go-review$ gomote run -dir=/tmp/buildlet/go/src $(<~/.gomote) make.bash
Error running run: Error trying to execute make.bash: buildlet: HTTP status 400 Bad Request: bogus 'dir' parameter

Apparently I need to repeat the full path to the command twice (once for the command and once for the -dir argument)‽

[ianlancetaylor]

I wrote https://go.dev/cl/418104 as an attempt at fixing this problem. However, I have no idea how to test it.

[gopherbot]

Change https://go.dev/cl/418104 mentions this issue: internal/coordinator/remote: set up enviroment for SSH clients

[ianlancetaylor]

Committed and tested.

[gopherbot]

Change https://go.dev/cl/418654 mentions this issue: internal/coordinator/remote: export base environment for ssh