x/build: add longtest builders for linux-arm64 #49649

bcmills · 2021-11-17T17:24:06Z

According to https://golang.org/wiki/PortingPolicy#first-class-ports, linux/arm and linux/arm64 are currently “first class ports”.

However, we have no longtest builder coverage whatsoever for these architectures. If we really intend to support them on par with linux/386 and linux/amd64, they ought to have builders running the tests at the same level of depth.

(CC @golang/release, @thanm)

The text was updated successfully, but these errors were encountered:

toothrot · 2021-11-17T18:58:43Z

I have to do some math, but I am not sure that we currently have the hardware capacity for this in the immediate term.

cagedmantis · 2022-11-17T19:19:01Z

@toothrot Now that we have linux-amd64 instances on GCP, do you see any reason why we wouldn't be able to add a longtest builder?

heschi · 2022-11-22T19:55:16Z

@cherrymui do you think you can take a look at this? For now we can only do arm64 because arm isn't working on GCE.

cherrymui · 2022-11-22T19:56:26Z

Sure. I'll take a look.

gopherbot · 2022-11-23T02:14:54Z

Change https://go.dev/cl/452858 mentions this issue: dashboard: add linux-arm64-longtest builder

For golang/go#49649. Updates golang/go#53851. Change-Id: Ifbfd2b089b909542d0568d529ba738e4b217cf5c Reviewed-on: https://go-review.googlesource.com/c/build/+/452858 Reviewed-by: Dmitri Shuralyov <[email protected]> Run-TryBot: Cherry Mui <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

cherrymui · 2022-11-23T19:28:40Z

The linux-arm64-longtest builder is up and running. But there are some test failures. I'll look into them.

cherrymui · 2022-11-23T22:38:02Z

    --- FAIL: TestScript/build_issue48319 (15.85s)
        script_test.go:134: 2022-11-23T20:27:16Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-3429322971/tmpdir1263247943/build_issue483194135768660
        script_test.go:154: 
            # Regression test for https://go.dev/issue/48319:
            # cgo builds should not include debug information from a stale GOROOT_FINAL. (0.000s)
            # This test is sensitive to cache invalidation,
            # so use a separate build cache that we can control. (0.000s)
            # Build a binary using a specific value of GOROOT_FINAL. (7.952s)
            # Now clean the cache and build using a different GOROOT_FINAL.
            # The resulting binaries should differ in their debug metadata. (7.411s)
            # Set GOROOT_FINAL back to the first value.
            # If the build is properly reproducible, the two binaries should match. (0.445s)
            > env GOROOT_FINAL=$WORK${/}goroot1
            > go build -o main.exe
            > cmp -q main.exe main1.exe
        script_test.go:154: FAIL: build_issue48319.txt:30: cmp -q main.exe main1.exe: main.exe and main1.exe differ

I looked into this. It is not that #48319 is regressed or anything to do with GOROOT_FINAL, but external linking looks nondeterministic. Somehow the (externally linked) output contains the path of a temporary file. I'll see if there is a way to prevent that.

The C toolchain on the builder is

gcc (Debian 10.2.1-6) 10.2.1 20210110

GNU ld (GNU Binutils for Debian) 2.35.2

cherrymui · 2022-11-23T22:44:22Z

   --- FAIL: TestScript/version_buildvcs_git_gpg (0.04s)
        script_test.go:134: 2022-11-23T22:15:30Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-262463489/tmpdir1357961938/version_buildvcs_git_gpg2188942372
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.043s)
            > exec gpg --batch --passphrase '' --quick-generate-key [email protected]
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-262463489/tmpdir1357961938/version_buildvcs_git_gpg2188942372/.gpupg/pubring.kbx' created
            gpg: error running '/usr/bin/gpg-agent': probably not installed
            gpg: failed to start agent '/usr/bin/gpg-agent': Configuration error
            gpg: can't connect to the agent: Configuration error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key [email protected]: exit status 2

The builder has gpg command installed, but not gpg-agent. The test fails when gpg tries to run gpg-agent. Installing gpg-agent makes it pass.

@bcmills perhaps we could make the test tolerate missing gpg-agent?
@cagedmantis should we install gpg-agent on the builder image?
Besides, on our linux-amd64-longtest builder (AMD64, not ARM64), despite we try to install gpg in our script https://cs.opensource.google/go/x/build/+/master:env/linux-x86-bullseye/Dockerfile;l=47 , it seems the builder actually does NOT have the gpg binary.

cherrymui · 2022-11-23T23:55:57Z

I ran the tests manually on gomote. Besides the two failures above, and a flaky failure of #56574, another issue is in misc/reboot it runs out of disk space.

##### ../misc/reboot
Building Go cmd/dist using /tmp/workdir/go. (devel gomote.XXXXX linux/arm64)
Building Go toolchain1 using /tmp/workdir/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
go build cmd/compile/internal/noder: # cmd/compile/internal/noder
compile: writing output: write $WORK/b148/_pkg_.a: no space left on device
go tool dist: FAILED: /tmp/TestRepeatBootstrap2475739123/001/goroot/pkg/tool/linux_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1

The machine seems to have rather small amount of disk space:

Filesystem      Size  Used Avail Use% Mounted on
overlay         5.7G  5.0G  642M  89% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/nvme0n1p1  5.7G  5.0G  642M  89% /etc/hosts

@cagedmantis would it be possible to increase the disk space? Thanks.

gopherbot · 2022-11-28T17:45:18Z

Change https://go.dev/cl/453656 mentions this issue: env/linux-arm64-bullseye: add GPG agent to the image

cagedmantis · 2022-11-28T17:49:45Z

@cherrymui @bcmills adding gpg-agent on the linux-arm64-bullseye image changes the TestScript/version_buildvcs_git_gpg test failure output to:

    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
        script_test.go:134: 2022-11-28T17:41:48Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-4248730206/tmpdir3252825733/version_buildvcs_git_gpg386675436
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.013s)
            > exec gpg --batch --passphrase '' --quick-generate-key [email protected]
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-4248730206/tmpdir3252825733/version_buildvcs_git_gpg386675436/.gpupg/pubring.kbx' created
            gpg: error running '/usr/bin/gpg-agent': exit status 2
            gpg: failed to start agent '/usr/bin/gpg-agent': General error
            gpg: can't connect to the agent: General error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key [email protected]: exit status 2

cagedmantis · 2022-11-28T17:51:08Z

I can roll the increase in disk space into another set of changes I am making. That will be mailed shortly.

cherrymui · 2022-11-28T18:05:56Z

@cagedmantis thanks for the quick update!

I'm not very familiar with gpg. @bcmills do you recognize that failure mode? Thanks.

bcmills · 2022-11-28T19:18:21Z

@bcmills perhaps we could make the test tolerate missing gpg-agent?

My understanding from https://www.gnupg.org/documentation/manuals/gnupg/GPG-Configuration-Options.html#GPG-Configuration-Options is that recent versions of gpg always require gpg-agent. I don't know why distro packagers aren't treating it as a hard dependency.

From that page:

--use-agent
--no-use-agent
This is dummy option. gpg always requires the agent.

So I think it's fine for the test to fail if gpg is installed but broken. (The user always has the option to either fix it or remove the broken tool.)

I'm not very familiar with gpg. @bcmills do you recognize that failure mode? Thanks.

I do not. I've mailed https://go.dev/cl/453775 to try to produce more useful output for debugging.

gopherbot · 2022-11-28T19:22:37Z

Change https://go.dev/cl/453775 mentions this issue: cmd/go: run the gpg command verbosely in TestScript/version_buildvcs_git_gpg

…git_gpg Also update test helper programs to avoid the deprecated io/ioutil package and fix minor formatting issues. For #49649. Change-Id: Id404acbb2795470420854d682f849d959d2080c0 Reviewed-on: https://go-review.googlesource.com/c/go/+/453775 Auto-Submit: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>

This change adds gpg-agent to the image used for linux-arm64-longtest. TestScript/version_buildvcs_git_gpg is failing because it is missing. This also adds gpg-agent and triggers a rebuild of the linux-x86-bullseye image. Upates golang/go#49649 Change-Id: I8adac58f21beeb7a1403d8d672b5a7409fd14f74 Reviewed-on: https://go-review.googlesource.com/c/build/+/453656 Reviewed-by: Carlos Amedee <[email protected]> Run-TryBot: Carlos Amedee <[email protected]> Reviewed-by: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

cagedmantis · 2022-11-28T20:25:08Z

The linux-x86-bullseye and linux-arm64-bullseye containers images have been updated with the changes in go.dev/cl/453656.

gopherbot · 2022-11-29T17:03:19Z

Change https://go.dev/cl/453955 mentions this issue: all: add additonal storage to linux-arm64-longtest builder

heschi · 2022-11-29T19:56:18Z

Retitling to reflect the fact that we don't have short-term plans to add 32-bit ARM since it isn't available on GCP at the moment.

cherrymui · 2022-11-30T16:26:39Z

Thank you @cagedmantis !

cagedmantis · 2022-11-30T20:13:23Z

TestScript/version_buildvcs_git_gpg continues to fail and the verbose flag isn't really giving us any new information:

    --- FAIL: TestScript/version_buildvcs_git_gpg (0.09s)
        script_test.go:134: 2022-11-30T19:55:13Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-761598066/tmpdir1312967028/version_buildvcs_git_gpg3191988827
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.090s)
            > exec gpg --batch --passphrase '' --quick-generate-key --verbose [email protected]
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-761598066/tmpdir1312967028/version_buildvcs_git_gpg3191988827/.gpupg/pubring.kbx' created
            gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
            gpg: error running '/usr/bin/gpg-agent': exit status 2
            gpg: failed to start agent '/usr/bin/gpg-agent': General error
            gpg: can't connect to the agent: General error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key --verbose [email protected]: exit status 2

If we explicitly start gpg-agent before calling gpg we see the following error:

    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
        script_test.go:134: 2022-11-30T19:20:43Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-2211690138/tmpdir4063638195/version_buildvcs_git_gpg1952539583
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.005s)
            > exec gpg-agent --daemon
            [stderr]
            gpg-agent[99526]: directory '/tmp/workdir/tmp/cmd-go-test-2211690138/tmpdir4063638195/version_buildvcs_git_gpg1952539583/.gpupg/private-keys-v1.d' created
            gpg-agent[99526]: socket name '/tmp/workdir/tmp/cmd-go-test-2211690138/tmpdir4063638195/version_buildvcs_git_gpg1952539583/.gpupg/S.gpg-agent' is too long
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg-agent --daemon: exit status 2

This seems like the reason why we are seeing an error when gpg is starting gpg-agent. I believe the limit is 100 characters. The socket name used above is 110 characters. Does anybody have a preferred method for addressing this? Perhaps using a temp directory instead of something in the work directory?

bcmills · 2022-11-30T21:06:51Z

Oof. That seems like an upstream bug in gpg-agent, which really ought to avoid assuming that it can create sockets in arbitrary locations. Given how fragile gpg seems to be, I wonder if we should just delete that test.

The test was added it CL 367034 — @andremarianiello, do you have any suggestions as to how we can make the gpg invocations more robust?

cherrymui · 2022-12-01T18:49:28Z

The gpg failure is interesting. When I gomote ssh to the builder and run the test, it passes.

root@43856dcab278:/tmp/workdir/go/src# go test -run=TestScript/version_buildvcs_git_gpg cmd/go
ok  	cmd/go	2.719s

But if I run it without ssh, with gomote run, it fails

$ gomote run $VM go/bin/go test -run=TestScript/version_buildvcs_git_gpg cmd/go
# Streaming results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote356094641/cherryyz-linux-arm64-longtest-0.stdout"...
vcs-test.golang.org rerouted to http://127.0.0.1:46213
https://vcs-test.golang.org rerouted to https://127.0.0.1:44069
go test proxy running at GOPROXY=http://127.0.0.1:37133/mod
--- FAIL: TestScript (0.02s)
    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
        script_test.go:134: 2022-12-01T18:39:44Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-585135258/tmpdir109437432/version_buildvcs_git_gpg800350627
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.005s)
            > exec gpg --batch --passphrase '' --quick-generate-key --verbose [email protected]
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-585135258/tmpdir109437432/version_buildvcs_git_gpg800350627/.gpupg/pubring.kbx' created
            gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
            gpg: error running '/usr/bin/gpg-agent': exit status 2
            gpg: failed to start agent '/usr/bin/gpg-agent': General error
            gpg: can't connect to the agent: General error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key --verbose [email protected]: exit status 2
FAIL
FAIL	cmd/go	0.063s
FAIL
# Wrote results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote356094641/cherryyz-linux-arm64-longtest-0.stdout".
Error running run: unable to execute go/bin/go: rpc error: code = Unknown desc = command execution failed: exit status 1

Weird. Maybe the gpg command has to connect to a terminal, despite we pass --batch and it is not actually interactive?

Also, even after CL 453656, the x86 builder, linux-amd64-longtest, still does not have gpg command installed so the test would be skipped. Weird...

andremarianiello · 2022-12-01T18:54:38Z

Oof. That seems like an upstream bug in gpg-agent, which really ought to avoid assuming that it can create sockets in arbitrary locations. Given how fragile gpg seems to be, I wonder if we should just delete that test.

The test was added it CL 367034 — @andremarianiello, do you have any suggestions as to how we can make the gpg invocations more robust?

I don't, unfortunately. Ideally we could run gpg with an agent, but it appears that gpg2 requires an agent to run. I also couldn't find a setting to control the max socket name length. I wouldn't bother me personally to remove the test, but it was requested (IIRC) by one of the reviewers of my MR, so you might want to ask for that person's opinion as well.

bcmills · 2022-12-01T19:08:11Z

@andremarianiello

you might want to ask for that person's opinion as well.

I'm pretty sure that person was me. 🙃

bcmills · 2022-12-01T19:11:14Z

@cherrymui, the difference between SSH and gomote run might have something to do with #56015 — maybe try running the test with -v in the SSH session to see what TMPDIR paths it's using?

cagedmantis · 2022-12-01T19:33:12Z

@cherrymui I just rebuilt the linux-x86-bullseye image and tagged it as latest. I created and gomote ssh into an instance and can confirm gpg is installed. I must have made some sort of error when I ran through this procedure before.

cherrymui · 2022-12-01T19:33:27Z

@bcmills thanks for the information!

SSH:

# go test -run=TestScript/version_buildvcs_git_gpg -v cmd/go
=== RUN   TestScript
vcs-test.golang.org rerouted to http://127.0.0.1:35423
https://vcs-test.golang.org rerouted to https://127.0.0.1:41031
go test proxy running at GOPROXY=http://127.0.0.1:36889/mod
=== RUN   TestScript/version_buildvcs_git_gpg
=== PAUSE TestScript/version_buildvcs_git_gpg
=== CONT  TestScript/version_buildvcs_git_gpg
    script_test.go:134: 2022-12-01T19:12:53Z
    script_test.go:136: $WORK=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854
    script_test.go:154: 
        PATH=/tmp/cmd-go-test-85347730/tmpdir2963332174/testbin:/tmp/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tmp/workdir/go/bin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=arm64
        TESTGO_GOHOSTARCH=arm64
        GOCACHE=/tmp/workdir/gocache/
        GOCOVERDIR=
        GODEBUG=
        GOEXE=
        GOEXPERIMENT=
        GOOS=linux
        TESTGO_GOHOSTOS=linux
        GOPROXY=http://127.0.0.1:36889/mod
        GOPRIVATE=
        GOROOT=/tmp/workdir/go
        GOROOT_FINAL=
        GOTRACEBACK=system
        TESTGO_GOROOT=/tmp/workdir/go
        TESTGO_VCSTEST_HOST=127.0.0.1:35423
        TESTGO_VCSTEST_TLS_HOST=127.0.0.1:41031
        TESTGO_VCSTEST_CERT=/tmp/cmd-go-test-85347730/vcstest3018419858/cert.pem
        GOSUMDB=localhost.localdev/sumdb+00000c67+AcTrnkbUA+TU4heY3hkjiSES/DSQniBqIeQ/YppAUtK6
        GONOPROXY=
        GONOSUMDB=
        GOVCS=*:all
        devnull=/dev/null
        goversion=1.20
        CMDGO_TEST_RUN_MAIN=true
        GIT_TRACE_CURL=1
        GIT_TRACE_CURL_NO_DATA=1
        GIT_REDACT_COOKIES=o,SSO,GSSO_Uberproxy
        WORK=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854
        TMPDIR=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/tmp
        GOPATH=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath
        PWD=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath/src
        
        # This test checks that VCS information is stamped into Go binaries even when
        # the current commit is signed and the use has configured git to display commit
        # signatures. (0.000s)
        > [!git] skip
        [condition not met]
        > [!exec:gpg] skip
        [condition not met]
        > [short] skip
        [condition not met]
        > env GOBIN=$GOPATH/bin
        > env GNUPGHOME=$WORK/.gpupg
        > mkdir $GNUPGHOME
        > chmod 0700 $GNUPGHOME
        # Create GPG key (2.362s)
        > exec gpg --batch --passphrase '' --quick-generate-key --verbose [email protected]
        [stderr]
        gpg: keybox '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/pubring.kbx' created
        gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
        gpg: waiting for the agent to come up ... (5s)
        gpg: connection to agent established
        gpg: writing self signature
        gpg: RSA/SHA512 signature from: "00F57831E92D5A59 [?]"
        gpg: writing key binding signature
        gpg: RSA/SHA512 signature from: "00F57831E92D5A59 [?]"
        gpg: writing public key to '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/pubring.kbx'
        gpg: /tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/trustdb.gpg: trustdb created
        gpg: using pgp trust model
        gpg: key 00F57831E92D5A59 marked as ultimately trusted
        gpg: directory '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/openpgp-revocs.d' created
        gpg: writing to '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/openpgp-revocs.d/27144942722831C9413D64E600F57831E92D5A59.rev'
        gpg: RSA/SHA512 signature from: "00F57831E92D5A59 [email protected]"
        gpg: revocation certificate stored as '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/openpgp-revocs.d/27144942722831C9413D64E600F57831E92D5A59.rev'
        > exec gpg --list-secret-keys --with-colons [email protected]
        [stdout]
        sec:u:3072:1:00F57831E92D5A59:1669921973:1732993973::u:::scESC:::+:::23::0:
        fpr:::::::::27144942722831C9413D64E600F57831E92D5A59:
        grp:::::::::1721A19CCA165EF2A4A5EC0F0C5DFFF351AF6F5D:
        uid:u::::1669921973::BECC172B7C69C614EE8645F3D9D1A9776D8A0C9D::[email protected]::::::::::0:
        ssb:u:3072:1:A55E7D2A642C0E1E:1669921973::::::e:::+:::23:
        fpr:::::::::3F4912A4EEB743DB04896097A55E7D2A642C0E1E:
        grp:::::::::B09AC1E76120C776ED01BCCEAE6F71B727CC0699:
        [stderr]
        gpg: checking the trustdb
        gpg: marginals needed: 3  completes needed: 1  trust model: pgp
        gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
        gpg: next trustdb check due at 2024-11-30
        > cp stdout keyinfo.txt
        > go run extract_key_id.go keyinfo.txt
        [stdout]
        00F57831E92D5A59> cp stdout keyid.txt
        # Initialize repo (0.127s)
        > cd repo/
        > exec git init
        [stdout]
        Initialized empty Git repository in /tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath/src/repo/.git/
        [stderr]
        hint: Using 'master' as the name for the initial branch. This default branch name
        hint: is subject to change. To configure the initial branch name to use in all
        hint: of your new repositories, which will suppress this warning, call:
        hint: 
        hint: 	git config --global init.defaultBranch <name>
        hint: 
        hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
        hint: 'development'. The just-created branch can be renamed via this command:
        hint: 
        hint: 	git branch -m <name>
        > exec git config user.email [email protected]
        > exec git config user.name 'J.R. Gopher'
        > exec git config --add log.showSignature true
        > go run ../configure_signing_key.go ../keyid.txt
        # Create signed commit (0.037s)
        > cd a
        > exec git add -A
        > exec git commit -m 'initial commit' --gpg-sign
        [stdout]
        [master (root-commit) bae5d74] initial commit
         3 files changed, 9 insertions(+)
         create mode 100644 README
         create mode 100644 a/a.go
         create mode 100644 a/go.mod
        > exec git log
        [stdout]
        commit bae5d7449cabf06cc7a205907d3e5039280c15e0
        gpg: Signature made Thu Dec  1 19:12:55 2022 UTC
        gpg:                using RSA key 27144942722831C9413D64E600F57831E92D5A59
        gpg: Good signature from "[email protected]" [ultimate]
        Author: J.R. Gopher <[email protected]>
        Date:   Thu Dec 1 19:12:55 2022 +0000
        
            initial commit
        # Verify commit signature does not interfere with versioning (0.102s)
        > go install
        > go version -m $GOBIN/a
        [stdout]
        /tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath/bin/a: devel gomote.XXXXX
        	path	example.com/a
        	mod	example.com/a	(devel)	
        	build	-buildmode=exe
        	build	-compiler=gc
        	build	CGO_ENABLED=1
        	build	CGO_CFLAGS=
        	build	CGO_CPPFLAGS=
        	build	CGO_CXXFLAGS=
        	build	CGO_LDFLAGS=
        	build	GOARCH=arm64
        	build	GOOS=linux
        	build	vcs=git
        	build	vcs.revision=bae5d7449cabf06cc7a205907d3e5039280c15e0
        	build	vcs.time=2022-12-01T19:12:55Z
        	build	vcs.modified=false
        > stdout '^\tbuild\tvcs\.revision='
        matched: 	build	vcs.revision=bae5d7449cabf06cc7a205907d3e5039280c15e0
        > stdout '^\tbuild\tvcs\.time='
        matched: 	build	vcs.time=2022-12-01T19:12:55Z
        > stdout '^\tbuild\tvcs\.modified=false$'
        matched: 	build	vcs.modified=false
--- PASS: TestScript (0.02s)
    --- PASS: TestScript/version_buildvcs_git_gpg (2.63s)
PASS
ok  	cmd/go	2.681s

gomote run:

$ gomote run $VM go/bin/go test -run=TestScript/version_buildvcs_git_gpg -v cmd/go
# Streaming results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote3455930797/cherryyz-linux-arm64-longtest-0.stdout"...
=== RUN   TestScript
vcs-test.golang.org rerouted to http://127.0.0.1:42595
https://vcs-test.golang.org rerouted to https://127.0.0.1:40005
go test proxy running at GOPROXY=http://127.0.0.1:46363/mod
=== RUN   TestScript/version_buildvcs_git_gpg
=== PAUSE TestScript/version_buildvcs_git_gpg
=== CONT  TestScript/version_buildvcs_git_gpg
    script_test.go:134: 2022-12-01T19:13:59Z
    script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812
    script_test.go:154: 
        PATH=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/testbin:/tmp/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=arm64
        TESTGO_GOHOSTARCH=arm64
        GOCACHE=/tmp/workdir/gocache
        GOCOVERDIR=
        GODEBUG=
        GOEXE=
        GOEXPERIMENT=
        GOOS=linux
        TESTGO_GOHOSTOS=linux
        GOPROXY=http://127.0.0.1:46363/mod
        GOPRIVATE=
        GOROOT=/tmp/workdir/go
        GOROOT_FINAL=
        GOTRACEBACK=system
        TESTGO_GOROOT=/tmp/workdir/go
        TESTGO_VCSTEST_HOST=127.0.0.1:42595
        TESTGO_VCSTEST_TLS_HOST=127.0.0.1:40005
        TESTGO_VCSTEST_CERT=/tmp/workdir/tmp/cmd-go-test-1077973377/vcstest1961311730/cert.pem
        GOSUMDB=localhost.localdev/sumdb+00000c67+AcTrnkbUA+TU4heY3hkjiSES/DSQniBqIeQ/YppAUtK6
        GONOPROXY=
        GONOSUMDB=
        GOVCS=*:all
        devnull=/dev/null
        goversion=1.20
        CMDGO_TEST_RUN_MAIN=true
        GIT_TRACE_CURL=1
        GIT_TRACE_CURL_NO_DATA=1
        GIT_REDACT_COOKIES=o,SSO,GSSO_Uberproxy
        WORK=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812
        TMPDIR=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/tmp
        GOPATH=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/gopath
        PWD=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/gopath/src
        
        # This test checks that VCS information is stamped into Go binaries even when
        # the current commit is signed and the use has configured git to display commit
        # signatures. (0.000s)
        > [!git] skip
        [condition not met]
        > [!exec:gpg] skip
        [condition not met]
        > [short] skip
        [condition not met]
        > env GOBIN=$GOPATH/bin
        > env GNUPGHOME=$WORK/.gpupg
        > mkdir $GNUPGHOME
        > chmod 0700 $GNUPGHOME
        # Create GPG key (0.005s)
        > exec gpg --batch --passphrase '' --quick-generate-key --verbose [email protected]
        [stderr]
        gpg: keybox '/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/.gpupg/pubring.kbx' created
        gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
        gpg: error running '/usr/bin/gpg-agent': exit status 2
        gpg: failed to start agent '/usr/bin/gpg-agent': General error
        gpg: can't connect to the agent: General error
        gpg: agent_genkey failed: No agent running
        gpg: key generation failed: No agent running
    script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key --verbose [email protected]: exit status 2
--- FAIL: TestScript (0.02s)
    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
FAIL
FAIL	cmd/go	0.055s
FAIL
# Wrote results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote3455930797/cherryyz-linux-arm64-longtest-0.stdout".
Error running run: unable to execute go/bin/go: rpc error: code = Unknown desc = command execution failed: exit status 1

It looks like SSH use /tmp based tmp dir, whereas gomote run uses /tmp/workdir/tmp. I wouldn't think this is a problem. But interestingly, on SSH, TMPDIR=/tmp/workdir/tmp go test -run=TestScript/version_buildvcs_git_gpg cmd/go fails. So it may be actually it! Not sure why gpg is sensitive to the tmp dir path, though...

cherrymui · 2022-12-01T19:37:33Z

@cagedmantis yeah, it does have gpg now. Thanks!

cherrymui · 2022-12-01T19:58:41Z

@cagedmantis with gpg installed on the linux-amd64-longtest builder, the test actually fails https://build.golang.org/log/31dc3ccbeed3498171be92216b02a7a6db44fb5a

Maybe we want to roll it back or skip the test...

cagedmantis · 2022-12-01T20:12:01Z

I've tagged linux-x86-bullseye@sha256:b7509b8ff467dca29b1606e2171ce5dd6b92efa1aba479fa3fd41778aa6d2e69 (the previous version) as latest. This should prevent the linux-amd64-longtest and linux-386-longtest builders from failing. The GPG test failure should probably become its own issue. Creating a separate issue would facilitate backporting fixes if that becomes necessary.

andremarianiello · 2022-12-01T20:49:45Z

I'm pretty sure that person was me. 🙃

WOMP, sorry! I'm fine with whatever you want to do haha!

bcmills · 2022-12-01T22:18:26Z

@cherrymui, the difference in TMPDIR lengths is the likely explanation for the difference. The struct passed to the system call to create a Unix socket has an astonishingly short hard-coded length for the path.

cherrymui · 2022-12-01T22:36:07Z

Split the gpg failure and the nondeterministic build failure to #57034 and #57035 . Thanks.

cherrymui · 2022-12-01T23:02:05Z

@bcmills exactly! Thanks!
Added strace to gpg-agent, and

write(2, "gpg-agent[35726]: socket name '/"..., 141gpg-agent[35726]: socket name '/tmp/workdir/tmp/cmd-go-test-1432085988/tmpdir2462573141/version_buildvcs_git_gpg4118434835/.gpupg/S.gpg-agent) = 141
write(2, "' is too long\n", 14' is too long

(If we want to keep the test) is there something the test script could do to work around?

cherrymui · 2022-12-05T17:31:08Z

With all the CLs/issues above the linux-arm64-longtest builder is now "ok". Thank you @cagedmantis @bcmills !

I'll watch it for a while and if it works reliably we can remove the "known issue".

cherrymui · 2022-12-06T01:29:56Z

Tried a trybot run

  2022-12-06T01:19:33Z tests_complete took 9m4.037541154s; aggregate 41m57.48768334s; saved 32m53.450142186s
  2022-12-06T01:19:33Z finish_make_and_test after 10m16.3s
  2022-12-06T01:19:33Z done all tests passed

Seems reasonably fast, faster than the linux-arm-aws and windows-386-2008 builders.
Seems okay to turn it on as trybot for release branch.

gopherbot · 2022-12-08T23:02:49Z

Change https://go.dev/cl/456043 mentions this issue: dashboard: enable linux-arm64-longtest builder on 1.20+ release branches

gopherbot · 2023-06-20T12:33:32Z

Change https://go.dev/cl/504525 mentions this issue: internal/releasetargets: set LongTestBuilder for linux-arm64

The linux/arm64 port is first class and a longtest builder was added for it during the Go 1.20 development cycle. It's a mandatory trybot on 1.20+ release branches and is working well. The builder recently found a real problem in go.dev/issue/60825, which has been resolved by now. Upgrade this builder from an advisory TryBot to have more complete release test coverage. For golang/go#40561. For golang/go#49649. For golang/go#29252. Change-Id: I1765e820deb0d34b7bb9a72aba397d87c1369ced Reviewed-on: https://go-review.googlesource.com/c/build/+/504525 Run-TryBot: Dmitri Shuralyov <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> Reviewed-by: Cherry Mui <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]>

bcmills added new-builder NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 17, 2021

bcmills added this to the Unreleased milestone Nov 17, 2021

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 17, 2021

bcmills mentioned this issue Nov 17, 2022

x/build: add longtest builders for all first class ports #56822

Closed

bcmills added this to Go Release Nov 18, 2022

heschi moved this to In Progress in Go Release Nov 22, 2022

heschi assigned cherrymui Nov 22, 2022

heschi changed the title ~~x/build: add longtest builders for linux-arm and linux-arm64~~ x/build: add longtest builders for linux-arm64 Nov 29, 2022

This was referenced Dec 1, 2022

cmd/go: TestScript/version_buildvcs_git_gpg (if enabled) fails on linux longtest builders #57034

Closed

cmd/link: external linking is nondeterministic on linux-arm64-longtest builder #57035

Closed

gopherbot closed this as completed in golang/build@8278419 Dec 9, 2022

Repository owner moved this from In Progress to Done in Go Release Dec 9, 2022

dmitshur mentioned this issue Jun 19, 2023

x/build/cmd/coordinator: include -longtest and -race builders for trybots on release branches #37827

Open

5 tasks

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 19, 2023

golang locked and limited conversation to collaborators Jun 19, 2024

gopherbot added the FrozenDueToAge label Jun 19, 2024

x/build: add longtest builders for linux-arm64 #49649

x/build: add longtest builders for linux-arm64 #49649

Comments

bcmills commented Nov 17, 2021 • edited Loading

toothrot commented Nov 17, 2021

cagedmantis commented Nov 17, 2022

heschi commented Nov 22, 2022

cherrymui commented Nov 22, 2022

gopherbot commented Nov 23, 2022

cherrymui commented Nov 23, 2022

cherrymui commented Nov 23, 2022

cherrymui commented Nov 23, 2022

cherrymui commented Nov 23, 2022 • edited Loading

gopherbot commented Nov 28, 2022

cagedmantis commented Nov 28, 2022

cagedmantis commented Nov 28, 2022

cherrymui commented Nov 28, 2022

bcmills commented Nov 28, 2022

gopherbot commented Nov 28, 2022

cagedmantis commented Nov 28, 2022

gopherbot commented Nov 29, 2022

heschi commented Nov 29, 2022

cherrymui commented Nov 30, 2022

cagedmantis commented Nov 30, 2022 • edited Loading

bcmills commented Nov 30, 2022

cherrymui commented Dec 1, 2022

andremarianiello commented Dec 1, 2022

bcmills commented Dec 1, 2022

bcmills commented Dec 1, 2022

cagedmantis commented Dec 1, 2022

cherrymui commented Dec 1, 2022

cherrymui commented Dec 1, 2022

cherrymui commented Dec 1, 2022

cagedmantis commented Dec 1, 2022

andremarianiello commented Dec 1, 2022

bcmills commented Dec 1, 2022

cherrymui commented Dec 1, 2022

cherrymui commented Dec 1, 2022

cherrymui commented Dec 5, 2022

cherrymui commented Dec 6, 2022

gopherbot commented Dec 8, 2022

gopherbot commented Jun 20, 2023

bcmills commented Nov 17, 2021 •

edited

Loading

cherrymui commented Nov 23, 2022 •

edited

Loading

cagedmantis commented Nov 30, 2022 •

edited

Loading