Use -distpack to create the tar.gz and zip archives of our builds

[dagood]

When we set up the project, Go created the binary archives with a series of commands run through their buildlet APIs. Instead of trying to incorporate that in our AzDO builds, it was much easier to create a Go command that reproduced the end results: eng/_core/archive/archive.go.

As part of a reproducibility push, Go is putting the ability to create a Go binary archive into the Go repo itself with -distpack:

• x/build/cmd/relui: ensure fully reproducible builds, including tar.gz/zip archives golang/go#24904
• https://go-review.googlesource.com/c/go/+/470676

We should use this so we don't have to maintain our own packing, which can break when upstream Go makes layout changes. (#904)

[dagood]

I tried upstream distpack with our patch files applied. I hit an issue where distpack requires a VERSION file, so I defined a fake go1.21 one and ran it. We'll probably want to have our build wrapper place a "dev" VERSION file in main branch CI.

The first thing I noticed is that the distpack outputs don't have tar entries for each directory like ours do (and their old archives did). So, I did some cleanup to only show the file diffs.

On linux-amd64, the produced source archive has this diff vs. what we would produce:

-go/.gitattributes
-go/.github/CODE_OF_CONDUCT.md
-go/.github/ISSUE_TEMPLATE/00-bug.md
-go/.github/ISSUE_TEMPLATE/01-pkgsite.md
-go/.github/ISSUE_TEMPLATE/02-pkgsite-removal.md
-go/.github/ISSUE_TEMPLATE/03-gopls.md
-go/.github/ISSUE_TEMPLATE/04-vuln.md
-go/.github/ISSUE_TEMPLATE/10-proposal.md
-go/.github/ISSUE_TEMPLATE/11-language-change.md
-go/.github/ISSUE_TEMPLATE/config.yml
-go/.github/PULL_REQUEST_TEMPLATE
-go/.github/SUPPORT.md
-go/.gitignore

So: just a little more filtering than we do. The source diff looks good to me.

The binary archive has this diff (I ran our pack after upstream distpack, so some junk files are present in our archive):

--- a/ours.txt
+++ b/theirs.txt
@@ -65,6 +65,7 @@ go/api/next/59339.txt
 go/api/README
 go/bin/go
 go/bin/gofmt
+go/codereview.cfg
 go/CONTRIBUTING.md
 go/doc/asm.html
 go/doc/go1.17_spec.html
@@ -72,6 +73,7 @@ go/doc/go1.21.html
 go/doc/godebug.md
 go/doc/go_mem.html
 go/doc/go_spec.html
+go/go.env
 go/lib/time/mkzip.go
 go/lib/time/README
 go/lib/time/update.bash
@@ -104,11 +106,6 @@ go/misc/wasm/wasm_exec.html
 go/misc/wasm/wasm_exec.js
 go/misc/wasm/wasm_exec_node.js
 go/PATENTS
-go/pkg/distpack/go1.21.linux-amd64.tar.gz
-go/pkg/distpack/go1.21.src.tar.gz
-go/pkg/distpack/v0.0.1-go1.21.linux-amd64.info
-go/pkg/distpack/v0.0.1-go1.21.linux-amd64.mod
-go/pkg/distpack/v0.0.1-go1.21.linux-amd64.zip
 go/pkg/include/asm_amd64.h
 go/pkg/include/asm_ppc64x.h
 go/pkg/include/funcdata.h
@@ -120,8 +117,6 @@ go/pkg/tool/linux_amd64/cgo
 go/pkg/tool/linux_amd64/compile
 go/pkg/tool/linux_amd64/covdata
 go/pkg/tool/linux_amd64/cover
-go/pkg/tool/linux_amd64/dist
-go/pkg/tool/linux_amd64/distpack
 go/pkg/tool/linux_amd64/doc
 go/pkg/tool/linux_amd64/fix
 go/pkg/tool/linux_amd64/link
@@ -8025,7 +8020,11 @@ go/src/runtime/race/output_test.go
 go/src/runtime/race_ppc64le.s
 go/src/runtime/race/race_darwin_amd64.go
 go/src/runtime/race/race_darwin_arm64.go
+go/src/runtime/race/race_darwin_arm64.syso
 go/src/runtime/race/race.go
+go/src/runtime/race/race_linux_arm64.syso
+go/src/runtime/race/race_linux_ppc64le.syso
+go/src/runtime/race/race_linux_s390x.syso
 go/src/runtime/race/race_linux_test.go
 go/src/runtime/race/race_test.go
 go/src/runtime/race/race_unix_test.go

Including go/go.env and the various race syso files is likely fixing bugs!

go/codereview.cfg is benign.

The fact that the distpack archive doesn't include go/pkg/tool/linux_amd64/dist and distpack is interesting, but probably not too important, and just something where we should follow what upstream's doing.

[qmuntal]

Including go/go.env and the various race syso files is likely fixing bugs!

The missing go.env does cause an issue on our toolchain. This file contains the default values for GOPROXY, GOSUMDB, and GOTOOLCHAIN since this go1.21 commit: https://go-review.googlesource.com/c/go/+/462198. As we are currently not packing a go.env file, then GOPROXY defaults to an empty string instead of https://proxy.golang.org,direct, causing errors when trying to download not-vendored modules.

In fact, I found this issue while investigating an error in our fuzz pipeline, where an external dependency could not be downloaded due to GOPROXY not being set: https://dev.azure.com/dnceng/internal/_build/results?buildId=2154076&view=logs&j=041ba1a9-7061-5f54-2a8a-c2f725426ae0&t=1ff20deb-aa03-5320-4e9a-1703d8f592b0&l=2830.

We have to include go.env yes or yes before shipping go1.21, and we should probably also patch it so GOTOOLCHAIN don't default to auto, but to local, else the MSFT Go toolchain will silently download and use non-MSFT Go toolchains when compiling applications that require a Go version other than the one being used.