x/build/cmd/relui: ensure fully reproducible builds, including tar.gz/zip archives #24904
Assignees
Labels
Builders
x/build issues (builders, bots, dashboards)
early-in-cycle
A change that should be done early in the 3 month dev cycle.
NeedsFix
The path to resolution is known, but the work has not been done.
release-blocker
Milestone
Comments
|
@FiloSottile, were you going to do something here? |
|
I treated this as lower priority than the changes that would go in the release, but I am going to try to make the final tar.gz/zip reproducible. |
|
Are we going out of our way to set time stamps in the archives? |
|
The buildlet sends a tarball to the x/build/cmd/release client. The buildlet uses tar.FileInfoHeader. The source on the buildlet ultimately comes from git (via git archive, via the gitmirror service, via the coordinator), but when the buildlet writes the git archive to disk, it clamps the file time to system time: So it might all work for free today, assuming we have no files with future modtimes in the git repo. But it's a little fragile. It'd be nice if x/build/cmd/release enforced all the modtimes with something predictable, like using the same modtime for all files, picking the time of the git commit of the whole release. |
FiloSottile
added
the
NeedsFix
rsc
added
the
early-in-cycle
|
Change https://go.dev/cl/454836 mentions this issue: |
|
Change https://go.dev/cl/462198 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Jan 17, 2023
- Build cmd with CGO_ENABLED=0. Doing so removes the C compiler toolchain from the reproducibility perimeter and also results in cmd/go and cmd/pprof binaries that are statically linked, so that they will run on a wider variety of systems. In particular the Linux versions will run on Alpine and NixOS without needing a simulation of libc.so.6. The potential downside of disabling cgo is that cmd/go and cmd/pprof use the pure Go network resolver instead of the host resolver on Unix systems. This means they will not be able to use non-DNS resolver mechanisms that may be specified in /etc/resolv.conf, such as mDNS. Neither program seems likely to need non-DNS names like those, however. macOS and Windows systems still use the host resolver, which they access without cgo. - Build cmd with -trimpath when building a release. Doing so removes $GOPATH from the file name prefixes stored in the binary, so that the build directory does not leak into the final artifacts. - When CC and CXX are empty, do not pick values to hard-code into the source tree and binaries. Instead, emit code that makes the right decision at runtime. In addition to reproducibility, this makes cross-compiled toolchains work better. A macOS toolchain cross-compiled on Linux will now correctly look for clang, instead of looking for gcc because it was built on Linux. - Convert \ to / in file names stored in .a files. These are converted to / in the final binaries, but the hashes of the .a files affect the final build ID of the binaries. Without this change, builds of a Windows toolchain on Windows and non-Windows machines produce identical binaries except for the input hash part of the build ID. - Due to the conversion of \ to / in .a files, convert back when reading inline bodies on Windows to preserve output file names in error messages. Combined, these four changes (along with Go 1.20's removal of installed pkg/**.a files and conversion of macOS net away from cgo) make the output of make.bash fully reproducible, even when cross-compiling: a released macOS toolchain built on Linux or Windows will contain exactly the same bits as a released macOS toolchain built on macOS. The word "released" in the previous sentence is important. For the build IDs in the binaries to work out the same on both systems, a VERSION file must exist to provide a consistent compiler build ID (instead of using a content hash of the binary). For #24904. Fixes #57007. Change-Id: I665e1ef4ff207d6ff469452347dca5bfc81050e6 Reviewed-on: https://go-review.googlesource.com/c/go/+/454836 Reviewed-by: Bryan Mills <[email protected]> Run-TryBot: Russ Cox <[email protected]> Auto-Submit: Russ Cox <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
gopherbot
pushed a commit
that referenced
this issue
Jan 17, 2023
Various Linux distributions edit cmd/go/internal/cfg/cfg.go to change the default settings of GOPROXY and GOSUMDB. Make it possible for them to do this without editing the go command source code by introducing GOROOT/go.env and moving those defaults there. With the upcoming changes for reproducible builds (#24904), this should mean that Linux distributions distribute binaries that are bit-for-bit identical to the Go distribution binaries, even when rebuilding the distribution themselves. Fixes #57179. Change-Id: Ib2ecc61e6d036f97db6fd47dca757c94fdea5629 Reviewed-on: https://go-review.googlesource.com/c/go/+/462198 Auto-Submit: Russ Cox <[email protected]> Reviewed-by: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Russ Cox <[email protected]>
|
Change https://go.dev/cl/475377 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Mar 13, 2023
Fix a few lingering reproducibility problems. - Do not set CC during go install std if it is unset, so that the automatic disabling of cgo in cmd/go can run. - Since CC is not necessary, remove code insisting on it. - Use a fixed quoting algorithm instead of %q from the bootstrap toolchain, which can differ from release to release. - Remove go_bootstrap tool successfully on Windows. For #24904. Change-Id: I5c29ba6a8592e93bfab37f123b69f55c02f12ce3 Reviewed-on: https://go-review.googlesource.com/c/go/+/475377 Auto-Submit: Russ Cox <[email protected]> Run-TryBot: Russ Cox <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Bryan Mills <[email protected]>
gopherbot
pushed a commit
that referenced
this issue
Apr 11, 2023
We want to enable others to reproduce the exact distribution archives we are serving on go.dev/dl. Today the code for building those archives lives in golang.org/x/build, which is fundamentally tied to running on the Go team build infrastructure and not easy for others to run. This CL adds a new flag -distpack to cmd/dist, usually invoked as make.bash -distpack, to build the distribution archives using code in the main repository that anyone can run. Starting in Go 1.21, the Go team build infrastructure will run this instead of its current custom code to build those archives. The current builds are not reproducible even given identical infrastructure, because the archives are stamped with the current time. It is helpful to have a timestamp in the archives indicating when the code is from, but that time needs to be reproducible. To ensure this, the new -distpack flag extends the VERSION file to include a time stamp, which it uses as the modification time for all files in the archive. The new -distpack flag is implemented by a separate program, cmd/distpack, instead of being in cmd/dist, so that it can be compiled by the toolchain being distributed and not the bootstrap toolchain. Otherwise details like the exact compression algorithms might vary from one bootstrap toolchain to another and produce non-reproducible builds. So there is a new 'go tool distpack', but it's omitted from the distributions themselves, just as 'go tool dist' is. make.bash already accepts any flags for cmd/dist, including -distpack. make.bat is less sophisticated and looks for each known flag, so this CL adds an update to look for -distpack. The CL also changes make.bat to accept the idiomatic Go -flagname in addition to the non-idiomatic (for Go) --flagname. Previously it insisted on the --flag form. I have confirmed that using make.bash -distpack produces the identical distribution archives for windows/amd64, linux/amd64, darwin/amd64, and darwin/arm64 whether it is run on windows/amd64, linux/amd64, or darwin/amd64 hosts. For #24904. Change-Id: Ie6d69365ee3d7294d05b4f96ffb9159b41918074 Reviewed-on: https://go-review.googlesource.com/c/go/+/470676 TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> Run-TryBot: Russ Cox <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>
|
Starting in 1.21 we'll use the |
|
Change https://go.dev/cl/504117 mentions this issue: |
|
Change https://go.dev/cl/504116 mentions this issue: |
|
Change https://go.dev/cl/504057 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Jun 16, 2023
GOROOT/test is pruned out by cmd/distpack. It isn't really needed for the test anyway; the test can instead use the "src/unicode" subdirectory, which is even within the same module. This test was previously adjusted in CL 13467045 and CL 31859. Unlike in previous iterations of the test, the directories used in this revision are covered by the Go 1 compatibility policy and thus unlikely to disappear. For #24904. Change-Id: I156ae18354bcbc2ddd8d22b210f16ba1e97cd5d1 Reviewed-on: https://go-review.googlesource.com/c/go/+/504116 Auto-Submit: Bryan Mills <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Bryan Mills <[email protected]>
|
Change https://go.dev/cl/504059 mentions this issue: |
|
Change https://go.dev/cl/504060 mentions this issue: |
|
Change https://go.dev/cl/504058 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Jun 16, 2023
…is missing cmd/distpack removes GOROOT/test, so skip the test if it isn't there. For #24904. Change-Id: Iac381517d0540056b2ccea0dc1bd716113b18468 Reviewed-on: https://go-review.googlesource.com/c/go/+/504117 TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]> Auto-Submit: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]>
gopherbot
pushed a commit
that referenced
this issue
Jun 16, 2023
cmd/distpack removes GOROOT/test from its archive of the distribution. For #24904. Change-Id: Ifde441f048f8af52f8973555b196ab0520b48ab7 Reviewed-on: https://go-review.googlesource.com/c/go/+/504057 Auto-Submit: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>
gopherbot
pushed a commit
that referenced
this issue
Jun 16, 2023
cmd/distpack removes GOROOT/test from its distribution. For #24904. Change-Id: I6d1a8c608a1a1fe3fddfe0cd5279202ea9c2b3ce Reviewed-on: https://go-review.googlesource.com/c/go/+/504058 Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]> Auto-Submit: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
gopherbot
pushed a commit
that referenced
this issue
Jun 16, 2023
…sn't exist cmd/distpack deletes GOROOT/misc from its distribution. For #24904. Change-Id: I47c60e9a6d39d015683dde7f44bf7c34517b6a8e Reviewed-on: https://go-review.googlesource.com/c/go/+/504059 TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]> Auto-Submit: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]>
gopherbot
pushed a commit
that referenced
this issue
Jun 16, 2023
cmd/distpack deletes that module from its distribution. For #24904. Change-Id: I69dd328d0f790a49db7a053d703ae985d9ebe9e4 Reviewed-on: https://go-review.googlesource.com/c/go/+/504060 Reviewed-by: Ian Lance Taylor <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Auto-Submit: Bryan Mills <[email protected]>
|
Change https://go.dev/cl/511977 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Jul 26, 2023
The code was sorting files in the archives entirely by path string, but that's not what fs.WalkDir would do. In a directory with subdirectory foo/bar and file foo/bar.go, foo/bar gets visited first, so foo/bar/baz appears before foo/bar.go, even though "foo/bar/baz" > "foo/bar.go". This CL replaces the string comparison with a path-aware comparison that places foo/bar/baz before foo/bar.go, so that if the tar file is extracted and then repacked using fs.WalkDir, the files will remain in the same order. This will make it easier to compare the pristine distpack-produced tgz for darwin against the rebuilt tgz with signed binaries. Before: % tar tzvf /tmp/cmddist.tgz | grep -C1 runtime/cgo.go -rw-r--r-- 0 0 0 11122 Jul 13 15:00 go/src/runtime/callers_test.go -rw-r--r-- 0 0 0 2416 Jul 13 15:00 go/src/runtime/cgo.go -rw-r--r-- 0 0 0 2795 Jul 13 15:00 go/src/runtime/cgo/abi_amd64.h After: % tar tzvf pkg/distpack/go1.21rsc.src.tar.gz | grep -C1 runtime/cgo.go -rw-r--r-- 0 0 0 1848 Dec 31 1969 go/src/runtime/cgo/signal_ios_arm64.s -rw-r--r-- 0 0 0 2416 Dec 31 1969 go/src/runtime/cgo.go -rw-r--r-- 0 0 0 2479 Dec 31 1969 go/src/runtime/cgo_mmap.go For #24904. For #61513. Change-Id: Ib7374bc0d6324377f81c561bef57fd87b2111b98 Reviewed-on: https://go-review.googlesource.com/c/go/+/511977 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Heschi Kreinick <[email protected]> Run-TryBot: Russ Cox <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
|
Change https://go.dev/cl/514898 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Aug 1, 2023
macOS may write a .DS_Store file to any directory at any time in the general case, unfortunately. Expand test rules to catch them no matter where they may appear. For #24904. Change-Id: I98a2bb5d0ae2ab3a191ff87b33d6f8048ca39aa8 Reviewed-on: https://go-review.googlesource.com/c/go/+/514898 Reviewed-by: Russ Cox <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]>
|
Change https://go.dev/cl/560015 mentions this issue: |
gopherbot
pushed a commit
to golang/website
that referenced
this issue
Jan 31, 2024
As of Go 1.21, the default GOARM value when cross-compiling is 7. This was done in CL 470695. Update its documentation accordingly. For golang/go#24904. For golang/go#65241. Change-Id: I04a22b1e6d5b324615b4822f18db3c53270cf4fb Reviewed-on: https://go-review.googlesource.com/c/website/+/560015 Reviewed-by: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Go already supports reproducible builds (with no action required), but our Go releases have .tar.gz/.zip archives that have timestamps.
We should probably make cmd/release also produce reproducible output and pin the archive file entry timestamps to the git commit time of the tagged commit we're building.
/cc @FiloSottile
The text was updated successfully, but these errors were encountered: