tinygo: -compiler command line argument (e.g. tinygo)

[archie2x]

help:

• I haven't figured out how to propagate this to mkuimage.
• it may make more sense as a buildopt instead of environ in build.go(?)
  • but the BuildOpts isn't always available
• name of flag? -go-compiler(?)
• any feedback!

This allows specifying the compiler explicitly in makebb and handles peculiarities of tinygo. It works by resolving and caching the path to the go-compiler then queries tinygo for its hidden go-build-tags and propagates them to package lookup / install. This allows the pruning algorithm to correctly add / remove files when syncing dependent packages.

Usage:

% makebb -h 
Usage of makebb:
  -compiler string
    	override go compiler to use (e.g. "/path/to/tinygo")
[...]

Resulting busybox is significantly smaller. I compiled a slightly more complicated project than e.g. u-root/cmds/core/{true,false} to test package resolution.

Standard go:

u-root % GOOS=linux makebb cmds/core/init
10:10:54 Build environment: GOARCH=arm64 GOOS=linux GOPATH=/Users/roger/go CGO_ENABLED=0 GO111MODULE= GOROOT=/opt/homebrew/Cellar/go/1.22.2/libexec PATH=/opt/homebrew/Cellar/go/1.22.2/libexec/bin
10:10:54 Compiler: go version go1.22.2 darwin/arm64
10:10:59 Successfully built "bb" (size 2556039 bytes -- 2.4 MiB). <============

tinygo:

u-root % GOOS=linux makebb -compiler tinygo cmds/core/init
10:10:06 Build environment: GOARCH=arm64 GOOS=linux GOPATH=/Users/roger/go CGO_ENABLED=0 GO111MODULE= GOROOT=/opt/homebrew/Cellar/go/1.22.2/libexec PATH=/opt/homebrew/Cellar/go/1.22.2/libexec/bin
10:10:06 Compiler: tinygo version 0.33.0-dev-6184a6cd darwin/arm64 (using go version go1.22.2 and LLVM version 18.1.2)
10:10:20 Successfully built "bb" (size 1408888 bytes -- 1.3 MiB).  <============

Needs:

• tinygo dev branch (for syscall fixes supplied by @leongross )
• this patch to u-root to compile (link) with tinygo
• (rebased) Full Go "net" package port, WIP tinygo-org/tinygo#4273

diff --git a/cmds/core/init/fns_other.go b/cmds/core/init/fns_other.go
index 850ce067..f979aaea 100644
--- a/cmds/core/init/fns_other.go
+++ b/cmds/core/init/fns_other.go
@@ -2,8 +2,8 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build !tinygo && !(darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
-// +build !tinygo,!darwin,!dragonfly,!freebsd,!linux,!netbsd,!openbsd,!solaris
+//go:build !(darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
+// +build !darwin,!dragonfly,!freebsd,!linux,!netbsd,!openbsd,!solaris
 
 package main
 
diff --git a/cmds/core/init/fns_unix.go b/cmds/core/init/fns_unix.go
index 4bcac715..4db8d9d0 100644
--- a/cmds/core/init/fns_unix.go
+++ b/cmds/core/init/fns_unix.go
@@ -2,8 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build !tinygo && (darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
-// +build !tinygo
+//go:build (darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
 // +build darwin dragonfly freebsd linux netbsd openbsd solaris
 
 package main
diff --git a/cmds/core/init/init_plan9.go b/cmds/core/init/init_plan9.go
index 4fb93f94..6b88f159 100644
--- a/cmds/core/init/init_plan9.go
+++ b/cmds/core/init/init_plan9.go
@@ -2,8 +2,8 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build !tinygo && plan9
-// +build !tinygo,plan9
+//go:build plan9
+// +build plan9
 
 package main
 
diff --git a/cmds/core/init/init_tinygo.go b/cmds/core/init/init_tinygo.go
new file mode 100644
index 00000000..1d670023
--- /dev/null
+++ b/cmds/core/init/init_tinygo.go
@@ -0,0 +1,29 @@
+//go:build tinygo
+
+package main
+
+/*
+ld.lld: error: undefined symbol: syscall.runtime_BeforeExec
+>>> referenced by exec_unix.go:282 ([...]/go/1.22.2/libexec/src/syscall/exec_unix.go:282)
+>>>               /var/folders/7f/_6cnvk8j475gw2rk8q3hv8q00000gn/T/tinygo3894403787/main.lto.main.o:(runtime.run$1$gowrapper)
+
+ld.lld: error: undefined symbol: syscall.runtime_AfterExec
+>>> referenced by exec_unix.go:308 ([...]/go/1.22.2/libexec/src/syscall/exec_unix.go:308)
+>>>               /var/folders/7f/_6cnvk8j475gw2rk8q3hv8q00000gn/T/tinygo3894403787/main.lto.main.o:(runtime.run$1$gowrapper)
+failed to run tool: ld.lld
+failed to link /var/folders/7f/_6cnvk8j475gw2rk8q3hv8q00000gn/T/tinygo3894403787/main: exit status 1
+*/
+
+import (
+	_ "unsafe" // for go:linkname
+)
+
+//go:linkname runtime_BeforeExec syscall.runtime_BeforeExec
+func runtime_BeforeExec() {
+
+}
+
+//go:linkname runtime_AfterExec syscall.runtime_AfterExec
+func runtime_AfterExec() {
+
+}

[leongross]

Nice work, I anticipate this will help a lot to enable u-root builds with tinygo

[leongross]

@archie2x if you need said changes

diff --git a/cmds/core/init/fns_other.go b/cmds/core/init/fns_other.go

in u-root, do you mind opening a pr? at some point it needs to be merged anyway, right?

[archie2x]

@archie2x if you need said changes

diff --git a/cmds/core/init/fns_other.go b/cmds/core/init/fns_other.go

in u-root, do you mind opening a pr? at some point it needs to be merged anyway, right?

I was going to to, and actually, maybe a larger patch for enabling tinygo build of all cmds in u-root but I saw you already have a PR to make init in u-root/u-root#3062 or am I misunderstanding?

I think you're on a better track: putting the link fixes for tinygo in a shared package.

For the sake of managing all this, I just created a patch to tinygo to help us identify versions in the sources: tinygo-org/tinygo#4389

[leongross]

@archie2x in this project you have to sign your commits (see here). Without that, the CI is not run and with that reviewing changes might take longer.

[leongross]

@rminnich @hugelgupf what do you think of this?

[archie2x]

@rminnich @hugelgupf what do you think of this?

I marked it as draft because I want to work through the GOROOT. I think, unless overridden by env, it has to be the same as the tinygo 'cached-GOROOT'

[codecov]

Codecov Report

Attention: Patch coverage is 61.29032% with 48 lines in your changes missing coverage. Please review.

Project coverage is 61.61%. Comparing base (d8fbaca) to head (81f0c36).

Files with missing lines	Patch %	Lines
src/pkg/golang/compiler.go	60.00%	46 Missing ⚠️
src/cmd/makebb/makebb.go	75.00%	1 Missing ⚠️
src/pkg/golang/build.go	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #121      +/-   ##
==========================================
- Coverage   62.44%   61.61%   -0.84%     
==========================================
  Files          16       17       +1     
  Lines        1185     1266      +81     
==========================================
+ Hits          740      780      +40     
- Misses        445      486      +41     

Flag	Coverage Δ
1.20	81.27% <61.29%> (-2.96%)	⬇️
1.21.x	81.27% <61.29%> (-2.96%)	⬇️
1.22.x	61.61% <61.29%> (-0.84%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[leongross]

@hugelgupf I think @archie2x addressed all the requested changes, should we get this in?

[leongross]

Another thing: To enable tinygo support in u-root we had do move from go assembly to CGO. I tested this branch and realized that gobusybox does not support Cgo (also mentioned in the docs, I know). I tried to hack on this for a while but could not get it running yet.
I tried to build exp/esxiboot and got the following error:

14:27:07 Preserving bb generated source directory at /tmp/bb-2772405532 due to error: collecting and putting dependencies in place failed: writing package github.com/u-root/u-root/
pkg/boot/multiboot/internal/trampoline failed: $HOME/u-root/pkg/boot/multiboot/internal/
trampoline/trampoline_linux_amd64_tinygo.go:28:8: could not import C (no metadata for C)

I tracked the error down into go/packages/packages.go:1219, but I am not sure where in the busybox builder I can check for the "C" package and ignore it.