syscall: use Docker container for making auto-generated files

[bradfitz]

The auto-generation of the syscall/z*.go files is a mess.

Their generation depends on the headers installed on the host system, so nobody can reliably generate them without making a bunch of accidental unrelated changes.

This applies equally to pkg syscall and to golang.org/x/sys/unix.

Let's use a Dockerfile in there and then run mkwhatever.{sh,pl,go} file in Docker, and then docker cp the auto-generated files out of the container back to the host.

Recent chaos:

• https://go-review.googlesource.com/#/c/21971/
• https://go-review.googlesource.com/#/c/21582/
• etc (basically any change)

It's less clear what to do about other Unix, like OpenBSD (https://go-review.googlesource.com/#/c/21797/, etc). @mdempsky? Jails? But openbsd is tricky for other reasons: #15227

Windows seems to be fine.

/cc @ianlancetaylor

[mdempsky]

I don't think there's anything fundamental to generating OpenBSD's z*.go files that requires running on an OpenBSD host. It looks like we just need access to the kernel headers, which we could get by either downloading from OpenBSD's cvsweb server or grabbing an appropriate install or source tarball.

The same looks to be true of the other mksysnum_*.pl scripts.

[hirochachacha]

How do you plan to support non-amd64 architectures?

[bradfitz]

@hirochachacha, we're supposed to have builders for all Go architectures, and builders are supposed to be hermetic. So really we could/should just be using the builders to generate these. Dmitry generated the race runtime *.so files using the builders for various architectures. We use the builders to generate releases. Seems doable here too.

[hirochachacha]

@bradfitz Thanks. Are those builders are open to individual developers?
I think this is one of the head aching problem who wants to contribute to x/sys/unix.
For example, current x/sys/unix supports 12 architectures for linux.
If they want to add a new syscall, it's easy. they can use mksyscall.pl directly.
Once, they need to add a constant which is depends on kernel headers,
libc headers, per architecture, the problem is occurred.
It's not a trivial work for individual developers to prepare these environments.
That's the reason I created https://go-review.googlesource.com/#/c/20007/ before.

[hirochachacha]

(Sorry, conclusions are missing)

Although 12 (virtual) machines => 12 cross compilers will reduce some costs, it's still painful.
So, I want a promising alternative.

[bradfitz]

Are those builders are open to individual developers?

gomote access is available to those who ask and we recognize as being regular contributors.

So, I want a promising alternative.

Got a proposal?

[hirochachacha]

gomote access is available to those who ask and we recognize as being regular contributors.

I see.

Got a proposal?

Yeah, I'm convinced, thank you.

[gopherbot]

CL https://golang.org/cl/37943 mentions this issue.

[bradfitz]

/cc @josephlr who did https://golang.org/cl/37943

[josephlr]

Updated https://golang.org/cl/37943 with the basic design of how we should go about doing this in general. The steps should look something like:

1. Download necessary sources from the most recent stable version
2. Download go, all the necessary tools to build the source, the emulator, and cross compilers.
3. For each target, set GOARCH and GOOS to the right values. Then, generate the files normally, except:
   a. Use the correct cross compiler for gcc or go build
   b. Use the correct emulator when running any binaries (./myprog or go run)

Also, @bradfitz suggested docker cp to get the files out of the Docker container. This CL instead uses a shared directory (docker run -d) to both put the files in and take them out. The shared approach seemed simpler, but I'd like any feedback.

[bradfitz]

Also, @bradfitz suggested docker cp to get the files out of the Docker container. This CL instead uses a shared directory (docker run -d) to both put the files in and take them out.

The advantage of docker cp is that it is more portable. The shared directory bind mounts only work if your Docker client & server are on the same machine (and both Linux), precluding updating this from Mac/Windows running Docker there.

At least, that's how it used to be. I haven't really kept up with latest Docker. If you want to use -d, please verify it all works on macOS too.

[mdlayher]

Just gave it a run for the first time on my linux/amd64 host. It works great!! I glanced at the diff and it appears that there are quite a few constants that are currently outdated or incorrect in master for various architectures.

Well done. I'll review the code now.

[josephlr]

@bradfitz, it seems like the shared directories work on macOS and Windows, so it should be possible to now generate all the Linux go files from a macOS or Windows system. It seems like -d is at least portable enough, and it allows for hacking on the go files without any rebuilding (not that big of a deal though).

[josephlr]

Most of the changes introduced to the z*_linux*.go files will be purely additive with the following exceptions:

• I changed mkpost.go to be consistent across architectures, so now fields marked as __unused by glibc or pad fields generated by cgo -godefs are no longer exported.
• cgo -godefs has changed to no longer have zero-length fields like foo byte[0].
• Various constants that refer to the number of some quantity are incremented (e.g. AF_MAX or RTM_NR_FAMILIES).
• Some mask constants have had additional higher significance bits added (e.g. MS_RMT_MASK going from 0x800051 to 0x2800051).
• Some constants are private and shouldn't have been exposed to userspace (e.g. IFF_802_1Q_VLAN).
• Some contants were just removed by the kernel interface (e.g. PTRACE_SEIZE_DEVEL)
• Some constants either had incorrect values or never existed at all (e.g. EPOLL_NONBLOCK).

These are a lot of changes, would be easier to submit diffs on a per-architecture basis, or by the type of change (i.e. one diff fixes all the IFF_* constants, one diff removes all bad constants, etc...), or with one enormous change?

[josephlr]

Also, what should we do about sparc64? The go compiler doesn't even support that archetecture, so I don't even know how the z*_linux_sparc64.go files were even generated.

[ianlancetaylor]

Changes existing exported pad fields to no longer be exported is likely to break existing programs. While clearly those programs should not be referring directly to padding fields, in fact they do.

[ianlancetaylor]

In general while for x/sys/unix we can make things right, for syscall we need to preserve existing names even if they seem wrong to us.

[ianlancetaylor]

I think that diffs by architecture will be simplest.

I assume the sparc64 information in x/sys/unix was built using gccgo.

[josephlr]

That seems reasonable: change them in x/sys/unix but not in syscall. Would we even want this sort of process for syscall as it is "locked down"?

[bradfitz]

That seems reasonable: change them in x/sys/unix but not in syscall. Would we even want this sort of process for syscall as it is "locked down"?

It would still be nice for syscall also. We do occasionally still update it for Go's needs. By "locked down" we primarily mean that we don't add to it by user request just to add new specific symbols. But if something in the Go std library needs something, we sometimes do update it.

[josephlr]

The main issue I could see is that switching over the building of the z*_linux_*.go files will introduce a one-time large change that could potentially break users. Moving x/sys/unix to the docker build system was mostly additive, but still had deletions (see above). I guess we could try to find the versions of the Linux kernel and glibc that introduce the fewest changes, but we would probably still have some breakage. I guess it just depends if removing things like IFF_802_1Q_VLAN is a "bug fix" or a change to the syscall package interface.

[laboger]

Since this is going to regenerate all the z* files in x/sys/unix, could this include the change that was suggested by @ianlancetaylor in issue #19560 to define Signal and Errno within x/sys/unix and eliminate the need to continue to use syscall.Signal and syscall.Errno? That will allow Go users to (almost) migrate completely from syscall to x/sys/unix.

I don't understand how this proposal works for non-amd64 architectures. The values that are generated and put into the z files are pulled from the host system's header files, which aren't the same for all architectures. So if you are going to use cross compilers you are still reading the header files from the amd64 host.

[ianlancetaylor]

I think we should worry about Signal and Errno separately from this effort.

[gopherbot]

Change https://golang.org/cl/102655 mentions this issue: unix: generate linux/sparc64 go files using Docker

[bcmills]

Since syscall is no longer deprecated (#60797), it seems like maybe we should reprioritize this technical debt in the package — it makes me uncomfortable to review CLs with manual changes in files that are clearly marked DO NOT EDIT. 😅

[bcmills]

For example, it appears that a few constants were added to types_linux.go in https://go.dev/cl/407694, but as far as I can tell were not propagated to the ztypes files for arches other than loong64.