Fails to build on Mac M1 with docker.elastic.co/beats-dev/golang-crossbuild:1.17.6-darwin-arm64-debian10

[aleksmaus]

The elastic agent, for example, fails to build on Mac M1 with docker.elastic.co/beats-dev/golang-crossbuild:1.17.6-darwin-arm64-debian10
It seems like there is a known issue with qemu segfaulting while running x86_64 architecture on arm64.
The image is tagged as arm64 but it's actually x86_64.

Related to this discussion:
elastic/beats#29585 (comment)

[aleksmaus]

@v1v assigned to you for now, but feel free to reassign or close if there is an explanation for this

[v1v]

Thanks @aleksmaus , can you point to the build failure and URL? (if it's a public accessible CI controller).

I'll look at how those docker images are created and see if a similar docke rimage is override for some unexpected reason when running in x86

[v1v]

Maybe related to -> #76

[v1v]

As far as I see, docker.elastic.co/beats-dev/golang-crossbuild:1.17.6-darwin-arm64-debian10 is created only when running on a ubuntu x86_64 workers, the debian-10-arm combination is not enabled as defined in #76

1. I'll add support for debian-10
2. I'll look at what's cuasing a tagging in a different architecture, since it does smell weird

[v1v]

#100 is the one that enabled docker.elastic.co/beats-dev/golang-crossbuild:1.17.6-darwin-arm64-debian10 the suffix arm64 is regarding the docker base image -> https://github.com/elastic/golang-crossbuild/tree/main/go1.17/darwin-arm64

So I'm enabling support for arm and I think it should add an extra suffix regarding the arch.

IMO, there is no an issue with the docker image itself. Docker images for darwin-arm64 on ARM64 were not enabled.

Let's see if #157 solves the issue 🤞

[aleksmaus]

Thanks @aleksmaus , can you point to the build failure and URL? (if it's a public accessible CI controller).

I'll look at how those docker images are created and see if a similar docke rimage is override for some unexpected reason when running in x86

haven't tried with CI, segfaults locally building on M1 as mentioned in: elastic/beats#29585
which renders new M1 macbooks not usable for beats/agent development.

[v1v]

The existing darwin-arm64 docker images were built to be supported on intel only and qemu the way to use them. Since qemu has got a bug or dependency with the upstream, then this is a bit more complex now.

We are trying to see if elastic/observability-dev#157 can help with, but I cannot warranty this will work. If not, then I'll let you know and we can then figure out what to do next, as we might need some help from you or your team.

[v1v]

I'm afraid I've found a blocker regarding building clang for m1 arch:

• ci: enable arm for (Makefile.debian10, Makefile) and enable darwin-arm64 for arm64 #157 (comment)

I've tried to solve the issue as suggested in some place with apple-m1 and bump the clang/llvm version from 11 to 13 but it didn't work, see #157 (comment)

Who is more knowledge about those toolchains?

[cmacknz]

I had a look at this since I both have an M1 Mac and have dealt with cross-compilation in the past.

Problem 1

For the cc1: error: unknown value 'apple-m1' for -mcpu from #157 (comment) what is happening is that CFLAGS=-mcpu=apple-m1 is going to the host Debian Linux CC (probably GCC) that is used to build clang from source, and the Debian CC doesn't support that option. Specifying -mcpu in that way isn't correct. Just upgrading clang to 13.0.0 will give the resulting osxcross toolchain the ability to use the -mcpu=apple-m1 flag, we don't need to build clang itself with that flag.

If you remove the -mcpu option we go back to the original error:

[2022-02-18T00:31:08.900Z]     /tmp/osxcross/build/llvm-10.0.1.src/build_stage1/bin/clang   -march=native    -o CMakeFiles/cmTC_c5c1e.dir/testCCompiler.c.o   -c /tmp/osxcross/build/llvm-10.0.1.src/build_stage2/CMakeFiles/CMakeTmp/testCCompiler.c
[2022-02-18T00:31:08.900Z]     clang-10: error: the clang compiler does not support '-march=native'

That problem is caused because when running on an M1 Mac clang doesn't actually support the -march=native flag and the Clang/LLVM build here hard codes it into their CMake build scripts. There is an LLVM discuss thread explaining where this problem comes from which links to a patch to fix it.

@v1v are you trying to run this locally on an M1 Mac? I can reproduce it on my own M1 machine but I wonder if this would go away if trying to build on an X86 machine (like the Jenkins workers) since -march=native would work there.

Problem 2

I am not sure that just upgrading clang to 13.0.0 will completely solve this problem. I think we need to produce an arm variant of the actual crossbuild Docker container so that docker doesn't have to emulate it via qemu when run on Apple silicon:

Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.17.6-darwin-arm64-debian10
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

This warning is tell us the the actual Docker image golang-crossbuild:1.17.6-darwin-arm64-debian1 is for X86 and not ARM. This is the target architecture of the container itself, not the build tools inside it. These are two separate issues. The docker build tooling itself should support producing an ARM variant of the crossbuild container. Some examples of how to do this are:

1. https://docs.docker.com/desktop/multi-arch/#build-and-run-multi-architecture-images
2. https://www.docker.com/blog/multi-arch-images/

Hopefully this helps!

[v1v]

Thanks @cmacknz

Problem 1

@v1v are you trying to run this locally on an M1 Mac? I can reproduce it on my own M1 machine but I wonder if this would go away if trying to build on an X86 machine (like the Jenkins workers) since -march=native would work there.

I don't use my local development as I don't have a M1 Mac, in addition the https://github.com/elastic/golang-crossbuild relies on x86 and aarch64 to create those docker images.

#157 (comment) happened in aarch64 build

Problem 2

This warning is tell us the the actual Docker image golang-crossbuild:1.17.6-darwin-arm64-debian1 is for X86 and not ARM.

Somehow I though, wrongly, that we had to use the existing platform, so let me dig into the buildx and see if it's possible to use it (I recall I asked for the support for buildx in our existing CI workers long time ago) I only need to figure out what bits and pieces need to be changed to support this with the existing Makefile/Pipelines

[v1v]

Unfortunately the docker buildx is not working straight away:

• build-and-publish multi-architecture docker images #160 failed with

[2022-03-03T11:18:01.700Z] elastic/obs-dc-team#102 0.099 .buildkit_qemu_emulator: /bin/sh: Invalid ELF image for this architecture

Therefore, it might require further debugging/analysis. I don't know if the original contributors can provide further insights for this project and whether buildx was somehow considered initially?

[cmacknz]

There is some additional documentation here: https://github.com/docker/buildx#building-multi-platform-images

It looks like buildx has a few ways to support multi-arch builds, and the most common one uses QEMU which might need some additional installation steps. That documentation suggests that running docker run --privileged --rm tonistiigi/binfmt --install all might help.

There is at least one random blog post confirming that as a fix for this specific issue: https://vikaspogu.dev/posts/docker-buildx-setup/

[eyalkraft]

Almost all of us at the @elastic/cloud-security-posture team have M1 macbooks and we are currently forced to build the agent on remote machines which is less than ideal.
Building the agent image locally is something we need to develop and test e2e features we are working on.
Are there any expectations for when will this be resolved? How can we help or influence the prioritization of this issue?
@tehilashn @amirbenun @DaveSys911

[cmacknz]

CC-ing more of the agent and robots teams for visibility: @cachedout @KseniaElastic @jlind23 @ph

You can't package the agent locally on an M1 Mac which is big productivity issue for those who are affected (the whole Cloud Posture Security team for example). There was some initial investigation to try to make it work, but it looks like it will need a larger time investment to fix.

It's not clear who should own fixing this right now.

[jlind23]

@cachedout we (the elastic agent team) are happy to help if we can but as far as I remember you (robots) were the one owning the golang-crossbuild repo so we may need some help from your end!

[amitkanfer]

thank you @jlind23 @cmacknz for your comments and will to assist.
Who should we be in touch with to track the fix for this issue? Who should we talk to in order to prioritize this?

[cachedout]

@jlind23 and @cmacknz This one is a bit tricky in terms of ownership. If I'm understanding this issue correctly, the build doesn't even work locally on M1 machines, irrespective of anything having to do with goland-crossbuild.

I don't mean to play hot potato with this one, because I know this needs to get fixed but IMHO this is something that first needs to be resolved by the product team before we can even think about cross-compilation. Definitely happy to jump on a call to discuss because we need to find a solution for these folks. LMK.

[jlind23]

@AndersonQ as you were the one working on the M1 stuff for the agents. What is missing to make the build work locally?

[AndersonQ]

Hey folks,

Let me try to split the problem into parts.

1. It's possible to compile (a.k.a build) the elastic-agent on a M1 chip. As long as the Go command is invoked naively, on the machine. Both go build and mage build work.

2. Running the golang-crossbuild images isn't possible due to a bug on qemu, a dependency of Docker for Mac. There is a few issues on docker/for-mac repo, see below. However none of them provide a final solution. And right now I cannot try this or I found a solution. I haven't put much time into researching solutions though.

   • qemu: uncaught target signal 11 (Segmentation fault) - core dumped when running docker-compose up on Apple Silicon docker/for-mac#5123
   • Docker preview version for apple silicon is fail to run 'apache/airflow' image docker/for-mac#5232

A few possible solutions directly on the agent side of things:

• (quick and easy): document how to manually build the agent and replace the agent on a downloaded package by the manually built.
  This would allow to modify and compile the agent on M1 chips, but adds the burden of replacing the binary manuall.
• Make it possible to package the agent without docker:
  • PH plans to work on simplifying the agent's build process on our ON Week. It might, or may not solve the problem.
  • We might make it a task to only use docker when either necessary, on CI or as a opt-in/out
• Make a golang-crossbuild image using another base image, one that works on M1
  From what I read it seems that docker-for-mac can run other platforms, so we could try make a golang-crossbuild image using one of the platform that work well on M1 chips.

I'd go for the quick and easy for now, wait to see what comes out of the On Week, and then reassess the issue.

[DaveSys911]

@AndersonQ Great news.
Please note some integrations are k8s only which forces us to have a docker build to verify our changes on a local k8s setup/debug.

[cmacknz]

To follow up on Anderson's comments, we can avoid the QEMU issue by providing an arm64 version of the golang-crossbuild image per docker/for-mac#5123 (comment). Currently it is x86, which requires use of QEMU to run the container on M1 machines, which doesn't appear to work. The containers themselves have a target architecture much like a regular binary.

We could alternatively stop using golang-crossbuild entirely as part of regular development workflows, but we would need some other solution for reliably cross-compiling when using CGO.

[kuisathaverat]

Running the golang-crossbuild images isn't possible due to a bug on qemu, a dependency of Docker for Mac. There is a few issues on docker/for-mac repo, see below. However none of them provide a final solution. And right now I cannot try docker/for-mac#5232 (comment) or I found a solution. I haven't put much time into researching solutions though.

Which golang-cross build Docker images would be needed with support for running on M1?
Which exact command not build on M1?
Would be enough to build linux/x86_64 binaries on M1?

[cmacknz]

The mage package target of the elastic agent fails on an M1 mac. The key here is that building just the agent binary itself often isn't that useful, you need the package that bundles in the beats it can start and supervise (this is the package target).

It may be that mage package is not selective enough it what it builds. Most of the time all you would want for local development is the M1/arm64 binaries for the agent and all the beats it can start. You shouldn't need to cross-compile anything.

For reference here is what I get when running mage package on the agent's main branch:

 mage package
Package 'filebeat-8.3.0-darwin-x86_64.tar.gz' does not exist on path: /Users/cmackenzie/go/src/github.com/elastic/beats/x-pack/filebeat/build/distributions/filebeat-8.3.0-darwin-x86_64.tar.gz
Generated fields.yml for filebeat to /Users/cmackenzie/go/src/github.com/elastic/beats/x-pack/filebeat/fields.yml
>> Building filebeat.yml for linux/amd64
>> Building filebeat.reference.yml for linux/amd64
>> Building filebeat.docker.yml for linux/amd64
exec: go "list" "-m"
>> golangCrossBuild: Building for darwin/arm64
Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.17.9-darwin-arm64-debian10
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[CC=oa64-clang, CXX=oa64-clang++, GOARCH=arm64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-arm64]
# runtime/cgo
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Error: running "go build -o build/golang-crossbuild/filebeat-darwin-arm64 -buildmode pie -trimpath -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2022-04-25T18:43:58Z -X github.com/elastic/beats/v7/libbeat/version.commit=fabd9f7ea8132e8ba4cf02081aa6138dc5b7982b" failed with exit code 2
Error: failed building for darwin/arm64: exit status 2
failed building for darwin/arm64: exit status 2
package ran for 3m46.202571958s
Error: running "docker run --env EXEC_UID=501 --env EXEC_GID=20 -v /Users/cmackenzie/go/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=false --env DEV=false -v /Users/cmackenzie/go/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/x-pack/filebeat docker.elastic.co/beats-dev/golang-crossbuild:1.17.9-darwin-arm64-debian10 --build-cmd build/mage-linux-arm64 golangCrossBuild --platforms darwin/arm64" failed with exit code 1
package ran for 3m51.39017225s
Error: exit status 1

It starts a Linux container and tries to produce Darwin ARM64 images from it on a native M1 machine which is obviously not ideal.

[kuisathaverat]

make package builds a bunch of binaries, we can control the binaries generated by using the PLATFORMS environment variable. Do we really build all those binaries locally? Do make sense to cross-compile Darwin ARM64 binaries in an M1? it is the native architecture you do not need to cross-compile mage build will generate the same binary. Let's narrow down the number of architectures we need to cross-compile in M1, Which architectures do we need to cross-compile in an M1?

[AndersonQ]

@kuisathaverat the problem isn't the binaries built, it's the docker image used to build them. The docker image does not work on the M1 macs.

As you can see on the docker-for-mac issue it seems to be possible to use a arm64 containers on M1 docker container.

Do you know how to build such a image? I had a quick try yesterday, testing to chhttps://github.com/elastic/golang-crossbuild/tree/main/go/base-armange the base image of darwin-arm64 to base-arm. However it did not work and I did not have time to investigate why.

For the record, this was my change and I run make build-arm inside go/darwin-arm64.
And I got the following error:

Step 4/17 : FROM ${REPOSITORY}/golang-crossbuild:${VERSION}-base-arm${TAG_EXTENSION}
manifest for docker.elastic.co/beats-dev/golang-crossbuild:1.18.1-base-arm not found: manifest unknown: manifest unknown
make: *** [../Makefile.common:30: build-arm] Error 1

I'm not sure is there isn't a image published as ...-base-arm or what :/
As I said, I did not have time to investigate further

[cmacknz]

The instructions to build a multi-CPU image are here: https://docs.docker.com/desktop/multi-arch/#build-and-run-multi-architecture-images

Depending on what goes into the image it may not be enough to simply change the target architecture of the docker build command. Anything you build as part of the image would also need to be updated to produce ARM artifacts, particularly if you building and linking in C libraries with clang/gcc for use with CGO.

I would say there are two problems:

1. The golang-crossbuild image cannot cross-compile from Linux X86 to Darwin ARM when run on a Darwin ARM host.
2. At a higher level, you do not need to cross-compile from Linux X86 to Darwin ARM when run on a Darwin ARM host. We could address this in a few ways:
   a) Produce an ARM variant of the golang-crossbuild image and only cross-compile when it makes sense (e.g. in this mode you would need to cross-compile to produce X86 artifacts).
   b) Create a mage target that only produces a packaged elastic agent for the current host architecture without cross-compiling anything. Right now mage package is more like mage release where it always builds all the release artifacts for every supported platform. For development, we just need a target that produces a packaged/bundled agent for the host architecture.

[AndersonQ]

a) Produce an ARM variant of the golang-crossbuild image and only cross-compile when it makes sense

The only detail here is that there might be some decencies needed besides Go. Filebeat needs some package as it uses CGO, to produce a M1 universal binary it's required lipo. These are things where a docker image helps even if it isn't a crossbuild.

Anyway I think we should manage to be able to build without docker. Also, for the agent specifically, as log as it downloads the beats, it should work just fine locally as long as it isn't a universal binary. I don't recall any other dependency we have.

[kuisathaverat]

@kuisathaverat the problem isn't the binaries built, it's the docker image used to build them. The docker image does not work on the M1 macs.

we can get rid of use the Docker image if it is not needed, in the case of build macOS binaries in an M1 it is not needed. It is easy than maintaining an M1 Docker image to (no-) cross-compile, we only need to add few lines to the package Go file.

Do you know how to build such a image? I had a quick try yesterday, testing to chhttps://github.com/elastic/golang-crossbuild/tree/main/go/base-armange the base image of darwin-arm64 to base-arm. However it did not work and I did not have time to investigate why.

buildx and a considerable amount of time to configure the proper (no-) cross-compile environment

The instructions to build a multi-CPU image are here: https://docs.docker.com/desktop/multi-arch/#build-and-run-multi-architecture-images

well, it is not that simple, the article is for build a Docker image for two or more architectures, it is and it is not our case, it is our case because we have to build a Docker image for two architectures, and it is not because in our case the Docker files for both architectures are radically different. The main packages are common, but the build toolchain is for different architectures, and here is where the work starts, to compile linux/X binaries is more or less easy, but for Darwin architecture, we need to LLVM and osxcross compiled for linux/arm64 that's is not the common platform to build cross-compilers (usually x86_64) and use to not be straight forward.

b) Create a mage target that only produces a packaged elastic agent for the current host architecture without cross-compiling anything. Right now mage package is more like mage release where it always builds all the release artifacts for every supported platform. For development, we just need a target that produces a packaged/bundled agent for the host architecture.

This is exactly my point. Have tools to cross-compile things that will not be used (or rarely used) for a platform that can produce those binaries directly.

[kuisathaverat]

I've made some tests and I found the tool we use to compile Darwin on Linux does not compile on arm64, it fails due to some link issues, there is a reported issue pretty similar. I have tested other images and not all crossbuild-essential-* and linux-libc-dev-* are available for arm64. I am testing a patch to compile osxcross, if that works, I'd probably compile the Darwin Docker image for amd64/arm64

[kuisathaverat]

I have found a Docker image that compiles the osxcross for darwin/arm64 but is not exactly the same distribution we use, so I am working to see what this image makes different that makes the compilation works.

[kuisathaverat]

I cannot test the Docker image on a M1 but it is a arm64 Docker image so it should compile darwin ona M1

> docker run -it --entrypoint /bin/bash docker.elastic.co/observability-ci/beats-dev/golang-crossbuild:1.18.1-darwin-m1 
Status: Downloaded newer image for docker.elastic.co/observability-ci/beats-dev/golang-crossbuild:1.18.1-darwin-m1
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
root@80a068a28eea:/app# uname -a
Linux 80a068a28eea 5.10.104-linuxkit elastic/observability-dev#1 SMP Wed Mar 9 19:05:23 UTC 2022 aarch64 GNU/Linux
root@80a068a28eea:/app# 

Could someone confirm it?

# checkout 
git clone [email protected]:elastic/beats.git
cd beats

# x86_64 binary
docker run -it --rm -e CGO_ENABLED=0 -v $(pwd):/beats  -w "/beats" --entrypoint /bin/bash  "docker.elastic.co/observability-ci/beats-dev/golang-crossbuild:1.18.1-darwin-m1" -c "go install github.com/magefile/mage && cd filebeat && CC=o64-clang CXX=o64-clang++ GOARCH=amd64 GOOS=darwin mage build && mv filebeat filebeat_x86_64"

# arm64 binary
docker run -it --rm -e CGO_ENABLED=0 -v $(pwd):/beats -w "/beats" --entrypoint /bin/bash  "docker.elastic.co/observability-ci/beats-dev/golang-crossbuild:1.18.1-darwin-m1" -c "go install github.com/magefile/mage && cd filebeat && CC=oa64-clang CXX=oa64-clang++ GOARCH=arm64 GOOS=darwin mage build && mv filebeat filebeat_arm64"

# Universal binary
docker run -it --rm -v $(pwd):/beats  -w "/beats" --entrypoint /bin/bash  "docker.elastic.co/observability-ci/beats-dev/golang-crossbuild:1.18.1-darwin-m1" -c  "cd filebeat && lipo -create -output filebeat_universal filebeat_x86_64 filebeat_arm64" 

[cmacknz]

@kuisathaverat I just tried it on my M1 machine and it worked. I can run all the binaries produced.

~/go/src/github.com/elastic/beats/filebeat main ?4 ······································ 10:28:10 AM
❯ file filebeat_x86_64
filebeat_x86_64: Mach-O 64-bit executable x86_64
~/go/src/github.com/elastic/beats/filebeat main ?4 ······································ 10:28:16 AM
❯ file filebeat_arm64
filebeat_arm64: Mach-O 64-bit executable arm64
~/go/src/github.com/elastic/beats/filebeat main ?4 ······································ 10:28:21 AM
❯ file filebeat_universal
filebeat_universal: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64Mach-O 64-bit executable x86_64] [arm64]
filebeat_universal (for architecture x86_64):   Mach-O 64-bit executable x86_64
filebeat_universal (for architecture arm64):    Mach-O 64-bit executable arm64

[kuisathaverat]

I have made the changes to have a chain of Docker multiarch Docker images, we will try to make multiarch as many as possible of out Docker images, but all are not possible and some will be not completely functional. We will start with the base, main, arm64, llvm, and darwin Docker images the rest probably is not possible or require a considerable amount of work/maintainance.

[kuisathaverat]

the Linux/arm Docker image for amd64 and arm64 was published, I have some issues building LLVM for both architectures it takes more than 24h in the CI and the worker fails, I will try to launch a bigger worker to see if it reduces the build time, an alternative is to not use the LLVM version of apple in the multi-arch Docker image, it is not a big deal we do not use the support for arm64e yet.

[kuisathaverat]

I've tested with bigger machines and does not help much, the ARM compilation is slow, I just launch a test using qemu-user-static instead of docker/binfmt, qemu-user-static is more updated and support more architectures.

[kuisathaverat]

The images for Debian 10 and 11 will be available in about 45 min, the final solution was to compile the compiles in the native architecture that stick the time to 3.5h to compile the LLVM and the osxcross, this process is made only when we add a new Debian version so it is not a big deal.

We publish the following Docker images for amd64 and arm64 architectures:

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-llvm-apple-debian10

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-base-debian10

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-arm-debian10

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-darwin-debian10

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-darwinr-arm64-debian10

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-llvm-apple-debian11

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-base-debian11

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-arm-debian11

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-darwin-debian11

• docker.elastic.co/beats-dev/golang-crossbuild:1.18.2-darwinr-arm64-debian11

[kuisathaverat]

I have realized that beat is still on Go 1.17, so I am backporting to the 1.17 branch, it seems feasible.

[kuisathaverat]

The images for Go 1.17 are published, I have tested the arm for packaging at elastic/beats#31779, Could anyone with an M1 test that can cross-compile darwin binaries without issues?

[amitkanfer]

@eyalkraft / @amirbenun ?

[DaveSys911]

@kuisathaverat Metricbeat successfully built on M1 by checking out elastic/beats#31779 you can see the mage output below.

Also seems to work for elastic-agent with minimal changes, I opened a pr for it elastic/elastic-agent#500. Should be merged after pr #31779. Would be great if you could review it.

↳ DEV=TRUE PLATFORMS="linux/arm64" TYPES=tar.gz  mage package                                                                                  23:39  02.06.22  kops-csp-demo-2.k8s.local/cloudbeat ⎈  ✔  6349
Generated fields.yml for metricbeat to /Users/daveops/git_repos/elastic/beats/metricbeat/fields.yml
Generated fields.yml for metricbeat to /Users/daveops/git_repos/elastic/beats/metricbeat/fields.yml
>> Building metricbeat.yml for linux/amd64
>> Building metricbeat.reference.yml for linux/amd64
>> Building metricbeat.docker.yml for linux/amd64
Generated fields.yml for metricbeat to /Users/daveops/git_repos/elastic/beats/metricbeat/build/fields/fields.all.yml
go: downloading github.com/jmoiron/sqlx v1.3.1
go: downloading github.com/denisenkom/go-mssqldb v0.9.0
go: downloading github.com/Masterminds/semver v1.5.0
>> buildGoDaemon: Building for linux/arm64
>> golangCrossBuild: Building for linux/arm64
Unable to find image 'docker.elastic.co/beats-dev/golang-crossbuild:1.17.10-arm' locally
Unable to find image 'docker.elastic.co/beats-dev/golang-crossbuild:1.17.10-arm' locally
1.17.10-arm: Pulling from beats-dev/golang-crossbuild
1.17.10-arm: Pulling from beats-dev/golang-crossbuild
de7fc2a3b80b: Pulling fs layer
7a29e1a4599e: Pulling fs layer
8a91a40bb4be: Pulling fs layer
9aaef5b0ba6b: Pulling fs layer
e363397906b7: Pulling fs layer
6a7239b0c233: Pulling fs layer
4f4fb700ef54: Pulling fs layer
ca47241150b4: Pulling fs layer
3cbd8d239166: Pulling fs layer
af1ec725c57c: Pulling fs layer
27e85cc2d7a2: Pulling fs layer
aa3e757025fc: Pulling fs layer
f4622acbe525: Pulling fs layer
dd3c03d25754: Pulling fs layer
252925a38eb3: Pulling fs layer
28f377a51d92: Pulling fs layer
3c462d796606: Pulling fs layer
bff3bc20a08c: Pulling fs layer
9aaef5b0ba6b: Waiting
e363397906b7: Waiting
6a7239b0c233: Waiting
4f4fb700ef54: Waiting
ca47241150b4: Waiting
3cbd8d239166: Waiting
af1ec725c57c: Waiting
27e85cc2d7a2: Waiting
aa3e757025fc: Waiting
f4622acbe525: Waiting
dd3c03d25754: Waiting
252925a38eb3: Waiting
28f377a51d92: Waiting
3c462d796606: Waiting
bff3bc20a08c: Waiting
de7fc2a3b80b: Pulling fs layer
7a29e1a4599e: Pulling fs layer
8a91a40bb4be: Pulling fs layer
9aaef5b0ba6b: Pulling fs layer
e363397906b7: Pulling fs layer
6a7239b0c233: Pulling fs layer
4f4fb700ef54: Pulling fs layer
ca47241150b4: Pulling fs layer
3cbd8d239166: Pulling fs layer
af1ec725c57c: Pulling fs layer
27e85cc2d7a2: Pulling fs layer
aa3e757025fc: Pulling fs layer
f4622acbe525: Pulling fs layer
dd3c03d25754: Pulling fs layer
252925a38eb3: Pulling fs layer
28f377a51d92: Pulling fs layer
3c462d796606: Pulling fs layer
bff3bc20a08c: Pulling fs layer
6a7239b0c233: Waiting
bff3bc20a08c: Waiting
4f4fb700ef54: Waiting
ca47241150b4: Waiting
3cbd8d239166: Waiting
af1ec725c57c: Waiting
27e85cc2d7a2: Waiting
aa3e757025fc: Waiting
f4622acbe525: Waiting
dd3c03d25754: Waiting
252925a38eb3: Waiting
28f377a51d92: Waiting
3c462d796606: Waiting
9aaef5b0ba6b: Waiting
e363397906b7: Waiting
7a29e1a4599e: Download complete
7a29e1a4599e: Verifying Checksum
7a29e1a4599e: Download complete
9aaef5b0ba6b: Verifying Checksum
9aaef5b0ba6b: Download complete
9aaef5b0ba6b: Verifying Checksum
9aaef5b0ba6b: Download complete
de7fc2a3b80b: Verifying Checksum
de7fc2a3b80b: Download complete
de7fc2a3b80b: Verifying Checksum
de7fc2a3b80b: Download complete
de7fc2a3b80b: Pull complete
de7fc2a3b80b: Pull complete
7a29e1a4599e: Pull complete
7a29e1a4599e: Pull complete
8a91a40bb4be: Download complete
8a91a40bb4be: Verifying Checksum
8a91a40bb4be: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
4f4fb700ef54: Download complete
6a7239b0c233: Verifying Checksum
6a7239b0c233: Download complete
6a7239b0c233: Verifying Checksum
6a7239b0c233: Download complete
8a91a40bb4be: Pull complete
8a91a40bb4be: Pull complete
9aaef5b0ba6b: Pull complete
9aaef5b0ba6b: Pull complete
e363397906b7: Verifying Checksum
e363397906b7: Download complete
e363397906b7: Verifying Checksum
e363397906b7: Download complete
af1ec725c57c: Download complete
af1ec725c57c: Download complete
ca47241150b4: ca47241150b4: Download complete
Download complete
3cbd8d239166: Verifying Checksum
3cbd8d239166: Download complete
3cbd8d239166: Verifying Checksum
3cbd8d239166: Download complete
e363397906b7: Pull complete
e363397906b7: Pull complete
6a7239b0c233: Pull complete
6a7239b0c233: Pull complete
4f4fb700ef54: Pull complete
4f4fb700ef54: Pull complete
ca47241150b4: Pull complete
ca47241150b4: Pull complete
3cbd8d239166: Pull complete
3cbd8d239166: Pull complete
af1ec725c57c: Pull complete
af1ec725c57c: Pull complete
27e85cc2d7a2: Verifying Checksum
27e85cc2d7a2: Download complete
27e85cc2d7a2: Verifying Checksum
27e85cc2d7a2: Download complete
27e85cc2d7a2: Pull complete
27e85cc2d7a2: Pull complete
aa3e757025fc: Verifying Checksum
aa3e757025fc: Verifying Checksum
aa3e757025fc: Download complete
aa3e757025fc: Download complete
aa3e757025fc: Pull complete
aa3e757025fc: Pull complete
dd3c03d25754: Verifying Checksum
dd3c03d25754: Download complete
dd3c03d25754: Verifying Checksum
dd3c03d25754: Download complete
f4622acbe525: Download complete
f4622acbe525: Verifying Checksum
252925a38eb3: Verifying Checksum
252925a38eb3: Download complete
252925a38eb3: Verifying Checksum
252925a38eb3: Download complete
f4622acbe525: Pull complete
f4622acbe525: Pull complete
3c462d796606: Verifying Checksum
3c462d796606: Download complete
3c462d796606: Verifying Checksum
3c462d796606: Download complete
dd3c03d25754: Pull complete
dd3c03d25754: Pull complete
252925a38eb3: Pull complete
252925a38eb3: Pull complete
28f377a51d92: Verifying Checksum
28f377a51d92: Download complete
28f377a51d92: Download complete
28f377a51d92: Pull complete
28f377a51d92: Pull complete
3c462d796606: Pull complete
3c462d796606: Pull complete
bff3bc20a08c: bff3bc20a08c: Verifying Checksum
Verifying Checksum
bff3bc20a08c: Download complete
bff3bc20a08c: Download complete
bff3bc20a08c: Pull complete
bff3bc20a08c: Pull complete
Digest: sha256:85a67f7df6065eb6352c86c2d9f1472b1f22143e9d4c747e666ee4c60f1ffd11
Digest: sha256:85a67f7df6065eb6352c86c2d9f1472b1f22143e9d4c747e666ee4c60f1ffd11
Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.17.10-arm
Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.17.10-arm
>> Building using: cmd='build/mage-linux-arm64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]
/tmp/cccbYKnV.o: In function `main':
god.c:(.text+0x2ac): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x230): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
>> package: Building metricbeat-oss type=tar.gz for platform=linux/arm64
>> package: Building metricbeat-oss type=docker for platform=linux/arm64
>> package: Building metricbeat-oss type=deb for platform=linux/arm64
>> package: Building metricbeat-oss type=rpm for platform=linux/arm64
[+] Building 3.5s (2/3)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
[+] Building 7.0s (4/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                  3.8s
 => [internal] load build context                                                                                                                                                                                3.0s
 => => transferring context: 98.63MB                                                                                                                                                                             3.0s
[+] Building 7.4s (4/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                  3.8s
 => [internal] load build context                                                                                                                                                                                3.4s
 => => transferring context: 108.66MB                                                                                                                                                                            3.4s
[+] Building 8.7s (5/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
[+] Building 11.8s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
[+] Building 13.2s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
[+] Building 14.1s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
[+] Building 15.7s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
[+] Building 21.9s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
[+] Building 25.1s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
[+] Building 25.2s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
[+] Building 25.5s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                  3.8s
 => [internal] load build context                                                                                                                                                                                4.6s
 => => transferring context: 150.75MB                                                                                                                                                                            4.6s
[+] Building 25.8s (7/18)
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
[+] Building 38.2s (19/19) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 3.86kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                  3.8s
 => [internal] load build context                                                                                                                                                                                4.6s
 => => transferring context: 150.75MB                                                                                                                                                                            4.6s
 => [stage-1  1/12] FROM docker.io/library/ubuntu:20.04@sha256:47f14534bda344d9fe6ffd6effb95eefe579f4be0d508b7445cf77f61a0e5724                                                                                  0.0s
 => => resolve docker.io/library/ubuntu:20.04@sha256:47f14534bda344d9fe6ffd6effb95eefe579f4be0d508b7445cf77f61a0e5724                                                                                            0.0s
 => => sha256:47f14534bda344d9fe6ffd6effb95eefe579f4be0d508b7445cf77f61a0e5724 1.42kB / 1.42kB                                                                                                                   0.0s
 => => sha256:ca83774d06420ceb4682ef73bd9cbbfc38a97a27e061b578547a6761206658b9 529B / 529B                                                                                                                       0.0s
 => => sha256:db1bc6aa58da386bc4ae8c4489e0bda0f6073c49a7b6c2ca24386671b36a1f19 1.48kB / 1.48kB                                                                                                                   0.0s
 => [stage-1  2/12] RUN for iter in {1..10}; do         apt-get update -y &&         DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-uti  30.9s
 => [home 2/3] COPY beat /usr/share/metricbeat                                                                                                                                                                   0.4s
 => [home 3/3] RUN mkdir -p /usr/share/metricbeat/data /usr/share/metricbeat/logs &&     chown -R root:root /usr/share/metricbeat &&     find /usr/share/metricbeat -type d -exec chmod 0755 {} ; &&     find /  1.5s
 => [stage-1  3/12] RUN set -e ;   TINI_BIN="";   TINI_SHA256="";   TINI_VERSION="v0.19.0";   echo "The arch value is $(arch)";   case "$(arch)" in     x86_64)         TINI_BIN="tini-amd64";         TINI_SHA  1.8s
 => [stage-1  4/12] COPY docker-entrypoint /usr/local/bin/docker-entrypoint                                                                                                                                      0.0s
 => [stage-1  5/12] RUN chmod 755 /usr/local/bin/docker-entrypoint                                                                                                                                               0.2s
 => [stage-1  6/12] COPY --from=home /usr/share/metricbeat /usr/share/metricbeat                                                                                                                                 0.2s
 => [stage-1  7/12] RUN mkdir /licenses                                                                                                                                                                          0.2s
 => [stage-1  8/12] COPY --from=home /usr/share/metricbeat/LICENSE.txt /licenses                                                                                                                                 0.0s
 => [stage-1  9/12] COPY --from=home /usr/share/metricbeat/NOTICE.txt /licenses                                                                                                                                  0.0s
 => [stage-1 10/12] RUN groupadd --gid 1000 metricbeat                                                                                                                                                           0.2s
 => [stage-1 11/12] RUN useradd -M --uid 1000 --gid 1000 --groups 0 --home /usr/share/metricbeat metricbeat                                                                                                      0.3s
 => [stage-1 12/12] WORKDIR /usr/share/metricbeat                                                                                                                                                                0.0s
 => exporting to image                                                                                                                                                                                           0.3s
 => => exporting layers                                                                                                                                                                                          0.3s
 => => writing image sha256:1e9d96383d8affcbf02ed73d5856ba6e59aa5a942a2edac53022d527197ef356                                                                                                                     0.0s
 => => naming to docker.elastic.co/beats/metricbeat-oss:8.4.0                                                                                                                                                    0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
>> Testing package contents
package ran for 5m50.862501375s

And this is the elastic-agent image

[
    {
        "Id": "sha256:ca9451c7065756a57bb6092fe8c2e24672d123472d736bf1e3562c81aa4998bd",
        "RepoTags": [
            "docker.elastic.co/beats/elastic-agent:8.4.0-SNAPSHOT"
        ],
        "RepoDigests": [],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2022-06-02T15:55:36.753152089Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "elastic-agent",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/share/elastic-agent:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "BEAT_SETUID_AS=elastic-agent",
                "ELASTIC_CONTAINER=true",
                "GODEBUG=madvdontneed=1",
                "LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVERRIDE=/"
            ],
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/usr/share/elastic-agent",
            "Entrypoint": [
                "/usr/bin/tini",
                "--",
                "/usr/local/bin/docker-entrypoint"
            ],
            "OnBuild": null,
            "Labels": {
                "description": "Agent manages other beats based on configuration provided.",
                "io.k8s.description": "Agent manages other beats based on configuration provided.",
                "io.k8s.display-name": "Elastic-Agent image",
                "license": "Elastic License",
                "maintainer": "[email protected]",
                "name": "elastic-agent",
                "org.label-schema.build-date": "2022-06-02T15:52:47Z",
                "org.label-schema.license": "Elastic License",
                "org.label-schema.name": "elastic-agent",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.url": "https://www.elastic.co/beats/elastic-agent",
                "org.label-schema.vcs-ref": "857b681dc5802e4ee82c3aa6855f9a5412181fb9",
                "org.label-schema.vcs-url": "github.com/elastic/elastic-agent",
                "org.label-schema.vendor": "Elastic",
                "org.label-schema.version": "8.4.0-SNAPSHOT",
                "org.opencontainers.image.created": "2022-06-02T15:52:47Z",
                "org.opencontainers.image.licenses": "Elastic License",
                "org.opencontainers.image.title": "Elastic-Agent",
                "org.opencontainers.image.vendor": "Elastic",
                "release": "1",
                "summary": "elastic-agent",
                "url": "https://www.elastic.co/beats/elastic-agent",
                "vendor": "Elastic",
                "version": "8.4.0-SNAPSHOT"
            }
        },
        "Architecture": "arm64",
        "Variant": "v8",
        "Os": "linux",
        "Size": 1483632954,
        "VirtualSize": 1483632954,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/idhs7vtyjjixh18ackanesi9w/diff:/var/lib/docker/overlay2/l12hzgvilgfvlaupn6zoc24ps/diff:/var/lib/docker/overlay2/ggo2z9trgcax0tt3q53t19jio/diff:/var/lib/docker/overlay2/fncuort16scuv2t358uok8ot1/diff:/var/lib/docker/overlay2/0rkwl6wt6adsuz0oaz4r7m7ts/diff:/var/lib/docker/overlay2/94mujnprz4hmb0kqv9ghxrfqz/diff:/var/lib/docker/overlay2/0yzjq4k49m5l4xu4uxc3p5v13/diff:/var/lib/docker/overlay2/t56ab2wc0fhw7kanmbffb0fr9/diff:/var/lib/docker/overlay2/k8ez123adv97gp3m3q9zll1rj/diff:/var/lib/docker/overlay2/kg86iya712mvttr5s8n08potu/diff:/var/lib/docker/overlay2/twxlc7x045ei8pwkzbr6n2o50/diff:/var/lib/docker/overlay2/8w5cyo4cdjj74guv1gybuf6y1/diff:/var/lib/docker/overlay2/276beedb128fc56b0273ca72b5df24e73ea9253d2ebf154337ce7dc7c89d6ab1/diff",
                "MergedDir": "/var/lib/docker/overlay2/v84i3k38crogbm6jpwj3ph239/merged",
                "UpperDir": "/var/lib/docker/overlay2/v84i3k38crogbm6jpwj3ph239/diff",
                "WorkDir": "/var/lib/docker/overlay2/v84i3k38crogbm6jpwj3ph239/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:e265835b28ac16782ef429b44427c7a72cdefc642794515d78a390a72a2eab42",
                "sha256:50fd10747d0c3c8a5dc78a95ad185d17686c42466c4b8cae65c907821aea6763",
                "sha256:d054b8d90973526b421644e4c3f1aee8a4843585d270aa3b8819bc6350342fa2",
                "sha256:dd723fa89ebe3d2faf8d3c322cc95216dc472af025d7ed2dcd8ff2f2ea6cb165",
                "sha256:aa06c70a0496a4574ae4dda10ee573712fa85a17f6d4a112b4790eb0454b7de4",
                "sha256:45248ec865363e3e2dfa14e4dec0efeb65783a094f0fd4c38777a7ac723ba131",
                "sha256:f80b23844aced45a1e16c2e96f414a20ba8c50f973a4752bea9e309da9bcbd59",
                "sha256:aa0c1c7e08c18bf271dba60ea00d9247eff6edc221374acdfe328cec939781da",
                "sha256:d9d75ad851a07e9b07bc76645c26fa3c45906159345d016e47a76acdd0040472",
                "sha256:8ed672ce63b357b6d7681c96c9207c5cd38dd0b72aeeacb7774dbe0fccdded8e",
                "sha256:1ba8d2f9445273f4a589cd4c14479d35c556227f6fb9c32eb2f7fedc75c1ed31",
                "sha256:d4f84fe81adc48d3b304d20e1e0ee0e3c6b6d44378db7ca2323fab267c7db698",
                "sha256:5ab8453592ce781030fab8a2bbe73b821e030aff3f8c2421cc5acb43765ac859",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
            ]
        },
        "Metadata": {
            "LastTagTime": "2022-06-02T15:55:39.896361299Z"
        }
    }
]

[kuisathaverat]

after confirming it works, I think we can close the issue.

[amitkanfer]

Thank you all !