Build darwin/aarch64 and universal binary #203

AndersonQ · 2022-03-14T17:37:16Z

What does this PR do?

It enables the Elastic-Agent and Beats to be built for darwin/arm64 and generates a universal binary (darwin/universal).

The universal binary is the merge of the darwin/arm64 with darwin/amd64. Therefore it requires to first build the darwin/arm64 and darwin/amd64 and them "merge" them together into the darwin/universal. Also darwin/universal isn't a valid platform or OS/ARCH pair, therefore it isn't recognised by the Go compiler. Given that, the choice was to add darwin/arm64 as a new platform for the Elastic-Agent and when ever both platforms (darwin/arm64 and darwin/amd64) are built, also generate and package the universal binary.
Even though it's ~~quite~~ an implicit behaviour, the alternative would also require a lot of changes and exceptions to have our build process recognising darwin/universal, but compiling darwin/arm64 and darwin/amd64 and merging them. Also the current process makes available all three of them, allowing the user to choose which fits better their needs. The drawback of the universal binary is its size, the double of the single binaries.

A new target, BuildDarwinUniversal was added on the x-packs magefiles required by the Elastic-Agent.

Also fixes some small typos, redundant information on the help command and remove the use of some deprecated functions.

Why is it important?

We need to support Mac with M1 chips

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

build darwin/arm64 binary
build universal 2 binary
use mage to automatically build the universal binary
don't forget beats PR: Beats to compile natively on a Mac M1 machine beats#29582

How to test this PR locally

Set up the beats repo:
- have the beats repo on the same level as the elastic-agent repo
- set the beats repo to the branch 26683-agent-arm64 while [Beats] Build darwin/arm64 and universal binary beats#29585 isn't merged
Build the elastic-agent for the new platforms:
- darwin/arm64: PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage package
- darwin/universal: PLATFORMS="darwin/arm64,darwin/amd64" PACKAGES="tar.gz" mage package
Run the elastic-agent

####Watch out:
The elastic-agent tries to download any binary it might need to comply with the assigned policy. However none of the new binaries from this PR are present on the artifact API and other applications (e.g. fleet-server and elastic-endpoint) also aren't publishing darwin/arm64 versions. Therefore if the elastic-agent tries to download them, it'll fail. This is the expected behavior and this PR does not address it, see Out of Scope.

Out of scope:

Handle downloading binaries that does not publish darwin-universal artifacts on the artifact API
Handle upgrading a darwin/amd64 agent running under Rosetta2 on a Mac with M1 chip. This will be addressed on another PR.

Related issues

Relates [Meta] Mac M1 Support #151
Closes Elastic Agent to compile natively on a Mac M1 machine #204

* docs small changes * add arm64 to darwin architectures * docs/comments wee changes * stop using deprecated funcations * it works * now it works * add darwin-binary-arm64 to packageArchMap * add darwin/arm64 to ci packaging * use aarch instead of arm64 for package names

mergify · 2022-03-14T17:37:44Z

This pull request does not have a backport label. Could you fix it @AndersonQ? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-03-14T17:57:17Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-22T13:41:57.936+0000
Duration: 37 min 29 sec

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

blakerouse

Overall I think this looks good.

blakerouse · 2022-03-16T20:53:44Z

dev-tools/mage/common.go

+	// ParallelCtx should be used instead and a refactor is need to remove calls
+	// Parallel? If so, context.TODO() is more appropriated as it makes explicit
+	// a context should came from the caller and the current context is just a
+	// temporary solution until the needed refactor is made.


I think you are correct, should be refactored to always take a context.

ph

Change LGTM, thanks for improving the comments too!
I am currently testing locally

Using this PR and the beats PR and the following command.

 EXTERNAL=false PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage package

ph · 2022-03-18T13:58:02Z

@AndersonQ Are we missing an update for osquerybeat in the beats repository PR? Running the above command yield the following error or maybe there is something missing in the osquerybeat build @aleksmaus ?

>> package: Building osquerybeat type=tar.gz for platform=darwin/arm64
package ran for 58.257699773s
Error: failed building osquerybeat type=tar.gz for platform=darwin/arm64: failed adding file={Source:build/data/install/darwin/amd64/osquery-5.0.1.pkg Content: Template: Target:osquery-5.0.1.pkg Mode:-rw-r--r-- Config:false Modules:false Dep:<nil> Owner: SkipOnMissing:false Symlink:false} to tar: lstat build/data/install/darwin/amd64/osquery-5.0.1.pkg: no such file or directory
package ran for 4m56.647351202s
Error: exit status 1

mergify · 2022-03-18T17:51:39Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agent-aarch upstream/agent-aarch
git merge upstream/main
git push upstream agent-aarch

aleksmaus · 2022-03-21T13:36:10Z

@AndersonQ Are we missing an update for osquerybeat in the beats repository PR? Running the above command yield the following error or maybe there is something missing in the osquerybeat build @aleksmaus ?

I'll take a look, the osquerybeat fetched the osquery distro during the build, looks like needs adjustment for darwin/arm64.

AndersonQ · 2022-03-21T13:37:32Z

oh, yes, it'll try to build the beats assuming the beats repo is on ../beats

aleksmaus · 2022-03-21T13:53:57Z

@AndersonQ
running this in the beats/x-pack/osquerybeat

SNAPSHOT=true DEV=true PLATFORMS="darwin/arm64" mage -v package

leads to this error:

Running dependency: CrossBuildGoDaemon
Running dependency: CrossBuild
Skipping cross-build of target=buildGoDaemon because platforms list is empty.
exec: docker info -f {{ json .}}
Running dependency: github.com/elastic/beats/v7/dev-tools/mage.CrossBuild.func1
exec: go mod download
Running dependency: github.com/elastic/beats/v7/dev-tools/mage.buildMage
exec: mage -f -goos=linux -goarch=amd64 -compile build/mage-linux-amd64
crossBuild: Platform list = [{darwin/arm64 cgo}]
package ran for 5.708173097s
Error: unsupported cross build platform darwin/arm64

what is the expectation of the beat repo? anybody is working on adjusting it to run for darwin/arm64?

AndersonQ · 2022-03-21T14:13:31Z

I'm on it. It seems the removal of the elastic-agent code from the beats repo broke the M1 PR.

AndersonQ · 2022-03-21T16:10:33Z

@aleksmaus @ph
The elastic-agent needs to build the beats as well, as the darwin-aarch64 isn't yet available on the artefact API. To do so the following is required:

Set up the beats repo:

have the beats repo on the same level as the elastic-agent repo
set the beats repo to the branch 26683-agent-arm64 while [Beats] Build darwin/arm64 and universal binary beats#29585 isn't merged

I updated the PR description with these instructions

aleksmaus · 2022-03-21T16:36:09Z

@aleksmaus @ph The elastic-agent needs to build the beats as well, as the darwin-aarch64 isn't yet available on the artefact API. To do so the following is required:

I updated the PR description with these instructions

Thank you! Will have PR for osquerybeat shortly to support darwin/arm64.

aleksmaus · 2022-03-21T16:46:24Z

Opened PR against branch 26683-agent-arm64
That should fix the distro packaging for osquerybeat

AndersonQ · 2022-03-21T18:19:58Z

/test

AndersonQ · 2022-03-22T09:29:47Z

/test

mergify · 2022-03-22T13:34:36Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agent-aarch upstream/agent-aarch
git merge upstream/main
git push upstream agent-aarch

ph · 2022-03-22T14:34:21Z

dev-tools/mage/common.go

@@ -573,7 +594,7 @@ func ParallelCtx(ctx context.Context, fns ...interface{}) {
 // Parallel runs the given functions in parallel with an upper limit set based
 // on GOMAXPROCS.
 func Parallel(fns ...interface{}) {
-	ParallelCtx(context.Background(), fns...)
+	ParallelCtx(context.TODO(), fns...)


ph

LGTM, I've tested the build, I haven't tested on m1.

aleksmaus · 2022-03-23T14:08:22Z

@ph @AndersonQ it's still a problem with building on M1

Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.17.8-darwin-arm64-debian10
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[CC=oa64-clang, CXX=oa64-clang++, GOARCH=arm64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-arm64]
# runtime/cgo
qemu: uncaught target signal 11 (Segmentation fault) - core dumped

I already mentioned before that the docker image tagged with arm64 is actually amd64 arch and qemu can't run that

AndersonQ · 2022-03-23T17:41:06Z

hi @aleksmaus, yes, we know that. So far we don't have a fix for that :/
For the record on this PR, this issue is already reported here: elastic/golang-crossbuild#153

…in/universal (#203)" This reverts commit 3fc13b4.

amitkanfer · 2022-03-28T07:50:44Z

@AndersonQ ,
The CSP team all have M1 laptops, and now that our cloudbeat is part of agent, it'll be great to be able to compile locally.
Do we have an ETA for the M1 issue to be resolved?
In the meantime. can we spin up a compilation server + script that will rsync the files from the local laptop, compile, and bring back the artifacts?

Thank you!

AndersonQ · 2022-03-28T16:55:40Z

@amitkanfer I'm on PTO this whole week, perhaps @jlind23 could help you

mergify bot assigned AndersonQ Mar 14, 2022

mergify bot added the backport-skip label Mar 14, 2022

AndersonQ mentioned this pull request Mar 14, 2022

[Meta] Mac M1 Support #151

Closed

23 tasks

AndersonQ added 3 commits March 15, 2022 15:35

fix lint issue

69a7692

add changelog

f1680fe

fix lint issues

14b4234

AndersonQ requested a review from ph March 17, 2022 18:03

blakerouse approved these changes Mar 17, 2022

View reviewed changes

ph added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Mar 18, 2022

ph reviewed Mar 18, 2022

View reviewed changes

Merge branch 'main' into agent-aarch

3f6eeea

add darwin/arm64 to ci build

25a6015

aleksmaus mentioned this pull request Mar 21, 2022

Osquerybeat: Add darwin/arm64 packaging support elastic/beats#30935

Merged

2 tasks

AndersonQ added the ci:end-to-end Enable the e2e-testing label Mar 21, 2022

remove question

122e1fb

Merge branch 'main' into agent-aarch

c5e46cf

ph reviewed Mar 22, 2022

View reviewed changes

ph approved these changes Mar 22, 2022

View reviewed changes

AndersonQ merged commit 3fc13b4 into main Mar 23, 2022

AndersonQ deleted the agent-aarch branch March 23, 2022 08:36

AndersonQ added a commit that referenced this pull request Mar 24, 2022

Revert "build and package the elastic-agent for darwin/arm64 and darw…

849363e

…in/universal (#203)" This reverts commit 3fc13b4.

AndersonQ mentioned this pull request Mar 24, 2022

Revert "Build darwin/aarch64 and universal binary " #250

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build darwin/aarch64 and universal binary #203

Build darwin/aarch64 and universal binary #203

AndersonQ commented Mar 14, 2022 •

edited

Loading

mergify bot commented Mar 14, 2022

elasticmachine commented Mar 14, 2022 •

edited

Loading

Build stats

blakerouse left a comment

blakerouse Mar 16, 2022

ph left a comment

ph commented Mar 18, 2022

mergify bot commented Mar 18, 2022

aleksmaus commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

aleksmaus commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

aleksmaus commented Mar 21, 2022

aleksmaus commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

AndersonQ commented Mar 22, 2022

mergify bot commented Mar 22, 2022

ph Mar 22, 2022

ph left a comment

aleksmaus commented Mar 23, 2022

AndersonQ commented Mar 23, 2022

amitkanfer commented Mar 28, 2022

AndersonQ commented Mar 28, 2022

Build darwin/aarch64 and universal binary #203

Build darwin/aarch64 and universal binary #203

Conversation

AndersonQ commented Mar 14, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Out of scope:

Related issues

mergify bot commented Mar 14, 2022

elasticmachine commented Mar 14, 2022 • edited Loading

💚 Build Succeeded

Build stats

🤖 GitHub comments

blakerouse left a comment

Choose a reason for hiding this comment

blakerouse Mar 16, 2022

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

ph commented Mar 18, 2022

mergify bot commented Mar 18, 2022

aleksmaus commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

aleksmaus commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

Set up the beats repo:

aleksmaus commented Mar 21, 2022

aleksmaus commented Mar 21, 2022

AndersonQ commented Mar 21, 2022

AndersonQ commented Mar 22, 2022

mergify bot commented Mar 22, 2022

ph Mar 22, 2022

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

aleksmaus commented Mar 23, 2022

AndersonQ commented Mar 23, 2022

amitkanfer commented Mar 28, 2022

AndersonQ commented Mar 28, 2022

AndersonQ commented Mar 14, 2022 •

edited

Loading

elasticmachine commented Mar 14, 2022 •

edited

Loading