Update darwin sdk and version #87

michel-laterman · 2021-05-13T16:57:11Z

Update MacOS version + sdk to 10.13 and the corresponding Darwin version to 17.

elasticmachine · 2021-05-13T17:09:28Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #87 updated
Start Time: 2021-05-19T15:44:18.164+0000
Duration: 30 min 51 sec
Commit: cf7402b

Trends 🧪

michel-laterman · 2021-05-14T16:26:41Z

jenkins run tests

michel-laterman · 2021-05-14T23:17:29Z

Currently the darwin build will fail; if the build debug flag is enabled and output is not piped to null we can see:

#6 113.4 [ 84%] Linking CXX static library ../libLLVMMC.a
#6 113.4 [ 84%] Built target LLVMMC
#6 113.4 CMakeFiles/Makefile2:28639: recipe for target 'projects/libtapi/tools/libtapi/CMakeFiles/libtapi.dir/rule' failed
#6 113.4 make[1]: *** [projects/libtapi/tools/libtapi/CMakeFiles/libtapi.dir/rule] Error 2
#6 113.4 Makefile:8361: recipe for target 'libtapi' failed
#6 113.4 make: *** [libtapi] Error 2
#6 113.4 + _exit
#6 113.4 + EC=2
#6 113.4 + '[' 2 -ne 0 ']'
#6 113.4 + test -z build.sh
#6 113.4 + echo ''
#6 113.4
#6 113.4 + echo 'exiting with abnormal exit code (2)'
#6 113.4 exiting with abnormal exit code (2)
#6 113.5 + test -n 1
#6 113.5 + '[' -n apple-libtapi ']'
#6 113.5 + rm -f 'build/*_built_successfully'
#6 113.5 + echo ''
#6 113.5
#6 113.5 + test build.sh = build.sh
#6 113.5 + check_for_bug_1242300
#6 113.5 + '[' -e /etc/issue ']'
#6 113.5 ++ grep -i ubuntu.13.10 /etc/issue
#6 113.5 + '[' '' ']'

ph · 2021-05-17T19:13:51Z

@michel-laterman This is from the container?

michel-laterman · 2021-05-17T19:42:47Z

Yes, there are currently 2 issues

debian9 (the base image) installs a version of clang that is too old to compile osxcross
the sdk for 10.13 is missing the libc++ headers

ph · 2021-05-17T19:45:45Z

@urso What do you think about updating to a more recent version of debian? IIRC Debian 9 doesn't get any security updates anymore.

@michel-laterman with Debian10 do the two problems get resolved?

ph · 2021-05-17T19:46:40Z

@andrewkroh would be good to have you comment on this.

michel-laterman · 2021-05-17T20:23:45Z

@ph after recompressing (from bz2 to xz) a 10.14 SDK that another user has made (https://github.com/joseluisq/macosx-sdks/releases/tag/10.14) I was able to get osxcross compiling on debian10.

It seems that the releases that https://github.com/phracker/MacOSX-SDKs provides are broken for earler SDKs

urso · 2021-05-18T14:06:32Z

GO 1.14 is EOL, and 1.15 will soon be EOL as well. This is why we switch to 1.16. I think we can remove 1.14 already.

What do you think about updating to a more recent version of debian? IIRC Debian 9 doesn't get any security updates anymore.

For cross compiling to MacOS/Windows, the actual debian version should not matter.

For compiling on linux it matters unfortunately, and we need to keep debian7 in order to ensure our binaries continue to function with CentOS/RH7.
By accident I shared a Filebeat not build using the debian7 container 2 weeks ago, and the user was not able to run the binary on on RH7. RH7 goes EOL end of 2021, but we might have users that continue to use it even after.
The Beats-tester using a centos7 image should be able to discover potential problems (system linker complains on startup) if we update our debian build image. But debian10 might be too new I'm afraid.

after recompressing (from bz2 to xz) a 10.14 SDK that another user has made

We are moving here from MacOS 10.12 to 10.14. As MacOS 10.13 is EOL since December 2020 I think this is fine. We still want to report this as a breaking change in the next Beats release.

Cross compilation can be kinda painful. Plus we have to jump through some hoops in order to sign our binaries. For the future we might consider to switch to native build environments. @v1v WDYT?

kuisathaverat · 2021-05-18T15:54:39Z

GO 1.14 is EOL, and 1.15 will soon be EOL as well. This is why we switch to 1.16. I think we can remove 1.14 already.

we have removed 1.14 a couple of days ago, we only build 1.15 and 1.16

The Beats-tester using a centos7 image should be able to discover potential problems (system linker complains on startup)

it also runs tests on CentOS 6

Cross compilation can be kinda painful. Plus we have to jump through some hoops in order to sign our binaries. For the future we might consider to switch to native build environments.

We can switch now to build on native environments for x86, ARM, and MacOS. We have workers for those architectures. However, this requires changes on the build system, some of the package parts must be replaced.

…cos-update

kuisathaverat · 2021-05-18T16:12:06Z

@ph after recompressing (from bz2 to xz) a 10.14 SDK that another user has made (https://github.com/joseluisq/macosx-sdks/releases/tag/10.14) I was able to get osxcross compiling on debian10.

It seems that the releases that https://github.com/phracker/MacOSX-SDKs provides are broken for earler SDKs

Update your PR with the changes on the master I have fixed all the stuff broken last week in the systemd dependencies. I seems like the new SDK does not work on old Libcs, it is fine we can use that new SDK only on Debian 10 and the old in the other versions of the cross-compiler Docker images. Then on the package build, we should choose the right Docker image depending on our target, on MacOS we do not have the issue related to libc compatibility is other OS.

kuisathaverat · 2021-05-18T16:13:49Z

go1.16/darwin/Dockerfile.tmpl

+ARG OSXCROSS_SDK_URL=https://s3.amazonaws.com/beats-files/deps/MacOSX10.13.sdk.tar.xz
 ARG OSXCROSS_PATH=/usr/osxcross
-ARG OSXCROSS_REV=3034f7149716d815bc473d0a7b35d17e4cf175aa
-ARG SDK_VERSION=10.11
-ARG DARWIN_VERSION=15
-ARG OSX_VERSION_MIN=10.10
+ARG OSXCROSS_REV=8a716a43a72dab1db9630d7824ee0af3730cb8f9
+ARG SDK_VERSION=10.13
+ARG DARWIN_VERSION=17
+ARG OSX_VERSION_MIN=10.12


{{if eq .DEBIAN_VERSION "10"}} ARG OSXCROSS_SDK_URL=https://s3.amazonaws.com/beats-files/deps/MacOSX10.13.sdk.tar.xz ARG OSXCROSS_PATH=/usr/osxcross ARG OSXCROSS_REV=8a716a43a72dab1db9630d7824ee0af3730cb8f9 ARG SDK_VERSION=10.13 ARG DARWIN_VERSION=17 ARG OSX_VERSION_MIN=10.12 {{ else }} ARG OSXCROSS_SDK_URL=https://s3.amazonaws.com/beats-files/deps/MacOSX10.11.sdk.tar.xz ARG OSXCROSS_PATH=/usr/osxcross ARG OSXCROSS_REV=3034f7149716d815bc473d0a7b35d17e4cf175aa ARG SDK_VERSION=10.11 ARG DARWIN_VERSION=15 ARG OSX_VERSION_MIN=10.10 {{ end }}

this resolve the issue we will have the SDK 10.11 on Debian 7/8/9 and the SDK 10.13 on Debian 10

…sbuild into macos-update

fix: use GCP bucket

kuisathaverat · 2021-05-19T09:11:46Z

now the SDK is downloaded and the osxcross compiled, but the helloWorld test falied

[2021-05-19T09:05:04.551Z] Step 15/21 : RUN cd /   && o64-clang helloWorld.c -o helloWorld   && file helloWorld   && file helloWorld | grep -c 'Mach-O 64-bit x86_64'

[2021-05-19T09:05:04.551Z]  ---> Running in 5c632da30120

[2021-05-19T09:05:04.551Z] /usr/osxcross/bin/x86_64-apple-darwin18-ld: error while loading shared libraries: libtapi.so.8svn: cannot open shared object file: No such file or directory

go1.16/darwin/Dockerfile.tmpl

kuisathaverat · 2021-05-19T11:09:33Z

ready to go

…sbuild into macos-update

Jenkinsfile

v1v · 2021-05-19T12:36:43Z

README.md

+```
+
+The SDK should be in the working directory.
+The tmp dir can be safely deleted after this.


Do you think the upload to the bucket should be also added in this section?

Yes, please.

michel-laterman marked this pull request as draft May 13, 2021 16:58

Update darwin sdk and version

79087d7

michel-laterman force-pushed the macos-update branch from fc4cdbf to 79087d7 Compare May 14, 2021 16:46

Update crossbuild repo

383ac51

Merge branch 'master' of github.com:elastic/golang-crossbuild into ma…

445a07d

…cos-update

kuisathaverat reviewed May 18, 2021

View reviewed changes

michel-laterman and others added 6 commits May 18, 2021 09:44

Merge branch 'master' into macos-update

b96b1e1

Add MacOS SDK instructions, support debian10 for darwin images

c756c1a

Fix README formatting

7b16125

Merge branch 'macos-update' of github.com:michel-laterman/golang-cros…

03884df

…sbuild into macos-update

fix: use eq condition

820d3c1

fix: use GCP bucket

fix: wrong URL for 10.14 SDK

f7e0aac

fix: make symbolic links to the libraries on /usr/lib

29d2988

kuisathaverat reviewed May 19, 2021

View reviewed changes

go1.16/darwin/Dockerfile.tmpl Outdated Show resolved Hide resolved

Update go1.16/darwin/Dockerfile.tmpl

f438c54

kuisathaverat added 2 commits May 19, 2021 13:14

fix: re-enable the 1.15 build

ed6fbe1

Merge branch 'macos-update' of github.com:michel-laterman/golang-cros…

7eb96a9

…sbuild into macos-update

fix: missing colon

a6617a4

kuisathaverat reviewed May 19, 2021

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

disable 1.15

3060e7a

v1v reviewed May 19, 2021

View reviewed changes

Add GCP bucket details

cf7402b

michel-laterman marked this pull request as ready for review May 19, 2021 15:45

kuisathaverat approved these changes May 19, 2021

View reviewed changes

v1v approved these changes May 19, 2021

View reviewed changes

michel-laterman merged commit e96f6ca into elastic:master May 19, 2021

michel-laterman deleted the macos-update branch May 19, 2021 19:30

michel-laterman mentioned this pull request May 28, 2021

Upgrade to Golang 1.16.4 elastic/beats#25346

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update darwin sdk and version #87

Update darwin sdk and version #87

michel-laterman commented May 13, 2021

elasticmachine commented May 13, 2021 •

edited

Loading

Build stats

Trends 🧪

michel-laterman commented May 14, 2021

michel-laterman commented May 14, 2021

ph commented May 17, 2021

michel-laterman commented May 17, 2021

ph commented May 17, 2021

ph commented May 17, 2021

michel-laterman commented May 17, 2021

urso commented May 18, 2021

kuisathaverat commented May 18, 2021 •

edited

Loading

kuisathaverat commented May 18, 2021

kuisathaverat May 18, 2021 •

edited

Loading

kuisathaverat May 18, 2021

kuisathaverat commented May 19, 2021

kuisathaverat commented May 19, 2021

v1v May 19, 2021

urso May 19, 2021

michel-laterman May 19, 2021

Update darwin sdk and version #87

Update darwin sdk and version #87

Conversation

michel-laterman commented May 13, 2021

elasticmachine commented May 13, 2021 • edited Loading

💚 Build Succeeded

Build stats

Trends 🧪

michel-laterman commented May 14, 2021

michel-laterman commented May 14, 2021

ph commented May 17, 2021

michel-laterman commented May 17, 2021

ph commented May 17, 2021

ph commented May 17, 2021

michel-laterman commented May 17, 2021

urso commented May 18, 2021

kuisathaverat commented May 18, 2021 • edited Loading

kuisathaverat commented May 18, 2021

kuisathaverat May 18, 2021 • edited Loading

Choose a reason for hiding this comment

kuisathaverat May 18, 2021

Choose a reason for hiding this comment

kuisathaverat commented May 19, 2021

kuisathaverat commented May 19, 2021

v1v May 19, 2021

Choose a reason for hiding this comment

urso May 19, 2021

Choose a reason for hiding this comment

michel-laterman May 19, 2021

Choose a reason for hiding this comment

elasticmachine commented May 13, 2021 •

edited

Loading

kuisathaverat commented May 18, 2021 •

edited

Loading

kuisathaverat May 18, 2021 •

edited

Loading