Enable macos packaging and tests #44

gbaraldi · 2022-03-09T14:07:58Z

Depends on #42. I have no idea if this is working or not. This is a rebase on this PR but with a similar structure to what is on #42

staticfloat · 2022-03-09T16:15:05Z

I just merged a big re-organization, sorry about that. You'll probably need to rebase this significantly.

gbaraldi · 2022-03-09T16:26:15Z

I was already building it on top of the branch of the merged PR, so it shouldn't be too hard.

staticfloat · 2022-03-09T19:21:52Z

Can you enable aarch64 as well?

staticfloat · 2022-03-09T19:22:23Z

pipelines/main/platforms/build_macos.arches

@@ -0,0 +1,13 @@
+# OS       TRIPLET                 ALLOW_FAIL    ARCH_ROOTFS    MAKE_FLAGS     TIMEOUT       ROOTFS_TAG    ROOTFS_HASH


No need for ARCH_ROOTFS, ROOTFS_TAG or ROOTFS_HASH here. Those columns can be deleted.

I think ARCH_ROOTFS is how the script currently knows what arch is running. Just need to make it so it gets the info from the triplet or the ARCH variable.

ARCH_ROOTFS should only be used with sandbox; any other uses of the arch should parse it out from the triplet, which is done by calc_version_envs.sh

pipelines/main/platforms/build_macos.yml

pipelines/main/platforms/test_macos.yml

pipelines/main/platforms/build_macos.yml

staticfloat · 2022-03-09T20:20:16Z

I got so, so confused for a minute looking at the build:

Notice how it purports to be building on x86_64, but the agent is an aarch64 agent! 🤯
We need to provide arch: ${ARCH} specifications in our agents: block. For that, we actually do need an ARCH column in our .arches.

gbaraldi · 2022-03-09T20:21:02Z

Yeah, I was very very confused, I was like wow, aarch64 is building very nice, then it was on the x86 builder 😄 . Do you want to add an specific ARCH columns, or just keep ARCH_ROOTFS, rename it or somerhing?

staticfloat · 2022-03-09T20:38:51Z

ARCH_ROOTFS is supposed to refer to the architecture of the rootfs image that we're downloading, which isn't happening on these machines, so I'd rather name it just ARCH.

gbaraldi · 2022-03-09T20:47:33Z

Isn't there an ARCH variable already? That calc_version_envs.sh sets?

staticfloat · 2022-03-09T20:52:15Z

Yes, but that variable is not available at YAML parse time, sadly.

gbaraldi · 2022-03-09T20:55:27Z

Fair enough, I guess adding to the .arches file won't hurt. Not sure why the aarch64 build didn't work. I donwloaded it and it gave me broken dylib errors 🤔 . Will try to see if current master actually works on the m1.
EDIT: it does.

staticfloat · 2022-03-09T22:44:53Z

The aarch64 tester is failing because the binaries are not codesigned. I've updated our testing fork of Julia to include the latest fixes from Mosè for that, so the next build you push should have more luck there.

This is looking pretty good! I think we're ready to embed the codesigning toolchain, so I went ahead and did that for you. Let's have the workflow be:

build
- generates .tar.gz
test
- waits for build
- downloads .tar.gz and extracts it
- re-signs the binaries with ad-hoc signatures (because the build doesn't have a codesigning key, so I don't think julia will run without that)
- runs tests
upload
- waits for test to complete
- downloads .tar.gz and extracts it
- re-signs the binaries with an actual codesigning key
- generates, signs and notarizes .dmg
- uploads tarball, tarball GPG key, and signed .dmg

gbaraldi · 2022-03-09T23:01:17Z

That sounds great to me, thanks :). I will see if later tonight or maybe tomorrow I can get to that.
It's quite incredible how much faster the m1 builds, it's almost 3x faster than the normal x86.
Getting the CI for CI working here was awesome for developing this., thanks elliot and @brenhinkeller .

staticfloat · 2022-03-09T23:48:23Z

It's quite incredible how much faster the m1 builds, it's almost 3x faster than the normal x86.

The x86_64 machine is pretty old: Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz, it's a personal machine of mine that I'm using for testing. The "true" buildbots that we tend to use are newer: Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz, so we should get significantly better performance out of those. Those new ones also have 12 cores instead of 4, so I think we can parallelize builds and test suites such that things run a little faster on them! :D

staticfloat · 2022-03-10T02:14:01Z

pipelines/main/platforms/test_macos.yml

+          echo "--- Ad-hoc sign for testing"
+          MACHO_FILES=$(find "$${JULIA_INSTALL_DIR}" -type f -perm -0111 | cut -d: -f1)
+          for f in ${MACHO_FILES}; do
+            codesign -s - --option=runtime -vvv --timestamp --deep --force "${f}"


If possible, let's make use of https://github.com/JuliaLang/julia/blob/master/contrib/codesign.sh here

Do we have access to it directly from the julia repo or do I add it to utilities?

You'll have access to it from the Julia repo, but we probabily need to update our fork.

gbaraldi · 2022-03-10T11:07:49Z

Why does adhoc signing not work here 🤔 why does apple make this so more difficult than it should be.

DilumAluthge · 2022-03-17T06:40:40Z

@staticfloat @gbaraldi Now that Buildkite has a built-in matrix feature, what do you guys think of trying it out for this PR?

Docs: https://buildkite.com/docs/pipelines/build-matrix

staticfloat · 2022-03-19T05:06:38Z

We may be able to do that, as long as we don't have to use the matrix attributes in a plugin. (E.g. we only interpolate the values within the command step).

Gabriel; are you blocked on anything, or just busy? I want to get this PR moving again, and just want to understand where things stand.

gbaraldi · 2022-03-19T12:20:59Z

Yeah my week was a bit busier than i expected, however today I will work on it some.

pipelines/main/platforms/upload_macos.yml

DilumAluthge · 2022-03-23T21:32:41Z

@gbaraldi Anything that would have been $ in the original Bash script will need to instead be $$ in the .yaml file.

pipelines/main/platforms/upload_macos.yml

staticfloat · 2022-04-05T15:31:40Z

Alright! I think the config here is good. Now we just need to figure out how to fix the Pkg problems without shunting them to node 1.

staticfloat · 2022-04-05T16:02:57Z

Pkg problems are worked around as of JuliaLang/julia#44828

So we can merge this when ready.

The main idea is to merge our `build`, `test` and `upload` steps into single, shared `.sh` files that get included from our `.yml` files. We try to walk the line correctly between storing Julia-related content (such as `julia.icns`) in the Julia repository, while storing the build rules (such as `build_dmg.sh`) within this repository.

gbaraldi · 2022-04-06T00:51:44Z

🥳 It's purple and green 🥳 thousands of segfaults notwithstanding.

staticfloat · 2022-04-06T01:03:42Z

Nice work, Gabriel!

gbaraldi closed this Mar 9, 2022

gbaraldi reopened this Mar 9, 2022

gbaraldi changed the title ~~Enable macos packaging on x86_64~~ Enable macos packaging and tests on x86_64 Mar 9, 2022

staticfloat reviewed Mar 9, 2022

View reviewed changes

pipelines/main/platforms/build_macos.yml Outdated Show resolved Hide resolved

staticfloat reviewed Mar 9, 2022

View reviewed changes

pipelines/main/platforms/test_macos.yml Outdated Show resolved Hide resolved

staticfloat reviewed Mar 9, 2022

View reviewed changes

pipelines/main/platforms/build_macos.yml Outdated Show resolved Hide resolved

gbaraldi changed the title ~~Enable macos packaging and tests on x86_64~~ Enable macos packaging and tests Mar 9, 2022

staticfloat reviewed Mar 10, 2022

View reviewed changes

DilumAluthge reviewed Mar 23, 2022

View reviewed changes

pipelines/main/platforms/upload_macos.yml Show resolved Hide resolved

DilumAluthge reviewed Mar 23, 2022

View reviewed changes

pipelines/main/platforms/upload_macos.yml Outdated Show resolved Hide resolved

DilumAluthge reviewed Mar 23, 2022

View reviewed changes

pipelines/main/platforms/upload_macos.yml Outdated Show resolved Hide resolved

JuliaCI deleted a comment from gbaraldi Mar 23, 2022

staticfloat force-pushed the macos_packaging branch 2 times, most recently from 55c160d to 08297ea Compare April 2, 2022 06:44

Allow for slightly easier overriding of upstream branch choice

e20da7a

staticfloat force-pushed the macos_packaging branch 14 times, most recently from 66ff398 to 52dda3c Compare April 5, 2022 14:16

staticfloat force-pushed the macos_packaging branch from 52dda3c to 2085542 Compare April 5, 2022 16:04

Add macos packaging, testing and uploading

43f62de

staticfloat force-pushed the macos_packaging branch from 2085542 to e197260 Compare April 5, 2022 16:07

staticfloat enabled auto-merge (rebase) April 5, 2022 16:07

staticfloat mentioned this pull request Apr 5, 2022

Build also x86_64-linux-musl with -Werror #86

Merged

staticfloat force-pushed the macos_packaging branch from e197260 to 3880043 Compare April 5, 2022 21:54

staticfloat merged commit 3a3c203 into JuliaCI:main Apr 6, 2022

staticfloat mentioned this pull request Dec 14, 2022

Add a build pipeline for FreeBSD #266

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable macos packaging and tests #44

Enable macos packaging and tests #44

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

staticfloat Mar 9, 2022

gbaraldi Mar 9, 2022 •

edited

Loading

staticfloat Mar 9, 2022

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022 •

edited

Loading

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022 •

edited

Loading

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

staticfloat Mar 10, 2022

gbaraldi Mar 10, 2022

DilumAluthge Mar 10, 2022

gbaraldi commented Mar 10, 2022

DilumAluthge commented Mar 17, 2022

staticfloat commented Mar 19, 2022

gbaraldi commented Mar 19, 2022

DilumAluthge commented Mar 23, 2022

staticfloat commented Apr 5, 2022

staticfloat commented Apr 5, 2022

gbaraldi commented Apr 6, 2022 •

edited

Loading

staticfloat commented Apr 6, 2022

		@@ -0,0 +1,13 @@
		# OS TRIPLET ALLOW_FAIL ARCH_ROOTFS MAKE_FLAGS TIMEOUT ROOTFS_TAG ROOTFS_HASH

Enable macos packaging and tests #44

Enable macos packaging and tests #44

Conversation

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

staticfloat Mar 9, 2022

Choose a reason for hiding this comment

gbaraldi Mar 9, 2022 • edited Loading

Choose a reason for hiding this comment

staticfloat Mar 9, 2022

Choose a reason for hiding this comment

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022 • edited Loading

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022 • edited Loading

staticfloat commented Mar 9, 2022

gbaraldi commented Mar 9, 2022

staticfloat commented Mar 9, 2022

staticfloat Mar 10, 2022

Choose a reason for hiding this comment

gbaraldi Mar 10, 2022

Choose a reason for hiding this comment

DilumAluthge Mar 10, 2022

Choose a reason for hiding this comment

gbaraldi commented Mar 10, 2022

DilumAluthge commented Mar 17, 2022

staticfloat commented Mar 19, 2022

gbaraldi commented Mar 19, 2022

DilumAluthge commented Mar 23, 2022

staticfloat commented Apr 5, 2022

staticfloat commented Apr 5, 2022

gbaraldi commented Apr 6, 2022 • edited Loading

staticfloat commented Apr 6, 2022

gbaraldi Mar 9, 2022 •

edited

Loading

gbaraldi commented Mar 9, 2022 •

edited

Loading

gbaraldi commented Mar 9, 2022 •

edited

Loading

gbaraldi commented Apr 6, 2022 •

edited

Loading