Native CI for osx-arm64 (run tests, native compilation, ...)

[h-vetinari]

Things work pretty well for M1's with all the work that's gone into cross-compilation (and the daily(?) job testing published artefacts), but it would still be nice to be able to test the packaged artifacts on the feedstocks before publishing them.

Adding M1 agents is now on the github roadmap (github/roadmap#528), so perhaps once deployed this could be an interesting way to test the packages. conda-forge already uses GHA's for various things, but not to my knowledge on any feedstock, so this would probably need some work on smithy.

The roadmap issue isn't yet assigned to a specific quarter in the GH roadmap, so it might still be a while, unfortunately. But then, maybe other CI providers like azure end up being faster with offering osx-arm. In any case, I thought I'd open this issue for tracking.

[ngam]

But then, maybe other CI providers like azure end up being faster with offering osx-arm

It's likely that Azure and GHA will offer it at the same time together.

It'll be interesting how we will smoothly move our CI; I guess it could simply be a change in conda-smithy... but idk.

It's also good to coordinate more closely with Anaconda because they now offer M1 packages https://repo.anaconda.com/pkgs/main/osx-arm64/ and I think they're natively built and tested, so it may make sense to coordinate with them, especially if they can communicate their findings (if any) with feedstocks. I tried to look into their CI logs but they're not publicly available.

Generally though, it seems that conda-forge's cross-compilation for osx-arm64 has worked really well. Fwiw, I personally have started using conda only recently (mid 2021) and my main experience with it has been on M1 machines --- I don't remember having any issues.

[BastianZim]

The roadmap issue isn't yet assigned to a specific quarter in the GH roadmap,

I reached out to GitHub support when they added it to the roadmap, and they plan it by the end of the financial year 2022.

[jaimergp]

The beta is now scheduled for Q3 2023:

I found these providers already (or soonish) available, in case someone wants to experiment:

• https://cirrus-ci.org/faq/#are-there-any-limits (you get a single VM with 12 CPUs, so orchestration might be tricky in terms of isolation, cache race conditions, etc, but I haven't looked into it)
• https://www.bitrise.io/why/technologies/virtualized-m1-environment (we might need to reach out to see how we can be part of the trial; their free -Hobby- plan has 5 concurrent jobs and 150 mins per month so... not too much)
• https://circleci.canny.io/cloud-feature-requests/p/support-new-m1-arm-based-macs (allegedly available by the end of February, and we have some infra for this already)

[jaimergp]

conda/conda experimenting with Cirrus M1 in this PR if you are curious on how it looks like: conda/conda#12079

[adrianinsaval]

circleci has m1 machines available, is that a possibility?

[jaimergp]

They are not part of the free plan, afaik: https://circleci.com/pricing/#comparison-table

[jakirkham]

GitHub posted a blogpost today about adding M1 workers

https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

[adrianinsaval]

It's not clear to me if that's also free for open repos like the rest of the runners

[BastianZim]

No, they are larger runners that always cost something: https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners#about-macos-larger-runners

[beckermr]

open-source runners have landed: https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

[jaimergp]

The ticket where we might hear from Azure about adding M1 runners: actions/runner-images#8971

[jaimergp]

Hey @conda-forge/core, I am curious about your opinion on the following idea: allowing selected feedstocks to run GHA osx-arm64 runners, like we do with the GPU CI.

• Maintainers would open up a PR in admin-requests applying for access with a justification of need. A core member would review and see if their case is justified (mostly that they can't cross-compile, I'd assume).
• Once merged, they rerender their feedstock and they'd be allowed to use it.

To note:

• We don't want to this to DoS our infra jobs.
• I don't know (yet) how to implement this so people don't abuse it.
• How do we restrict the number of CI jobs a single feedstock can request? And globally?

[h-vetinari]

allowing selected feedstocks to run GHA osx-arm64 runners, like we do with the GPU CI.

We have ~2200 feedstocks building arm64 packages according to the migrator status page, it seems to me we'd open a can of worms as in where to draw the line for who gets access to a GHA-based queue. For GPUs, we have much fewer affected packages, and a much cleaner line ("does the build time out on Azure?" or "do we absolutely need to test GPU-specific codepaths?").

For example, first-come-first-served sounds like a recipe to resource exhaustion; perhaps I'm too pessimistic but I foresee the possibility of many discussions with a high noise-to-utility ration.

On the other hand, it might still be better than nothing as a stopgap measure while we wait for arm-runners to (hopefully) arrive in Azure Pipelines. 🤷

[beckermr]

I think we'd only allow it if for cross-compiled builds are failing to work as opposed to first-come, first served.

[jaimergp]

Yes, I'd argue for a strong proof of "we have tried everything else and it doesn't work". Actually, the only feedstock I can think of is conda-standalone and even in that case it's ok to just build locally.

[anjos]

In this context, would e.g. luarocks-feedstock qualify for native builds? (c.f. conda-forge/luarocks-feedstock#28) ARM migration is stuck because tests want to run a native executable which of course doesn't work in a cross-compilation environment. Or would we prefer to avoid testing altogether in this instance?

[xhochy]

We avoid native testing for all osx-arm64 feedstocks and only run the tests for the feedstock that are just existence tests (i.e. using test -f)

[jaimergp]

I might be reading this wrong, but this page hints that macos-14 is available in preview in Azure now, which might mean arm64 runners?

The macOS-14 Sonoma image is available in preview

[xhochy]

I might be reading this wrong, but this page hints that macos-14 is available in preview in Azure now, which might mean arm64 runners?

The macOS-14 Sonoma image is available in preview

Immediately tried it out at conda-forge/mlx-feedstock#35 and it looks like amd64 runners.

[jaimergp]

Ah, pity. Yea, uname -m seems to return x86_64:

Installing a fresh version of Miniforge
+ MINIFORGE_URL=https://github.com/conda-forge/miniforge/releases/latest/download
++ uname -m
+ MINIFORGE_FILE=Mambaforge-MacOSX-x86_64.sh

[ilia-nikiforov-umn]

daily(?) job testing published artefacts

Not directly related to this issue, but am I understanding correctly that, even though the test section of my meta.yaml is being skipped on osx-arm64 in the feedstock, it is being run elsewhere after the package is built? If so, is there a place to see the output of these runs?

[beckermr]

It is NOT being run elsewhere.

[ilia-nikiforov-umn]

What does "testing published artefacts" entail, then?

[beckermr]

Download the artifact to an arm64 Mac. Then you can run conda build with tests only right on the artifact. I forget the exact command.

[h-vetinari]

daily(?) job testing published artefacts

If so, is there a place to see the output of these runs?

Note that when I wrote this, this was actually my understanding. It's possible that I had misunderstood this (hence the "(?)"), but AFAIU there had been some effort to do more systematic testing as we ramped up osx-arm64 support. I had never seen it, but heard it spoken about (obviously I'm not going to find references to the exact comments years later). It's possible though that this never extended beyond some core developers doing manual checks.

[ilia-nikiforov-umn]

Got it, thank you. We will continue testing manually until hopefully native CI becomes available soon!

[beckermr]

We did a bunch of systematic testing at first to verify the initial cross-compiled builds worked. That was all done by hand, never automated. We don't currently have automated testing, even daily, setup. Given that we are close to native ci (hopefully) and that we'd likely see more bug reports from people using the packages live, I don't think we'll invest time in testing now. It was much more useful when there many fewer osx-arm64 users.