Out of $$$ for CI

[asomers]

Since November, Cirrus has been limiting OSS projects' free compute time. Nix just hit that limit. So we need to either raise some cash, reduce our activity, or economize our CI pipeline. Here are some ideas:

• Maybe reduce the QEMU builds?
• Set up some precommit scripting to take care of stuff like rustfmt?
• Move the OSX/arm64 and Linux/x86_64 builds into Github Pipelines, leaving FreeBSD/x86_64 and Linux/arm64 on Cirrus?
• Shut off github merge queues? They don't work exactly like bors. With bors, batching PRs together could reduce your CI load. With github merge queues, it seems to increase it.
• Merge chore: use towncrier for CHANGELOG #2149 . By reducing merge conflicts, that should also reduce our CI builds.

https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/
https://cirrus-ci.com/settings/github/nix-rust

Originally posted by @asomers in #2155 (comment)

[Xuanwo]

First of all, I am familiar with GHA and would like to offer assistance for migration.

---

Here are my thoughts, from my perspective, which may be helpful for your decision:

• Set up some precommit scripting to take care of stuff like rustfmt?

Jobs like rustfmt are simple and cheap but required, we can migrate them to GHA. (I'm willing to help for this)

• Move the OSX/arm64 and Linux/x86_64 builds into Github Pipelines

GHA's free plan doesn't provide OSX/arm64 support, we may need to pay for macos-latest-xlarge.

Some projects are using qemu for OSX/arm64 with free plan: SLOW, but works.

• leaving FreeBSD/x86_64 and Linux/arm64 on Cirrus?

Maintaining workflows across different platforms can be challenging, so it's better for us to stick to the same one. I suggest maintaining a self-hosted FreeBSD runner instead. It may also be easier for us to find a sponsor of this runner!

[asomers]

First of all, I am familiar with GHA and would like to offer assistance for migration.

• Set up some precommit scripting to take care of stuff like rustfmt?

Jobs like rustfmt are simple and cheap but required, we can migrate them to GHA. (I'm willing to help for this)

We could move that one into GHA. But even better would be to force contributors to run it before opening a PR. That's what saltstack does, although their setup is so fragile that I had to disable it.

• Move the OSX/arm64 and Linux/x86_64 builds into Github Pipelines

GHA's free plan doesn't provide OSX/arm64 support, we may need to pay for macos-latest-xlarge.

Is GHA's default OSX runner still on x86_64? Weird. Cirrus discontinued that platform and replaced it with aarch64. I thought that Apple was pushing people to move in that direction. For our purposes, either would work. Nix contains plenty of OSX-specific code, but none is specific to a certain architecture on OSX.

• leaving FreeBSD/x86_64 and Linux/arm64 on Cirrus?

Maintaining workflows across different platforms can be challenging, so it's better for us to stick to the same one. I suggest maintaining a self-hosted FreeBSD runner instead. It may also be easier for us to find a sponsor of this runner!

Oh, I don't want to do that. Back before #1003 we did use a self-hosted FreeBSD runner. But it was a lot more work. And much of the configuration didn't live within the Nix repository, so it was hard to manage. Plus, it wasn't free, either, costing about $20/month even for a small VM.

I think that migrating some of the builds into GHA would likely have the biggest impact. It would create some annoyance in that we'd have to manage two CI files, but that's doable. Other projects do it, like https://github.com/tokio-rs/mio .

[SteveLauC]

#2163 moves format check to a pre-commit script

[SteveLauC]

Just gave a look at our CI configuration file, currently, we have 10 tasks:

1. FreeBSD Amd64/i386(cargo test)

   Since GitHub Action does not provide FreeBSD instances, leave it on Cirrus.

   There are some folks telling me that we can run FreeBSD test on GHA under macOS instances, haven't given it an investigation.

2. macOS aarch64(cargo test)

   Nix contains plenty of OSX-specific code, but none is specific to a certain architecture on OSX.

   Since we don't have arch-specific code on macOS, move it to GHA

   From my experience, macOS instances on GHA can be really slow, we can consider move it back if this happens in the future.

3. Use cross(QEMU) to run cross test for the following targets on x86_64-unknown-linux-gnu

   docker or podman is needed

   arm-unknown-linux-gnueabi
   armv7-unknown-linux-gnueabihf
   i686-unknown-linux-gnu
   i686-unknown-linux-musl
   mips-unknown-linux-gnu
   mips64-unknown-linux-gnuabi64
   mips64el-unknown-linux-gnuabi64
   mipsel-unknown-linux-gnu
   powerpc64le-unknown-linux-gnu
   

   Not sure if GH provides docker (or podman), friendly ping @Xuanwo, is this something GH can do?

4. Native Linux builds(cargo test)

   • aarch64-unknown-linux-gnu
   • x86_64-unknown-linux-gnu
   • x86_64-unknown-linux-musl

   For the x86_64 parts, move them to GHA, leaving aarch64 target in Cirrus

5. Latest Rust stable toolchain (cargo test)

   Runs on x86_64-unknown-linux-gnu, move it to GHA

   This task, seems to be the ONLY one task that runs with the toolchain other than our MSRV

6. cross-compiling(cargo check)

   Run cargo check for the following targets on x86_64-unknown-linux-gnu

   • aarch64-linux-android
   • arm-linux-androideabi
   • armv7-linux-androideabi
   • i686-linux-android
   • x86_64-linux-android
   • arm-unknown-linux-musleabi
   • x86_64-unknown-fuchsia(tier3-nix, tier2-rust)
   • x86_64-unknown-illumos
   • aarch64-apple-ios
   • powerpc64-unknown-linux-gnu(tier3-nix, tier1-rust)
   • s390x-unknown-linux-gnu
   • x86_64-unknown-linux-gnux32(tier3-nix, tier2-rust)
   • x86_64-unknown-netbsd

   Since the host is x86_64-unknown-linux-gnu, we can move it to GHA

7. Redox(cargo +nightly check)

8. Tier3

   Run cargo +nightly check on x86_64-unknown-linux-gnu for the following targets

   • DragonFly BSD x86_64
   • OpenBSD x86_64
   • armv7-unknown-linux-uclibceabihf
   • Haiku x86_64

   Host is x86_64-unknown-linux-gnu, move it to GHA

9. Minver

   $ cargo update -Zminimal-versions
   $ cargo check

   runs on x86_64-unknown-linux-gnu, move it to GHA

10. Rust formatter

    Move it to a pre-commit script

---

Most of them are running on x86_64-unknown-linux-gnu and thus can be migrated to GHA, our migration shouldn't be a problem :)

[SteveLauC]

Some updates on this:

Our Cirrus configuration file uses Yaml Anchors to reuse the code so that duplicate code blocks can be avoided, however, GitHub Action does not support this feature. It seems that the most recommended way to reuse code would be to define your own actions.

But this is the GitHub way, Cirrus does not use it, if we go with this approach, then we have to manage two configurations, which increases our maintenance overhead.

[SteveLauC]

#2166, our first try with GHA