Restructure our CI workflows

[jleibs]

Context

Our github CI workflows have grown organically over the last 6 months and are becoming somewhat haphazard / fragile. We've learned a number of tricks through that time that are helpful, but not everything is applied consistently.

Major pain points:

• Python code is not being tested on PRs.
• Release jobs aren't predictably executed. (Seems like an inconsistency between the triggering vs the filtering of tag-related changes).
• Need to build all wheels to create new rrd files even though only linux is necessary.
• Debugging & testing changes to the workflows themselves are slow.

Proposal

Here is an alternative workflow arrangement refactoring things into 4 major workflows.

The major changes are that it organizes workflows by phase of development rather than by type of build (e.g. python vs rust).

It also transitions us to making much heavier use of gcloud rather than github as an artifact cache. One notable change from this is pre-release wheels would simply be hosted on gcloud rather than needing to point at build/release artifacts. This generally gives us better control of lifecycle, etc. and lets us keep all the old builds around without cluttering the github releases page as we did previously. User experience would look something like:

pip install https://wheels.rerun.io/6e4ddaf/rerun_sdk-0.3.1+2023.3.23.6e4ddaf-cp38-abi3-manylinux_2_31_x86_64.whl

The actual workflows are as follows:

1. PR Gates

This is a small workflow that contains all jobs related to meta things like required labels and tags, keywords that should prevent merging, too-large-of-files, existence of invalid file patterns (symlinks?) etc.

Triggers:

• any commit on a PR

2. Lint, Build, Test

This is the primary workflow that contains most of the current jobs. By default runs on linux only, but windows / mac can be included when running via workflow_dispatch input..

Triggers:

• any commit on a PR
• any commit pushed to main
• manually via workflow input flags for which OS to build/test.
• nightly build across all OSes

Jobs:

• Linting for both rust and python.
• Matrixed:
  • Build & test rust code.
  • Build python wheel
  • Run python tests using built wheel.
• Benchmarks (optionally via dispatch input or push to main).

3. Create Packages

This workflow is just for preparing cargo packages, wheels, rrd files, and app.rerun.io wasm bundles. All artifacts are pushed to gcloud buckets based on the commit hash. This does not deploy to crates.io or pypi.

Almost of the things that could be built should be parameterizable via the job input.

The net result of this job is a bunch of package artifacts exist on gcloud. The output of the individual jobs should strive to be idempotent. If you realize you want more of the things to be built, running it a second or third time against the same commit should be safe. For example, we might want to run this for wasm bundles on every commit, and additionally for all wheels hourly.

Triggers:

• manually
• periodic "prerelease" (somewhere between every commit and nightly) if there are changes on head of main.

Jobs (all optional):

• Build linux wheel
• Build wasm bundle
• Create RRD files using linux wheel
• Build additional wheels
• Create python documentation
• Create rust documentation
• Upload wheels to gcloud
• Upload crates to gcloud
• Upload Python docs to gcloud
• Upload Rust docs to gcloud
• Upload RRD files to gcloud
• Upload wasm bundle to gcloud
• Create tag / draft release.

4. Release

This is another small workflow that just orchestrates uploading of pre-built artifacts from gcloud to crates.io, pypi. The script should be doing no runtime building -- only download/validation/upload/signing etc.

Triggers:

• manually only

Jobs:

• Publish pypi packages
• Publish crates.io packages
• Update 'latest' and 'prerelease' tags and/or gcloud symlinks
• Mark release as published

TODO:

• Draw the Proposed Execution Graph

[Wumpf]

Sounds really good to me overall, but I'm worried about having the PR check that small. It sure would be cheaper but that will lead to a lot more red main branch

[jleibs]

Both 1 and 2 should block PR. I need a better name for 1 though to make that more clear. 1 is about the meta content of the PR and 2 is about the code in the PR.

[emilk]

Maybe related:

• Tracking issue: Reduce friction in adding documentation #1754

[emilk]

Another CI fail: just py-format changes the cube.ipynb, but the CI didn't fail on it 🤦