Generalize GHA select statements (to avoid hard-coding versions)

[jakirkham]

As there is only need of Dask-CUDA build, we filter out one architecture, Python version, and CUDA version in GHA. So as to only build the package once

https://github.com/rapidsai/dask-cuda/blob/1eecb1b2ac79ae9aaff9c26d0a3c93dd57f859f3/.github/workflows/build.yaml#L69-L70

However this currently hard-codes the versions of each in the selection logic, which means it can go stale as new versions are added and old ones dropped. Potentially resulting in the build being lost altogether (maybe even silently)

To avoid pinning to a specific version, @ajschmidt8 made several suggestions in this thread: rapidsai/dask-cuda#1294 (comment)

Filing this to track for follow-up

[jakirkham]

Here is another hard-coding example in cuDF

https://github.com/rapidsai/cudf/blob/3adfddcfa2cdac4acb16a50916442763a1d8a78b/.github/workflows/build.yaml#L94-L95

[jakirkham]

Also some relevant discussion in this thread: rapidsai/cudf#15174 (comment)

[bdice]

I worked through a bit of jq and I think I have a decent solution.

---

Case 1: Pure Python, for each CUDA major version

For pure Python packages that need to build for each CUDA major version, like dask-cudf-cu11 / dask-cudf-cu12, we use the highest (Python, CUDA) pair, from groups of each CUDA major version.

map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))]))

This was originally posted here: rapidsai/cudf#15174 (comment)

Case 2: Pure Python, no CUDA dependency

For jobs that only need a single matrix entry (no dependence on a specific Python or CUDA version), we should use the highest (Python, CUDA) pair.

map(select(.ARCH == "amd64")) | max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))]) | [.]

---

Optionally, you can drop the map(select(.ARCH == "amd64")) if you want jobs for both amd64 and arm64.

These expressions assume that we do not care what OS value (LINUX_VER), GPU hardware (gpu), or driver (driver) is used in the resulting jobs.

[jameslamb]

@bdice I tried the "Pure Python, no CUDA dependency" example in rapidsai/rapids-dask-dependency#28

Worked perfectly, thank you 😊

[jakirkham]

Also needed in cuxfilter

https://github.com/rapidsai/cuxfilter/blob/cfab998ffc4f348f649cd839bea5697fd8aeef02/.github/workflows/pr.yaml#L67

[bdice]

I did cudf as well. rapidsai/cudf#15191

There are some hardcoded references remaining but they're not really used in the same way as the cases we've addressed above.

Docker needs something like if latest CUDA but there's no matrix filter so I'm not sure if there's a good solution.
https://github.com/rapidsai/docker/blob/32a60e107c3ff777c7f2ddc2ac4d11a3669c3f83/.github/workflows/build-image.yml#L145

cuGraph has some PyG tests that only run on CUDA 11. I don't think these are worth modifying since it's pinning to an older version. There is some open work needed to upgrade to PyTorch 2, which I think may open up the ability to use CUDA 12 here.
https://github.com/rapidsai/cugraph/blob/ac65b17ee3e9b85368f266da1a6a3b8e5717e292/.github/workflows/pr.yaml#L165
https://github.com/rapidsai/cugraph/blob/ac65b17ee3e9b85368f266da1a6a3b8e5717e292/.github/workflows/test.yaml#L79

[jakirkham]

Thanks Bradley! 🙏

Yeah think we can close this once cuDF is fixed

[bdice]

cuDF's PR is merged: rapidsai/cudf#15191

This should be safe to close now.