From 2aa3f0b0937e93fb0e1cd8c229e835ed8b1fe303 Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Mon, 13 Nov 2023 23:14:00 -0600 Subject: [PATCH] Add section on using CI artifacts in PRs. (#444) Adding a section on this to the docs because I've needed to refer to it when answering questions at least 3 times. cc: @ajschmidt8 @vyasr @harrism --------- Co-authored-by: AJ Schmidt Co-authored-by: Mark Harris <783069+harrism@users.noreply.github.com> --- resources/github-actions.md | 58 +++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/resources/github-actions.md b/resources/github-actions.md index 182377bc863..2e1fe466e9b 100644 --- a/resources/github-actions.md +++ b/resources/github-actions.md @@ -45,6 +45,64 @@ There is a link provided at the end of every C++ and Python build job where the ![](/assets/images/downloads.png) +## Using Conda CI Artifacts in Other PRs + +For changes that cross library boundaries, it may be necessary to test a pull request to one library with changes from a pull request to another library. +Consider the overall RAPIDS dependency graph when testing. +For example, if you are testing artifacts from an RMM PR `rmm#A` in cuML, you probably also need to create a cuDF PR `cudf#B` that uses the artifacts from `rmm#A`, and then your cuML test PR will need to include the artifact channels for both `rmm#A` and `cudf#B`. + +To do this, it is necessary to download CI artifacts (described in the above section) from one library during the CI workflow of another library. +First, determine the pull request number(s) to be tested from the other library. +Then, fetch the CI artifacts from the other library's pull request and use them when building and testing. +The example code below demonstrates building and testing with conda packages from other library PRs. +Replace the pull request numbers and library names as needed. +Remember that changes to use CI artifacts should be _temporary_ and should be reverted prior to merging any required changes in that PR. + +**Example 1:** Building `libcuml` (C++) using `librmm`, `libraft`, `libcumlprims_mg` PR artifacts. + +```sh +# ci/build_cpp.sh + +LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1095 cpp) +LIBRAFT_CHANNEL=$(rapids-get-pr-conda-artifact raft 1388 cpp) +LIBCUMLPRIMS_CHANNEL=$(rapids-get-pr-conda-artifact cumlprims_mg 129 cpp) + +# Build library packages with the CI artifact channels providing the updated dependencies + +rapids-mamba-retry mambabuild \ + --channel "${LIBRMM_CHANNEL}" \ + --channel "${LIBRAFT_CHANNEL}" \ + --channel "${LIBCUMLPRIMS_CHANNEL}" \ + conda/recipes/libcuml +``` + +**Example 2:** Testing `cudf` (Python) using `librmm`, `rmm`, and `libkvikio` PR artifacts. + +```sh +# ci/test_python_common.sh + +LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 cpp) +RMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 python) +LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact kvikio 224 cpp) + +# Install library packages with the CI artifact channels providing the updated dependencies for testing + +rapids-mamba-retry install \ + --channel "${CPP_CHANNEL}" \ + --channel "${PYTHON_CHANNEL}" \ + --channel "${LIBRMM_CHANNEL}" \ + --channel "${LIBKVIKIO_CHANNEL}" \ + --channel "${RMM_CHANNEL}" \ + cudf libcudf +``` + +Note that the custom channel for PR artifacts is needed in the build scripts _and_ the test scripts, for C++ _and_ Python. +If building/testing a Python package that depends on a C++ library, it is necessary to use PR artifacts from that C++ library and not just Python (e.g. if testing `rmm` artifacts, you must use the corresponding `librmm` CI artifacts as well as `rmm`). +In some repos, the `test_python.sh` is quite complicated with multiple calls to conda/mamba. +We recommend that the Python and C++ artifact channels should be added to every call of `rapids-mamba-retry` / `rapids-conda-retry` "just in case." + +Note: By default `rapids-get-pr-conda-artifact` uses the most recent commit from the specified PR. +A commit hash from the dependent PR can be added as an optional 4th argument to test with an earlier commit or to pin testing to a commit even if the dependent PR is updated. ## Skipping CI for Commits See the GitHub Actions documentation page below on how to prevent GitHub Actions from running on certain commits. This is useful for preventing GitHub Actions from running on pull requests that are not fully complete. This also helps preserve the finite GPU resources provided by the RAPIDS Ops team.