Skip to content

Commit

Permalink
Add section on using CI artifacts in PRs. (#444)
Browse files Browse the repository at this point in the history
Adding a section on this to the docs because I've needed to refer to it
when answering questions at least 3 times.

cc: @ajschmidt8 @vyasr @harrism

---------

Co-authored-by: AJ Schmidt <[email protected]>
Co-authored-by: Mark Harris <[email protected]>
  • Loading branch information
3 people authored Nov 14, 2023
1 parent 1d9a390 commit 2aa3f0b
Showing 1 changed file with 58 additions and 0 deletions.
58 changes: 58 additions & 0 deletions resources/github-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,64 @@ There is a link provided at the end of every C++ and Python build job where the

![](/assets/images/downloads.png)

## Using Conda CI Artifacts in Other PRs

For changes that cross library boundaries, it may be necessary to test a pull request to one library with changes from a pull request to another library.
Consider the overall RAPIDS dependency graph when testing.
For example, if you are testing artifacts from an RMM PR `rmm#A` in cuML, you probably also need to create a cuDF PR `cudf#B` that uses the artifacts from `rmm#A`, and then your cuML test PR will need to include the artifact channels for both `rmm#A` and `cudf#B`.

To do this, it is necessary to download CI artifacts (described in the above section) from one library during the CI workflow of another library.
First, determine the pull request number(s) to be tested from the other library.
Then, fetch the CI artifacts from the other library's pull request and use them when building and testing.
The example code below demonstrates building and testing with conda packages from other library PRs.
Replace the pull request numbers and library names as needed.
Remember that changes to use CI artifacts should be _temporary_ and should be reverted prior to merging any required changes in that PR.

**Example 1:** Building `libcuml` (C++) using `librmm`, `libraft`, `libcumlprims_mg` PR artifacts.

```sh
# ci/build_cpp.sh

LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1095 cpp)
LIBRAFT_CHANNEL=$(rapids-get-pr-conda-artifact raft 1388 cpp)
LIBCUMLPRIMS_CHANNEL=$(rapids-get-pr-conda-artifact cumlprims_mg 129 cpp)

# Build library packages with the CI artifact channels providing the updated dependencies

rapids-mamba-retry mambabuild \
--channel "${LIBRMM_CHANNEL}" \
--channel "${LIBRAFT_CHANNEL}" \
--channel "${LIBCUMLPRIMS_CHANNEL}" \
conda/recipes/libcuml
```

**Example 2:** Testing `cudf` (Python) using `librmm`, `rmm`, and `libkvikio` PR artifacts.

```sh
# ci/test_python_common.sh

LIBRMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 cpp)
RMM_CHANNEL=$(rapids-get-pr-conda-artifact rmm 1223 python)
LIBKVIKIO_CHANNEL=$(rapids-get-pr-conda-artifact kvikio 224 cpp)

# Install library packages with the CI artifact channels providing the updated dependencies for testing

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
--channel "${LIBRMM_CHANNEL}" \
--channel "${LIBKVIKIO_CHANNEL}" \
--channel "${RMM_CHANNEL}" \
cudf libcudf
```

Note that the custom channel for PR artifacts is needed in the build scripts _and_ the test scripts, for C++ _and_ Python.
If building/testing a Python package that depends on a C++ library, it is necessary to use PR artifacts from that C++ library and not just Python (e.g. if testing `rmm` artifacts, you must use the corresponding `librmm` CI artifacts as well as `rmm`).
In some repos, the `test_python.sh` is quite complicated with multiple calls to conda/mamba.
We recommend that the Python and C++ artifact channels should be added to every call of `rapids-mamba-retry` / `rapids-conda-retry` "just in case."

Note: By default `rapids-get-pr-conda-artifact` uses the most recent commit from the specified PR.
A commit hash from the dependent PR can be added as an optional 4th argument to test with an earlier commit or to pin testing to a commit even if the dependent PR is updated.
## Skipping CI for Commits

See the GitHub Actions documentation page below on how to prevent GitHub Actions from running on certain commits. This is useful for preventing GitHub Actions from running on pull requests that are not fully complete. This also helps preserve the finite GPU resources provided by the RAPIDS Ops team.
Expand Down

0 comments on commit 2aa3f0b

Please sign in to comment.