Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install test dependencies at the same time as cuml packages. #5781

Merged
merged 4 commits into from
Feb 26, 2024
Merged

Install test dependencies at the same time as cuml packages. #5781

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1770700
Install test dependencies at the same time as cuml packages.
bdice Feb 22, 2024
44dcfc6
Add rapidsai channel to force DAG sort to prioritize local artifact c…
bdice Feb 22, 2024
3a3e1a8
Use rapids-dependency-file-generator with --prepend-channels.
bdice Feb 23, 2024
38ac4a1
Add test_cuml to test_notebooks.
bdice Feb 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions ci/test_cpp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,24 @@ rapids-logger "Generate C++ testing dependencies"
rapids-dependency-file-generator \
--output conda \
--file_key test_cpp \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee test.yaml

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

cat <<EOF | tee cuml.yaml
name: cuml
channels:
- ${CPP_CHANNEL}
bdice marked this conversation as resolved.
Show resolved Hide resolved
- rapidsai
dependencies:
- libcuml
- libcuml-tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming that conda-merge will place CPP_CHANNEL at a sufficient priority so that we grab build artifacts, but do we lose anything also constraining the packages to that channel?

Suggested change
- libcuml
- libcuml-tests
- ${CPP_CHANNEL}::libcuml
- ${CPP_CHANNEL}::libcuml-tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good question. I designed rapidsai/dependency-file-generator#67 in such a way that the CI artifact channels will always be prepended. It's still possible that another channel could be used, but there's not a good way to inject the channel names corresponding to those packages if we go with the route of rapidsai/dependency-file-generator#67.

EOF

# TODO: install conda-merge in the CI images
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created a PR to the CI images for this: rapidsai/ci-imgs#112 but that would be unnecessary if we adopt this alternative approach: #5781 (comment)

pip install conda-merge
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small question: why use pip for installing conda-merge?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fastest to do it this way for the prototype. I am installing this into the conda environment in rapidsai/ci-imgs#112.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may also want to add conda-merge to the base image if we find it useful. Could imagine this pattern showing up in other projects' CI

conda-merge test.yaml cuml.yaml > env.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we cat the output of this file to see the placement of the new channels in the resulting environment file?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checked this locally, looks like the order isn't super defined when we don't have a common channel between the merged environments:

channels:
- rapidsai
- /tmp/cpp_channel
- rapidsai-nightly
- /tmp/python_channel
- dask/label/dev
- conda-forge
- nvidia

Think that putting rapidsai below the artifact channels in the new environment file should be sufficient to get the channel ordering we desire (at least for this PR, think the approach in rapidsai/dependency-file-generator#67 should improve things)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, thanks for checking that. I agree the approach with prepend-channels will solve this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about the CPP test failure, but looks like we're grabbing from the correct channel now 🎉

https://github.com/rapidsai/cuml/actions/runs/8005972527/job/21867897863?pr=5781


rapids-mamba-retry env create --force -f env.yaml -n test

Expand All @@ -18,17 +35,11 @@ set +u
conda activate test
set -u

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
rapids-print-env

RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}/
mkdir -p "${RAPIDS_TESTS_DIR}"

rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
libcuml libcuml-tests

rapids-logger "Check GPU usage"
nvidia-smi

Expand Down
32 changes: 21 additions & 11 deletions ci/test_notebooks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2020-2023, NVIDIA CORPORATION.
# Copyright (c) 2020-2024, NVIDIA CORPORATION.
set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh
Expand All @@ -8,7 +8,26 @@ rapids-logger "Generate Notebook testing dependencies"
rapids-dependency-file-generator \
--output conda \
--file_key test_notebooks \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee test.yaml

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

cat <<EOF | tee cuml.yaml
name: cuml
channels:
- ${CPP_CHANNEL}
- ${PYTHON_CHANNEL}
bdice marked this conversation as resolved.
Show resolved Hide resolved
- rapidsai
dependencies:
- libcuml
- cuml
EOF

# TODO: install conda-merge in the CI images
pip install conda-merge
conda-merge test.yaml cuml.yaml > env.yaml

rapids-mamba-retry env create --force -f env.yaml -n test

Expand All @@ -17,17 +36,8 @@ set +u
conda activate test
set -u

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
libcuml cuml

rapids-logger "Check GPU usage"
nvidia-smi

Expand Down
34 changes: 22 additions & 12 deletions ci/test_python_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2022-2023, NVIDIA CORPORATION.
# Copyright (c) 2022-2024, NVIDIA CORPORATION.

set -euo pipefail

Expand All @@ -9,7 +9,26 @@ rapids-logger "Generate Python testing dependencies"
rapids-dependency-file-generator \
--output conda \
--file_key test_python \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee test.yaml

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

cat <<EOF | tee cuml.yaml
name: cuml
channels:
- ${CPP_CHANNEL}
- ${PYTHON_CHANNEL}
bdice marked this conversation as resolved.
Show resolved Hide resolved
- rapidsai
dependencies:
- libcuml
- cuml
EOF

# TODO: install conda-merge in the CI images
pip install conda-merge
conda-merge test.yaml cuml.yaml > env.yaml

rapids-mamba-retry env create --force -f env.yaml -n test

Expand All @@ -18,21 +37,12 @@ set +u
conda activate test
set -u

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
rapids-print-env

RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}
RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${PWD}/coverage-results"}
mkdir -p "${RAPIDS_TESTS_DIR}" "${RAPIDS_COVERAGE_DIR}"

rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
libcuml cuml

rapids-logger "Check GPU usage"
nvidia-smi

Expand Down
Loading