Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW] Reduce code duplication for dask & distributed nightly/stable installs #11565

Merged
merged 17 commits into from
Sep 27, 2022
Merged

[REVIEW] Reduce code duplication for dask & distributed nightly/stable installs #11565

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f620679
temp commit
galipremsagar Aug 18, 2022
43b944d
Update build.sh
galipremsagar Aug 18, 2022
8152ab5
Update ci/gpu/build.sh
galipremsagar Aug 18, 2022
945fdf5
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Aug 19, 2022
31bfa6c
Update build.sh
galipremsagar Aug 19, 2022
d65f786
Update build.sh
galipremsagar Aug 19, 2022
80f8085
Update meta.yaml
galipremsagar Aug 19, 2022
70b0564
Update ci/gpu/build.sh
galipremsagar Aug 22, 2022
ab59187
fix dask pinnings
galipremsagar Aug 22, 2022
e15e9c5
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Aug 22, 2022
1e19549
Update ci/benchmark/build.sh
galipremsagar Aug 23, 2022
87ce71c
Apply suggestions from code review
galipremsagar Aug 24, 2022
b6516df
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Sep 26, 2022
bdc3fed
Update ci/gpu/build.sh
galipremsagar Sep 26, 2022
2c7c957
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Sep 26, 2022
6335c68
Update ci/benchmark/build.sh
galipremsagar Sep 27, 2022
40ed5e5
Update ci/gpu/build.sh
galipremsagar Sep 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ci/benchmark/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ export LIBCUDF_KERNEL_CACHE_PATH="$HOME/.jitify-cache"
# Dask & Distributed option to install main(nightly) or `conda-forge` packages.
export INSTALL_DASK_MAIN=1
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved

# Dask version to install when `INSTALL_DASK_MAIN=0`
export DASK_STABLE_VERSION="2022.7.1"

function remove_libcudf_kernel_cache_dir {
EXITCODE=$?
logger "removing kernel cache dir: $LIBCUDF_KERNEL_CACHE_PATH"
Expand Down Expand Up @@ -82,8 +85,8 @@ if [[ "${INSTALL_DASK_MAIN}" == 1 ]]; then
gpuci_logger "gpuci_mamba_retry update dask"
gpuci_mamba_retry update dask
else
gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall"
gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall
gpuci_logger "gpuci_mamba_retry install dask=={$DASK_STABLE_VERSION}"
gpuci_mamba_retry install dask=={$DASK_STABLE_VERSION}
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
fi

# Install the master version of streamz
Expand Down
10 changes: 8 additions & 2 deletions ci/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ unset GIT_DESCRIBE_TAG
# Dask & Distributed option to install main(nightly) or `conda-forge` packages.
export INSTALL_DASK_MAIN=1
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved

# Dask version to install when `INSTALL_DASK_MAIN=0`
export DASK_STABLE_VERSION="2022.7.1"

# ucx-py version
export UCX_PY_VERSION='0.28.*'

Expand Down Expand Up @@ -91,8 +94,8 @@ function install_dask {
gpuci_mamba_retry update dask
conda list
else
gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall"
gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall
gpuci_logger "gpuci_mamba_retry install dask=={$DASK_STABLE_VERSION}"
gpuci_mamba_retry install dask=={$DASK_STABLE_VERSION}
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
fi
# Install the main version of streamz
gpuci_logger "Install the main version of streamz"
Expand Down Expand Up @@ -188,6 +191,9 @@ else
# copied by CI from the upstream 11.5 jobs into $CONDA_ARTIFACT_PATH
gpuci_logger "Installing cudf, dask-cudf, cudf_kafka, and custreamz"
gpuci_mamba_retry install cudf dask-cudf cudf_kafka custreamz -c "${CONDA_BLD_DIR}" -c "${CONDA_ARTIFACT_PATH}"

gpuci_logger "Check current conda environment"
conda list --show-channel-urls

gpuci_logger "GoogleTests"
# Run libcudf and libcudf_kafka gtests from libcudf-tests package
Expand Down