Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW] Reduce code duplication for dask & distributed nightly/stable installs #11565

Merged
merged 17 commits into from
Sep 27, 2022
Merged

[REVIEW] Reduce code duplication for dask & distributed nightly/stable installs #11565

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f620679
temp commit
galipremsagar Aug 18, 2022
43b944d
Update build.sh
galipremsagar Aug 18, 2022
8152ab5
Update ci/gpu/build.sh
galipremsagar Aug 18, 2022
945fdf5
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Aug 19, 2022
31bfa6c
Update build.sh
galipremsagar Aug 19, 2022
d65f786
Update build.sh
galipremsagar Aug 19, 2022
80f8085
Update meta.yaml
galipremsagar Aug 19, 2022
70b0564
Update ci/gpu/build.sh
galipremsagar Aug 22, 2022
ab59187
fix dask pinnings
galipremsagar Aug 22, 2022
e15e9c5
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Aug 22, 2022
1e19549
Update ci/benchmark/build.sh
galipremsagar Aug 23, 2022
87ce71c
Apply suggestions from code review
galipremsagar Aug 24, 2022
b6516df
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Sep 26, 2022
bdc3fed
Update ci/gpu/build.sh
galipremsagar Sep 26, 2022
2c7c957
Merge branch 'rapidsai:branch-22.10' into test_dask
galipremsagar Sep 26, 2022
6335c68
Update ci/benchmark/build.sh
galipremsagar Sep 27, 2022
40ed5e5
Update ci/gpu/build.sh
galipremsagar Sep 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ci/benchmark/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ export LIBCUDF_KERNEL_CACHE_PATH="$HOME/.jitify-cache"
# Dask & Distributed option to install main(nightly) or `conda-forge` packages.
export INSTALL_DASK_MAIN=1
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved

# Dask version to install when `INSTALL_DASK_MAIN=0`
export DASK_STABLE_VERSION="2022.8.0"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Want to note that the relevant fixes for the mixed stable/nightly issue here are in conda-forge/dask-feedstock#191 and conda-forge/distributed-feedstock#218, since our problem is that when installing conda-forge dask we ended up pulling in nightly dask-core/distributed.

With this context, conda-forge/conda-forge-repodata-patches-feedstock#312 applies this change across all stable dask/distributed packages that would've had this issue, so once that's in we shouldn't have to bump the stable version here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, I'll hold off until conda-forge/conda-forge-repodata-patches-feedstock#312 is merged and will revert back to the prev stable version that we were pointing to as a minimum.

galipremsagar marked this conversation as resolved.
Show resolved Hide resolved

function remove_libcudf_kernel_cache_dir {
EXITCODE=$?
logger "removing kernel cache dir: $LIBCUDF_KERNEL_CACHE_PATH"
Expand Down Expand Up @@ -82,8 +85,8 @@ if [[ "${INSTALL_DASK_MAIN}" == 1 ]]; then
gpuci_logger "gpuci_mamba_retry update dask"
gpuci_mamba_retry update dask
else
gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall"
gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall
gpuci_logger "gpuci_mamba_retry install dask=={$DASK_STABLE_VERSION}"
gpuci_mamba_retry install dask=={$DASK_STABLE_VERSION}
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
fi

# Install the master version of streamz
Expand Down
10 changes: 8 additions & 2 deletions ci/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ unset GIT_DESCRIBE_TAG
# Dask & Distributed option to install main(nightly) or `conda-forge` packages.
export INSTALL_DASK_MAIN=1
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved

# Dask version to install when `INSTALL_DASK_MAIN=0`
export DASK_STABLE_VERSION="2022.8.0"
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved

# ucx-py version
export UCX_PY_VERSION='0.28.*'

Expand Down Expand Up @@ -92,8 +95,8 @@ function install_dask {
gpuci_mamba_retry update dask
conda list
else
gpuci_logger "gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall"
gpuci_mamba_retry install conda-forge::dask>=2022.7.1 conda-forge::distributed>=2022.7.1 conda-forge::dask-core>=2022.7.1 --force-reinstall
gpuci_logger "gpuci_mamba_retry install conda-forge::dask=={$DASK_STABLE_VERSION} conda-forge::distributed=={$DASK_STABLE_VERSION} conda-forge::dask-core=={$DASK_STABLE_VERSION} --force-reinstall"
gpuci_mamba_retry install conda-forge::dask=={$DASK_STABLE_VERSION} conda-forge::distributed=={$DASK_STABLE_VERSION} conda-forge::dask-core=={$DASK_STABLE_VERSION} --force-reinstall
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
fi
# Install the main version of streamz
gpuci_logger "Install the main version of streamz"
Expand Down Expand Up @@ -178,6 +181,9 @@ else

gpuci_logger "Installing cudf, dask-cudf, cudf_kafka and custreamz"
gpuci_mamba_retry install cudf dask-cudf cudf_kafka custreamz -c "${CONDA_BLD_DIR}" -c "${CONDA_ARTIFACT_PATH}"

gpuci_logger "Check current conda environment"
conda list --show-channel-urls

gpuci_logger "GoogleTests"
# Run libcudf and libcudf_kafka gtests from libcudf-tests package
Expand Down
4 changes: 2 additions & 2 deletions conda/environments/cudf_dev_cuda11.5.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ dependencies:
- pydocstyle=6.1.1
- typing_extensions
- pre-commit
- dask>=2022.7.1
- distributed>=2022.7.1
- dask>=2022.8.0
- distributed>=2022.8.0
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
- streamz
- arrow-cpp=9
- dlpack>=0.5,<0.6.0a0
Expand Down
4 changes: 2 additions & 2 deletions conda/recipes/custreamz/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ requirements:
- python
- streamz
- cudf ={{ version }}
- dask>=2022.7.1
- distributed>=2022.7.1
- dask>=2022.8.0
- distributed>=2022.8.0
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
- python-confluent-kafka >=1.7.0,<1.8.0a0
- cudf_kafka ={{ version }}

Expand Down
8 changes: 4 additions & 4 deletions conda/recipes/dask-cudf/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ requirements:
host:
- python
- cudf ={{ version }}
- dask>=2022.7.1
- distributed>=2022.7.1
- dask>=2022.8.0
- distributed>=2022.8.0
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
- cudatoolkit ={{ cuda_version }}
run:
- python
- cudf ={{ version }}
- dask>=2022.7.1
- distributed>=2022.7.1
- dask>=2022.8.0
- distributed>=2022.8.0
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
- {{ pin_compatible('cudatoolkit', max_pin='x', min_pin='x') }}

test: # [linux64]
Expand Down
4 changes: 2 additions & 2 deletions python/dask_cudf/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@

install_requires = [
"cudf",
"dask>=2022.7.1",
"distributed>=2022.7.1",
"dask>=2022.8.0",
"distributed>=2022.8.0",
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
"fsspec>=0.6.0",
"numpy",
"pandas>=1.0,<1.5.0dev0",
Expand Down