Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify wheel build scripts and allow alphas of RAPIDS dependencies #13963

Merged
merged 11 commits into from
Aug 31, 2023
Merged

Simplify wheel build scripts and allow alphas of RAPIDS dependencies #13963

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3cac174
Centralize logic
vyasr Aug 25, 2023
4c35021
Make sure suffix exists in outer script
vyasr Aug 25, 2023
902fa7d
Move the comparator into the capture so that ptxcompiler and cubinlin…
vyasr Aug 25, 2023
3016a00
Fix the sed logic
vyasr Aug 25, 2023
a0fa3e6
Fix regex in dask-cudf
vyasr Aug 25, 2023
4aa9a1a
Merge remote-tracking branch 'origin/branch-23.10' into feat/simplify…
vyasr Aug 25, 2023
84a86da
More typo fixes
vyasr Aug 25, 2023
682959a
Merge branch 'branch-23.10' into feat/simplify_wheel_builds
vyasr Aug 30, 2023
98e8dcd
Merge remote-tracking branch 'origin/branch-23.10' into feat/simplify…
vyasr Aug 31, 2023
0455387
Remove now superfluous numba patch
vyasr Aug 31, 2023
1b68591
Merge branch 'branch-23.10' into feat/simplify_wheel_builds
vyasr Aug 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions ci/build_wheel.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/bin/bash
# Copyright (c) 2023, NVIDIA CORPORATION.

set -euo pipefail

package_name=$1
package_dir=$2

source rapids-configure-sccache
source rapids-date-string

# Use gha-tools rapids-pip-wheel-version to generate wheel version then
# update the necessary files
version_override="$(rapids-pip-wheel-version ${RAPIDS_DATE_STRING})"

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# This is the version of the suffix with a preceding hyphen. It's used
# everywhere except in the final wheel name.
PACKAGE_CUDA_SUFFIX="-${RAPIDS_PY_CUDA_SUFFIX}"

# Patch project metadata files to include the CUDA version suffix and version override.
pyproject_file="${package_dir}/pyproject.toml"

sed -i "s/^version = .*/version = \"${version_override}\"/g" ${pyproject_file}
sed -i "s/name = \"${package_name}\"/name = \"${package_name}${PACKAGE_CUDA_SUFFIX}\"/g" ${pyproject_file}

# For nightlies we want to ensure that we're pulling in alphas as well. The
# easiest way to do so is to augment the spec with a constraint containing a
# min alpha version that doesn't affect the version bounds but does allow usage
# of alpha versions for that dependency without --pre
vyasr marked this conversation as resolved.
Show resolved Hide resolved
alpha_spec=''
if ! rapids-is-release-build; then
alpha_spec=',>=0.0.0a0'
fi

if [[ ${package_name} == "dask_cudf" ]]; then
sed -r -i "s/cudf==(.*)\"/cudf${PACKAGE_CUDA_SUFFIX}==\1${alpha_spec}\"/g" ${pyproject_file}
else
sed -r -i "s/rmm(.*)\"/rmm${PACKAGE_CUDA_SUFFIX}\1${alpha_spec}\"/g" ${pyproject_file}
# ptxcompiler and cubinlinker aren't version constrained
sed -r -i "s/ptxcompiler\"/ptxcompiler${PACKAGE_CUDA_SUFFIX}\"/g" ${pyproject_file}
sed -r -i "s/cubinlinker\"/cubinlinker${PACKAGE_CUDA_SUFFIX}\"/g" ${pyproject_file}
fi

if [[ $PACKAGE_CUDA_SUFFIX == "-cu12" ]]; then
sed -i "s/cuda-python[<=>\.,0-9a]*/cuda-python>=12.0,<13.0a0/g" ${pyproject_file}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know if upper bound pinnings like this (<13.0a0) affect whether alphas are allowed, too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't, but you're right to find that strange. For example:

(main) dt08% pip download 'numpy>=1.25,<2.0.0a'
Collecting numpy<2.0.0a,>=1.25
  Obtaining dependency information for numpy<2.0.0a,>=1.25 from https://files.pythonhosted.org/packages/32/6a/65dbc57a89078af9ff8bfcd4c0761a50172d90192eaeb1b6f56e5fbf1c3d/numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 7.5 MB/s eta 0:00:00
Saved ./numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Successfully downloaded numpy
(main) dt08% pip download 'numpy>=1.25.0a0,<2.0.0a'
Collecting numpy<2.0.0a,>=1.25.0a0
  Obtaining dependency information for numpy<2.0.0a,>=1.25.0a0 from https://files.pythonhosted.org/packages/35/8b/b669836be53d7b6697bc290abcdda701fd924a22c702713bbdf2c6c5bef5/numpy-1.26.0b1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached numpy-1.26.0b1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (58 kB)
Using cached numpy-1.26.0b1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.2 MB)
Saved ./numpy-1.26.0b1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Successfully downloaded numpy

So there's definitely a degree of depending on pip implementation details here. My guess is that nobody ever wanted the upper bound alpha to trigger this so they didn't notice this oversight. I'm not sure we can improve on my solution at present, but our testing should catch if we run into issues fairly quickly since we won't get the necessary dependencies, so I'm not too worried. I'm confident pip won't break the >=a0 behavior.

sed -i "s/cupy-cuda11x/cupy-cuda12x/g" ${pyproject_file}
sed -i "s/numba[<=>\.,0-9]*/numba>=0.57/g" ${pyproject_file}
vyasr marked this conversation as resolved.
Show resolved Hide resolved
sed -i "/ptxcompiler/d" ${pyproject_file}
sed -i "/cubinlinker/d" ${pyproject_file}
fi

cd "${package_dir}"

python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check
24 changes: 7 additions & 17 deletions ci/build_wheel_cudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,14 @@

set -euo pipefail

source rapids-configure-sccache
source rapids-date-string
package_dir="python/cudf"

# Use gha-tools rapids-pip-wheel-version to generate wheel version then
# update the necessary files
version_override="$(rapids-pip-wheel-version ${RAPIDS_DATE_STRING})"
export SKBUILD_CONFIGURE_OPTIONS="-DCUDF_BUILD_WHEELS=ON -DDETECT_CONDA_ENV=OFF"

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

ci/release/apply_wheel_modifications.sh ${version_override} "-${RAPIDS_PY_CUDA_SUFFIX}"
echo "The package name and/or version was modified in the package source. The git diff is:"
git diff

cd python/cudf
./ci/build_wheel.sh cudf ${package_dir}

SKBUILD_CONFIGURE_OPTIONS="-DCUDF_BUILD_WHEELS=ON -DDETECT_CONDA_ENV=OFF" python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check
mkdir -p ${package_dir}/final_dist
python -m auditwheel repair -w ${package_dir}/final_dist ${package_dir}/dist/*

mkdir -p final_dist
python -m auditwheel repair -w final_dist dist/*

RAPIDS_PY_WHEEL_NAME="cudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 final_dist
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
RAPIDS_PY_WHEEL_NAME="cudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 ${package_dir}/final_dist
19 changes: 3 additions & 16 deletions ci/build_wheel_dask_cudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,9 @@

set -euo pipefail

source rapids-configure-sccache
source rapids-date-string
package_dir="python/dask_cudf"

# Use gha-tools rapids-pip-wheel-version to generate wheel version then
# update the necessary files
version_override="$(rapids-pip-wheel-version ${RAPIDS_DATE_STRING})"
./ci/build_wheel.sh dask_cudf ${package_dir}

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

ci/release/apply_wheel_modifications.sh ${version_override} "-${RAPIDS_PY_CUDA_SUFFIX}"
echo "The package name and/or version was modified in the package source. The git diff is:"
git diff

cd python/dask_cudf

# Hardcode the output dir
python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check

RAPIDS_PY_WHEEL_NAME="dask_cudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 dist
RAPIDS_PY_WHEEL_NAME="dask_cudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 ${package_dir}/dist
32 changes: 0 additions & 32 deletions ci/release/apply_wheel_modifications.sh
View file
Open in desktop

This file was deleted.