Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build C++ wheel #4340

Closed
wants to merge 67 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
50b83c8
Get libcugraph building
vyasr Mar 31, 2024
fdbaf78
Add entry point
vyasr Mar 31, 2024
a881784
Update dependencies.yaml
vyasr Mar 31, 2024
e1ef215
Build pylibcugraph wheel against libcugraph wheel
vyasr Mar 31, 2024
a9789db
Add missing cupy dep
vyasr Mar 31, 2024
990c3eb
Fix cugraph CMake to properly support using dynamic raft with static …
vyasr Mar 31, 2024
0b0e3a9
Switch from RPATH hacking to dynamic loading with ctypes
vyasr Apr 1, 2024
a6c9cb1
Also load libcugraph_c
vyasr Apr 1, 2024
4ac08db
Try fixing non-relocatable bundling of raft headers
vyasr Apr 1, 2024
396e7b3
Enable GHA
vyasr Apr 1, 2024
2c6617e
Fix PR number for raft
vyasr Apr 1, 2024
24c6c4c
Fix py variables
vyasr Apr 1, 2024
8ca6820
Use known raft commit
vyasr Apr 1, 2024
ddf474b
Fix variable name
vyasr Apr 1, 2024
eaafcc6
Turn off pure setting
vyasr Apr 1, 2024
1e274c9
Use space rather than semicolon-separated find links
vyasr Apr 1, 2024
a0ba13b
Audit the C++ wheel
vyasr Apr 2, 2024
14ed178
Remove legacy code
vyasr Apr 2, 2024
1d40482
Pull all artifacts needed to build wheels
vyasr Apr 2, 2024
84ad199
Make wheel py3
vyasr Apr 2, 2024
618ad3c
Clean up commit pulls
vyasr Apr 2, 2024
3fd2377
Make sure to update the libcugraph suffix
vyasr Apr 2, 2024
587d6f4
Download all requirements in test job
vyasr Apr 2, 2024
47ef4a3
Add PIP_FIND_LINKS
vyasr Apr 2, 2024
11db8d5
Update to use latest tools
vyasr Apr 3, 2024
b6bdb38
Fix typo
vyasr Apr 3, 2024
e4c026f
Add excludes to auditwheel
vyasr Apr 3, 2024
0f7cbb1
Fix one more typo
vyasr Apr 3, 2024
175c21b
Reenable cugraph-ops
vyasr Apr 4, 2024
f49c8b6
Point CMake to the cugraph-ops clone
vyasr Apr 4, 2024
57a0d43
Use libcugraphops wheel
vyasr Apr 4, 2024
eed9f85
Note that we don't need an extra repo anymore
vyasr Apr 4, 2024
cced123
Download libcugraphops wheel
vyasr Apr 4, 2024
8c650ff
Stop cloning extra repo
vyasr Apr 4, 2024
8c8c7c4
Specify the commit so that we don't need to ls the remote
vyasr Apr 4, 2024
ec4fb45
Also exclude libcugraph-ops from auditwheel
vyasr Apr 4, 2024
d044445
Also download libcugraphops when running tests
vyasr Apr 4, 2024
a80132f
Use new rapids-cmake to hide necessary CCCL symbols
vyasr Apr 9, 2024
d4c3742
Some cleanup
vyasr Apr 12, 2024
1a24565
Try restructuring CI
vyasr Apr 12, 2024
703a99a
Apply suggestions from code review
vyasr Apr 12, 2024
d986c1c
Apply suggestions from code review
vyasr Apr 12, 2024
ab1fc04
Update ci/test_wheel.sh
vyasr Apr 12, 2024
7bd2b44
Add exec perms
vyasr Apr 12, 2024
f3051c3
Update ci/build_wheel_python.sh
vyasr Apr 12, 2024
9449626
Merge remote-tracking branch 'upstream/branch-24.06' into feat/cpp_wheel
vyasr Apr 16, 2024
2baffb7
Delete now superfluous build scripts
vyasr Apr 16, 2024
7bcfb8b
Update remaining testing scripts
vyasr Apr 17, 2024
449e812
Remove unused build jobs
vyasr Apr 17, 2024
f41a3c6
Some cleanup
vyasr Apr 17, 2024
857c495
Apply suggestions from code review
vyasr Apr 17, 2024
6f47f56
Remove some more superfluous clones
vyasr Apr 17, 2024
be0896b
Improve handling of wheelhouses, grouping packages, etc
vyasr Apr 17, 2024
121c23c
Update cugraph-ops hash
vyasr Apr 17, 2024
bc6f85e
Fix hash again
vyasr Apr 17, 2024
3436e51
Fix typo
vyasr Apr 17, 2024
12c9fc0
Update test scripts
vyasr Apr 17, 2024
cd8c8b0
Make VERSION file a symlink
vyasr Apr 18, 2024
57255b3
Add missing alpha specs to installs
vyasr Apr 18, 2024
9c169f3
Remove incorrect package names
vyasr Apr 18, 2024
779a0b8
Fix cugraph-equivariant tests
vyasr Apr 18, 2024
18ef0e0
Use a consistent raft commit
vyasr Apr 18, 2024
8272184
Merge branch 'branch-24.06' into feat/cpp_wheel
nv-rliu May 13, 2024
e3af2c8
Merge branch 'branch-24.08' of github.com:rapidsai/cugraph into feat/…
msarahan Jun 26, 2024
fef5081
update RAPIDS dependency versions to 24.8
msarahan Jun 26, 2024
0da5611
remove rmm pr wheel artifact downloads
msarahan Jun 28, 2024
bd6015f
Merge branch 'branch-24.08' of github.com:rapidsai/cugraph into feat/…
msarahan Jun 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Try restructuring CI
vyasr committed Apr 12, 2024

Verified

This commit was signed with the committer’s verified signature.
vyasr Vyas Ramasubramani
commit 1a24565dd99ed1926656db648c02f1aa54748b23
96 changes: 48 additions & 48 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
@@ -21,17 +21,17 @@ jobs:
- conda-python-tests
- docs-build
- wheel-build-cpp
- wheel-build-pylibcugraph
- wheel-build-python
- wheel-tests-pylibcugraph
- wheel-build-cugraph
#- wheel-build-cugraph
- wheel-tests-cugraph
- wheel-build-nx-cugraph
#- wheel-build-nx-cugraph
- wheel-tests-nx-cugraph
- wheel-build-cugraph-dgl
#- wheel-build-cugraph-dgl
- wheel-tests-cugraph-dgl
- wheel-build-cugraph-pyg
#- wheel-build-cugraph-pyg
- wheel-tests-cugraph-pyg
- wheel-build-cugraph-equivariant
#- wheel-build-cugraph-equivariant
- wheel-tests-cugraph-equivariant
- devcontainer
secrets: inherit
@@ -103,87 +103,87 @@ jobs:
build_type: pull-request
script: ci/build_wheel_cpp.sh
node_type: cpu32
wheel-build-pylibcugraph:
wheel-build-python:
needs: wheel-build-cpp
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/build_wheel_pylibcugraph.sh
script: ci/build_wheel_python.sh
node_type: cpu32
wheel-tests-pylibcugraph:
needs: wheel-build-pylibcugraph
needs: wheel-build-python
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/test_wheel_pylibcugraph.sh
wheel-build-cugraph:
needs: wheel-build-pylibcugraph
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/build_wheel_cugraph.sh
#wheel-build-cugraph:
# needs: wheel-build-pylibcugraph
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/[email protected]
# with:
# build_type: pull-request
# script: ci/build_wheel_cugraph.sh
wheel-tests-cugraph:
needs: wheel-build-cugraph
needs: wheel-build-python
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/test_wheel_cugraph.sh
wheel-build-nx-cugraph:
needs: wheel-tests-pylibcugraph
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/build_wheel_nx-cugraph.sh
#wheel-build-nx-cugraph:
# needs: wheel-tests-pylibcugraph
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/[email protected]
# with:
# build_type: pull-request
# script: ci/build_wheel_nx-cugraph.sh
wheel-tests-nx-cugraph:
needs: wheel-build-nx-cugraph
needs: wheel-build-python
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/test_wheel_nx-cugraph.sh
wheel-build-cugraph-dgl:
needs: wheel-tests-cugraph
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/build_wheel_cugraph-dgl.sh
#wheel-build-cugraph-dgl:
# needs: wheel-tests-cugraph
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/[email protected]
# with:
# build_type: pull-request
# script: ci/build_wheel_cugraph-dgl.sh
wheel-tests-cugraph-dgl:
needs: wheel-build-cugraph-dgl
needs: wheel-build-python
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/test_wheel_cugraph-dgl.sh
matrix_filter: map(select(.ARCH == "amd64"))
wheel-build-cugraph-pyg:
needs: wheel-tests-cugraph
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/build_wheel_cugraph-pyg.sh
#wheel-build-cugraph-pyg:
# needs: wheel-tests-cugraph
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/[email protected]
# with:
# build_type: pull-request
# script: ci/build_wheel_cugraph-pyg.sh
wheel-tests-cugraph-pyg:
needs: wheel-build-cugraph-pyg
needs: wheel-build-python
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/test_wheel_cugraph-pyg.sh
matrix_filter: map(select(.ARCH == "amd64"))
wheel-build-cugraph-equivariant:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: pull-request
script: ci/build_wheel_cugraph-equivariant.sh
#wheel-build-cugraph-equivariant:
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/[email protected]
# with:
# build_type: pull-request
# script: ci/build_wheel_cugraph-equivariant.sh
wheel-tests-cugraph-equivariant:
needs: wheel-build-cugraph-equivariant
needs: wheel-build-python
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
102 changes: 102 additions & 0 deletions ci/build_wheel_python.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#!/bin/bash
# Copyright (c) 2023-2024, NVIDIA CORPORATION.

set -euo pipefail

if [[ ! -d "/tmp/gha-tools" ]]; then
git clone https://github.com/msarahan/gha-tools.git -b get-pr-wheel-artifact /tmp/gha-tools
fi
export PATH="/tmp/gha-tools/tools:${PATH}"

source rapids-configure-sccache
source rapids-date-string

version=$(rapids-generate-version)
git_commit=$(git rev-parse HEAD)

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp /tmp/libcugraph_dist

librmm_wheelhouse=$(RAPIDS_PY_WHEEL_NAME="librmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact rmm 1529 cpp)
libraft_wheelhouse=$(RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2264 cpp)
libcugraphops_wheelhouse=$(RAPIDS_PY_WHEEL_NAME="libcugraphops_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact cugraph-ops 629 cpp a3ac1f3)
vyasr marked this conversation as resolved.
Show resolved Hide resolved
pylibraft_wheelhouse=$(RAPIDS_PY_WHEEL_NAME="pylibraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2264 python)

wheelhouses=("${librmm_wheelhouse}" "${libraft_wheelhouse}" "${libcugraphops_wheelhouse}" "${pylibraft_wheelhouse}")
vyasr marked this conversation as resolved.
Show resolved Hide resolved

# This is the version of the suffix with a preceding hyphen. It's used
# everywhere except in the final wheel name.
PACKAGE_CUDA_SUFFIX="-${RAPIDS_PY_CUDA_SUFFIX}"

# For nightlies we want to ensure that we're pulling in alphas as well. The
# easiest way to do so is to augment the spec with a constraint containing a
# min alpha version that doesn't affect the version bounds but does allow usage
# of alpha versions for that dependency without --pre
alpha_spec=''
if ! rapids-is-release-build; then
alpha_spec=',>=0.0.0a0'
fi

build_wheel () {
local package_name="${1}"
local package_dir="${2}"
local underscore_package_name=$(echo "${package_name}" | tr "-" "_")
local version_package_name="$underscore_package_name"
if [[ "${version_package_name}" = "nx_cugraph" ]]; then
version_package_name="_nx_cugraph"
fi

local package_dir="python/${package_name}"
local pyproject_file="${package_dir}/pyproject.toml"
local version_file="${package_dir}/${version_package_name}/_version.py"

sed -i "s/name = \"${package_name}\"/name = \"${package_name}${PACKAGE_CUDA_SUFFIX}\"/g" ${pyproject_file}
echo "${version}" > VERSION
sed -i "/^__git_commit__ / s/= .*/= \"${git_commit}\"/g" ${version_file}

for dep in rmm cudf cugraph libcugraph raft-dask pylibcugraph pylibcugraphops pylibraft ucx-py; do
sed -r -i "s/${dep}==(.*)\"/${dep}${PACKAGE_CUDA_SUFFIX}==\1${alpha_spec}\"/g" ${pyproject_file}
done

# dask-cuda & rapids-dask-dependency doesn't get a suffix, but it does get an alpha spec.
for dep in dask-cuda rapids-dask-dependency; do
sed -r -i "s/${dep}==(.*)\"/${dep}==\1${alpha_spec}\"/g" ${pyproject_file}
done

if [[ $PACKAGE_CUDA_SUFFIX == "-cu12" ]]; then
sed -i "s/cupy-cuda11x/cupy-cuda12x/g" ${pyproject_file}
fi

pushd "${package_dir}"

local find_links=""
# Iterate over the array
for wheelhouse in "${wheelhouses[@]}"; do
find_links+="--find-links ${wheelhouse} "
done

python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check ${find_links}

# pure-python packages should be marked as pure, and not have auditwheel run on them.
if [[ ${package_name} == "nx-cugraph" ]] || \
[[ ${package_name} == "cugraph-dgl" ]] || \
[[ ${package_name} == "cugraph-pyg" ]] || \
[[ ${package_name} == "cugraph-equivariant" ]]; then
RAPIDS_PY_WHEEL_NAME="${package_name}_${RAPIDS_PY_CUDA_SUFFIX}" RAPIDS_PY_WHEEL_PURE="1" rapids-upload-wheels-to-s3 python dist
else
mkdir -p final_dist
python -m auditwheel repair -w final_dist --exclude libcugraph.so --exclude libcugraph_c.so --exclude libraft.so --exclude libcugraph-ops++.so dist/*
RAPIDS_PY_WHEEL_NAME="${package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 python final_dist
fi

# Append ${PWD}/final_dist to the list of wheelhouses for the next package
wheelhouses+=("${PWD}/final_dist")
popd
}

build_wheel pylibcugraph python/pylibcugraph
build_wheel cugraph python/cugraph
build_wheel nx-cugraph python/nx-cugraph
build_wheel cugraph-dgl python/cugraph-dgl
build_wheel cugraph-pyg python/cugraph-pyg
build_wheel cugraph-equivariant python/cugraph-equivariant