Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add rapids-get-pr-wheel-artifact for quickly downloading cpp wheels #101

Merged
merged 8 commits into from
Apr 16, 2024
Merged
10 changes: 9 additions & 1 deletion tools/rapids-download-wheels-from-s3
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ set -eo pipefail

source rapids-prompt-local-repo-config

pkg_name="$(rapids-package-name wheel_python)"
# For legacy reasons, allow this script to be run without the pkg_type being the first arg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option that would probably be more consistent with other scripts in this repo would be to use an environment variable. I'm not the biggest fan of how much we've proliferated those, and maybe what we really need to start doing is introducing some proper argument parsing (e.g. via getopts), but that's out of scope for this PR (but curious what others like @bdice and @ajschmidt8 think about that).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with the code here. We can have a follow-up to clean up all uses of the script so they declare the pkg_type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest I'm hoping we can remove a lot of these scripts entirely pending the outcome of https://github.com/rapidsai/ops/issues/2982.

If that issue ends up being rolled out, I think that would be a good time to rethink some of the interfaces for any of the helper scripts that we do keep around.

pkg_name="$(rapids-package-name "wheel_python")"

pkg_type="$1"
if [ "${pkg_type}" = "cpp" ] || [ "${pkg_type}" = "python" ]; then
# remove pkg_type from args because we handle it in this script
shift;
pkg_name="$(rapids-package-name "wheel_${pkg_type}")"
fi

rapids-download-from-s3 "${pkg_name}" "$@"
36 changes: 36 additions & 0 deletions tools/rapids-get-pr-wheel-artifact
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash
# Echo path to an artifact for a specific PR. Finds and uses the latest commit on the PR.
#
# Positional Arguments:
# 1) repo name
# 2) PR number
# 3) "cpp" or "python", to get the artifact for the C++ or Python build, respectively
# 4) [optional] commit hash, to get the artifact for a specific commit
#
# Example Usage:
# rapids-get-pr-conda-artifact rmm 1095 cpp
set -euo pipefail

repo="$1"
pr="$2"

pkg_type="$3"
case "${pkg_type}" in
cpp)
artifact_name=$(RAPIDS_REPOSITORY=$repo rapids-package-name wheel_cpp)
;;
python)
artifact_name=$(RAPIDS_REPOSITORY=$repo rapids-package-name wheel_python)
;;
*)
echo "Error: 3rd argument must be 'cpp' or 'python'"
exit 1
;;
esac

commit="${4:-}"
if [[ -z "${commit}" ]]; then
commit=$(git ls-remote https://github.com/rapidsai/"${repo}".git refs/heads/pull-request/"${pr}" | cut -c1-7)
msarahan marked this conversation as resolved.
Show resolved Hide resolved
fi

rapids-get-artifact "ci/${repo}/pull-request/${pr}/${commit}/${artifact_name}"
5 changes: 5 additions & 0 deletions tools/rapids-package-name
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,11 @@ case "${pkg_type}" in
append_pyver=1
append_arch=1
;;
wheel_cpp)
append_wheelname=1
append_cuda=1
msarahan marked this conversation as resolved.
Show resolved Hide resolved
append_arch=1
;;
wheel_python)
append_wheelname=1
# Pure wheels do not need a pyver or arch
Expand Down
12 changes: 10 additions & 2 deletions tools/rapids-upload-wheels-to-s3
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,20 @@
# 1) wheel path to tar up
set -e

# For legacy reasons, allow this script to be run without the pkg_type being the first arg
pkg_name="$(rapids-package-name "wheel_python")"

pkg_type="$1"
if [ "${pkg_type}" = "cpp" ] || [ "${pkg_type}" = "python" ]; then
# remove pkg_type from args because we handle it in this script
shift;
pkg_name="$(rapids-package-name "wheel_${pkg_type}")"
fi

if [ "${CI:-false}" = "false" ]; then
rapids-echo-stderr "Packages from local builds cannot be uploaded to S3."
rapids-echo-stderr "Open a PR to have successful builds uploaded."
exit 0
fi

pkg_name="$(rapids-package-name wheel_python)"

rapids-upload-to-s3 "${pkg_name}" "$@"