Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downgrade to Arrow 12.0.0 for aws-sdk-cpp and fix cudf_kafka builds for new CI containers #14296

Merged
merged 18 commits into from
Oct 19, 2023
Merged

Downgrade to Arrow 12.0.0 for aws-sdk-cpp and fix cudf_kafka builds for new CI containers #14296

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
3e8be1c
Temporarily add hard pins of a specific arrow build
vyasr Oct 18, 2023
985313f
Fix pinnings
vyasr Oct 18, 2023
2d1b406
Fix recipe
vyasr Oct 18, 2023
ff8a5b1
Fix build version
vyasr Oct 18, 2023
f98c903
Also fix dependencies.yaml
vyasr Oct 18, 2023
116c82c
Run dfg
vyasr Oct 18, 2023
c4e1a82
Specify libarrow dependency in the docs env as well
vyasr Oct 18, 2023
415bfbe
Add cuda-gdb to CUDA 12 requirements for cudf-kafka
vyasr Oct 18, 2023
b7431c3
Merge remote-tracking branch 'origin/branch-23.12' into fix/arrow_aws…
vyasr Oct 18, 2023
fb5e921
Make sure cuda.h is installed and patch path during conda build
vyasr Oct 19, 2023
240362b
Add some comments and clean up
vyasr Oct 19, 2023
05ece0a
Fix copyright
vyasr Oct 19, 2023
7691e0c
Merge branch 'branch-23.12' into fix/arrow_aws_pinning
vyasr Oct 19, 2023
f5a4e63
Only set CUDA_HOME for CUDA 12 builds
vyasr Oct 19, 2023
0b5bab4
Put back build command...
vyasr Oct 19, 2023
68818b2
Make sure RAPIDS_CUDA_VERSION is available in the build environment
vyasr Oct 19, 2023
ff6fe4a
Downgrade to arrow 12.0.0
vyasr Oct 19, 2023
0ffc3e7
Add one more comment
vyasr Oct 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions conda/environments/all_cuda-118_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ dependencies:
- hypothesis
- identify>=2.5.20
- ipython
- libarrow==12.0.1.*
- libarrow==12.0.0.*
- libcufile-dev=1.4.0.31
- libcufile=1.4.0.31
- libcurand-dev=10.3.0.86
Expand Down Expand Up @@ -69,7 +69,7 @@ dependencies:
- pre-commit
- protobuf>=4.21,<5
- ptxcompiler
- pyarrow==12.0.1.*
- pyarrow==12.0.0.*
- pydata-sphinx-theme
- pyorc
- pytest
Expand Down
5 changes: 3 additions & 2 deletions conda/environments/all_cuda-120_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ dependencies:
- cachetools
- cmake>=3.26.4
- cuda-cudart-dev
- cuda-gdb
- cuda-nvcc
- cuda-nvrtc-dev
- cuda-nvtx-dev
Expand All @@ -41,7 +42,7 @@ dependencies:
- hypothesis
- identify>=2.5.20
- ipython
- libarrow==12.0.1.*
- libarrow==12.0.0.*
- libcufile-dev
- libcurand-dev
- libkvikio==23.12.*
Expand All @@ -66,7 +67,7 @@ dependencies:
- pip
- pre-commit
- protobuf>=4.21,<5
- pyarrow==12.0.1.*
- pyarrow==12.0.0.*
- pydata-sphinx-theme
- pyorc
- pytest
Expand Down
2 changes: 1 addition & 1 deletion conda/recipes/cudf/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ requirements:
- scikit-build >=0.13.1
- setuptools
- dlpack >=0.5,<0.6.0a0
- pyarrow =12
- pyarrow =12.0.0
- libcudf ={{ version }}
- rmm ={{ minor_version }}
{% if cuda_major == "11" %}
Expand Down
14 changes: 13 additions & 1 deletion conda/recipes/cudf_kafka/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,16 @@
# Copyright (c) 2020-2022, NVIDIA CORPORATION.
# Copyright (c) 2020-2023, NVIDIA CORPORATION.

# This assumes the script is executed from the root of the repo directory
# Need to set CUDA_HOME inside conda environments because the hacked together
# setup.py for cudf-kafka searches that way.
# TODO: Remove after https://github.com/rapidsai/cudf/pull/14292 updates
# cudf_kafka to use scikit-build
CUDA_MAJOR=${RAPIDS_CUDA_VERSION%%.*}
if [[ ${CUDA_MAJOR} == "12" ]]; then
target_name="x86_64-linux"
if [[ ! $(arch) == "x86_64" ]]; then
target_name="sbsa-linux"
fi
export CUDA_HOME="${PREFIX}/targets/${target_name}/"
fi
./build.sh -v cudf_kafka
11 changes: 11 additions & 0 deletions conda/recipes/cudf_kafka/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ build:
- SCCACHE_S3_KEY_PREFIX=cudf-kafka-linux64 # [linux64]
- SCCACHE_S3_USE_SSL
- SCCACHE_S3_NO_CREDENTIALS
# TODO: Remove after https://github.com/rapidsai/cudf/pull/14292 updates
# cudf_kafka to use scikit-build
- RAPIDS_CUDA_VERSION

requirements:
build:
Expand All @@ -41,13 +44,21 @@ requirements:
- {{ compiler('cxx') }}
- ninja
- sysroot_{{ target_platform }} {{ sysroot_version }}
# TODO: Remove after https://github.com/rapidsai/cudf/pull/14292 updates
# cudf_kafka to use scikit-build
{% if cuda_major == "12" %}
- cuda-gdb
vyasr marked this conversation as resolved.
Show resolved Hide resolved
vyasr marked this conversation as resolved.
Show resolved Hide resolved
vyasr marked this conversation as resolved.
Show resolved Hide resolved
{% endif %}
host:
- python
- cython >=3.0.0
- cuda-version ={{ cuda_version }}
- cudf ={{ version }}
- libcudf_kafka ={{ version }}
- setuptools
{% if cuda_major == "12" %}
- cuda-cudart-dev
{% endif %}
run:
- python
- {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }}
Expand Down
4 changes: 2 additions & 2 deletions conda/recipes/libcudf/conda_build_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ gtest_version:
aws_sdk_cpp_version:
- "<1.11"

libarrow_version:
- "=12"
libarrow:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it intentional to drop the _version suffix?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No 🙁 good catch. But now that it's done and we have CI passing, I'm inclined to merge and fix in a follow-up to unblock CI.

- "==12.0.0"

dlpack_version:
- ">=0.5,<0.6.0a0"
Expand Down
4 changes: 2 additions & 2 deletions conda/recipes/libcudf/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ requirements:
{% endif %}
- cuda-version ={{ cuda_version }}
- nvcomp {{ nvcomp_version }}
- libarrow {{ libarrow_version }}
- libarrow {{ libarrow }}
- dlpack {{ dlpack_version }}
- librdkafka {{ librdkafka_version }}
- fmt {{ fmt_version }}
Expand Down Expand Up @@ -104,7 +104,7 @@ outputs:
- nvcomp {{ nvcomp_version }}
- librmm ={{ minor_version }}
- libkvikio ={{ minor_version }}
- libarrow {{ libarrow_version }}
- libarrow {{ libarrow }}
- dlpack {{ dlpack_version }}
- gtest {{ gtest_version }}
- gmock {{ gtest_version }}
Expand Down
2 changes: 1 addition & 1 deletion cpp/cmake/thirdparty/get_arrow.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ if(NOT DEFINED CUDF_VERSION_Arrow)
set(CUDF_VERSION_Arrow
# This version must be kept in sync with the libarrow version pinned for builds in
# dependencies.yaml.
12.0.1
12.0.0
CACHE STRING "The version of Arrow to find (or build)"
)
endif()
Expand Down
16 changes: 9 additions & 7 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ files:
includes:
- cudatoolkit
- docs
- libarrow_run
- py_version
py_build_cudf:
output: pyproject
Expand Down Expand Up @@ -225,7 +226,7 @@ dependencies:
- &gmock gmock>=1.13.0
# Hard pin the patch version used during the build. This must be kept
# in sync with the version pinned in get_arrow.cmake.
- libarrow==12.0.1.*
- &libarrow libarrow==12.0.0.*
- librdkafka>=1.9.0,<1.10.0a0
# Align nvcomp version with rapids-cmake
- nvcomp==2.6.1
Expand All @@ -243,7 +244,7 @@ dependencies:
- cython>=3.0.0
# Hard pin the patch version used during the build. This must be kept
# in sync with the version pinned in get_arrow.cmake.
- pyarrow==12.0.1.*
- &pyarrow pyarrow==12.0.0.*
# TODO: Pin to numpy<1.25 until cudf requires pandas 2
- &numpy numpy>=1.21,<1.25
build_python:
Expand All @@ -260,16 +261,14 @@ dependencies:
- protoc-wheel
libarrow_run:
common:
- output_types: [conda, requirements]
- output_types: conda
packages:
# Allow runtime version to float up to minor version
- libarrow==12.*
- *libarrow
pyarrow_run:
common:
- output_types: [conda, requirements, pyproject]
packages:
# Allow runtime version to float up to minor version
- pyarrow==12.*
- *pyarrow
cudatoolkit:
specific:
- output_types: conda
Expand All @@ -282,6 +281,9 @@ dependencies:
- cuda-nvrtc-dev
- cuda-nvtx-dev
- libcurand-dev
# TODO: Remove after https://github.com/rapidsai/cudf/pull/14292 updates
# cudf_kafka to use scikit-build
- cuda-gdb
vyasr marked this conversation as resolved.
Show resolved Hide resolved
vyasr marked this conversation as resolved.
Show resolved Hide resolved
- matrix:
cuda: "11.8"
packages:
Expand Down
4 changes: 2 additions & 2 deletions python/cudf/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ requires = [
"ninja",
"numpy>=1.21,<1.25",
"protoc-wheel",
"pyarrow==12.0.1.*",
"pyarrow==12.0.0.*",
"rmm==23.12.*",
"scikit-build>=0.13.1",
"setuptools",
Expand Down Expand Up @@ -38,7 +38,7 @@ dependencies = [
"pandas>=1.3,<1.6.0dev0",
"protobuf>=4.21,<5",
"ptxcompiler",
"pyarrow==12.*",
"pyarrow==12.0.0.*",
"rmm==23.12.*",
"typing_extensions>=4.0.0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
Expand Down
2 changes: 1 addition & 1 deletion python/cudf_kafka/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
requires = [
"cython>=3.0.0",
"numpy>=1.21,<1.25",
"pyarrow==12.0.1.*",
"pyarrow==12.0.0.*",
"setuptools",
"wheel",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
Expand Down