-
Notifications
You must be signed in to change notification settings - Fork 915
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix issues with building cudf in a non-conda environment (#7647)
Extension to rapidsai/rmm#733 Also resolves: #7578 This PR introduces fixes to `setup.py` in cudf build and also adds a DockerFile with which one can create a container with cudf fully built from source. Arrow & rmm will also be built from the source. This PR also contains a `requirements.txt` which has packages needed by cudf and that are available to install via pip. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Devavret Makkar (https://github.com/devavret) - Keith Kraus (https://github.com/kkraus14) - Christopher Harris (https://github.com/cwharris) - AJ Schmidt (https://github.com/ajschmidt8) URL: #7647
- Loading branch information
1 parent
b9415ca
commit f4d5bde
Showing
26 changed files
with
747 additions
and
28 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Copyright (c) 2021, NVIDIA CORPORATION. | ||
|
|
||
| name: cudf_dev | ||
| channels: | ||
| - rapidsai | ||
| - nvidia | ||
| - rapidsai-nightly | ||
| - conda-forge | ||
| - defaults | ||
| dependencies: | ||
| - clang=8.0.1 | ||
| - clang-tools=8.0.1 | ||
| - cupy>7.1.0,<9.0.0a0 | ||
| - rmm=0.19.* | ||
| - cmake>=3.14 | ||
| - cmake_setuptools>=0.1.3 | ||
| - python>=3.7,<3.9 | ||
| - numba>=0.49,!=0.51.0 | ||
| - numpy | ||
| - pandas>=1.0,<1.3.0dev0 | ||
| - pyarrow=1.0.1 | ||
| - fastavro>=0.22.9 | ||
| - notebook>=0.5.0 | ||
| - cython>=0.29,<0.30 | ||
| - fsspec>=0.6.0 | ||
| - pytest | ||
| - pytest-benchmark | ||
| - pytest-xdist | ||
| - sphinx | ||
| - sphinx_rtd_theme | ||
| - sphinxcontrib-websupport | ||
| - nbsphinx | ||
| - numpydoc | ||
| - ipython | ||
| - recommonmark | ||
| - pandoc=<2.0.0 | ||
| - cudatoolkit=11.1 | ||
| - pip | ||
| - flake8=3.8.3 | ||
| - black=19.10 | ||
| - isort=5.0.7 | ||
| - mypy=0.782 | ||
| - typing_extensions | ||
| - pre_commit | ||
| - dask>=2021.3.1 | ||
| - distributed>=2.22.0 | ||
| - streamz | ||
| - dlpack | ||
| - arrow-cpp=1.0.1 | ||
| - arrow-cpp-proc * cuda | ||
| - boost-cpp>=1.72.0 | ||
| - double-conversion | ||
| - rapidjson | ||
| - flatbuffers | ||
| - hypothesis | ||
| - sphinx-markdown-tables | ||
| - sphinx-copybutton | ||
| - mimesis | ||
| - packaging | ||
| - protobuf | ||
| - nvtx>=0.2.1 | ||
| - cachetools | ||
| - pip: | ||
| - git+https://github.com/dask/dask.git@main | ||
| - git+https://github.com/dask/distributed.git@main | ||
| - git+https://github.com/python-streamz/streamz.git | ||
| - pyorc |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Copyright (c) 2021, NVIDIA CORPORATION. | ||
|
|
||
| name: cudf_dev | ||
| channels: | ||
| - rapidsai | ||
| - nvidia | ||
| - rapidsai-nightly | ||
| - conda-forge | ||
| - defaults | ||
| dependencies: | ||
| - clang=8.0.1 | ||
| - clang-tools=8.0.1 | ||
| - cupy>7.1.0,<9.0.0a0 | ||
| - rmm=0.19.* | ||
| - cmake>=3.14 | ||
| - cmake_setuptools>=0.1.3 | ||
| - python>=3.7,<3.9 | ||
| - numba>=0.49,!=0.51.0 | ||
| - numpy | ||
| - pandas>=1.0,<1.3.0dev0 | ||
| - pyarrow=1.0.1 | ||
| - fastavro>=0.22.9 | ||
| - notebook>=0.5.0 | ||
| - cython>=0.29,<0.30 | ||
| - fsspec>=0.6.0 | ||
| - pytest | ||
| - pytest-benchmark | ||
| - pytest-xdist | ||
| - sphinx | ||
| - sphinx_rtd_theme | ||
| - sphinxcontrib-websupport | ||
| - nbsphinx | ||
| - numpydoc | ||
| - ipython | ||
| - recommonmark | ||
| - pandoc=<2.0.0 | ||
| - cudatoolkit=11.2 | ||
| - pip | ||
| - flake8=3.8.3 | ||
| - black=19.10 | ||
| - isort=5.0.7 | ||
| - mypy=0.782 | ||
| - typing_extensions | ||
| - pre_commit | ||
| - dask>=2021.3.1 | ||
| - distributed>=2.22.0 | ||
| - streamz | ||
| - dlpack | ||
| - arrow-cpp=1.0.1 | ||
| - arrow-cpp-proc * cuda | ||
| - boost-cpp>=1.72.0 | ||
| - double-conversion | ||
| - rapidjson | ||
| - flatbuffers | ||
| - hypothesis | ||
| - sphinx-markdown-tables | ||
| - sphinx-copybutton | ||
| - mimesis | ||
| - packaging | ||
| - protobuf | ||
| - nvtx>=0.2.1 | ||
| - cachetools | ||
| - pip: | ||
| - git+https://github.com/dask/dask.git@main | ||
| - git+https://github.com/dask/distributed.git@main | ||
| - git+https://github.com/python-streamz/streamz.git | ||
| - pyorc |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # Copyright (c) 2021, NVIDIA CORPORATION. | ||
|
|
||
| ARG CUDA_VERSION=11.2.2 | ||
| FROM nvidia/cuda:${CUDA_VERSION}-devel | ||
| ENV CUDA_SHORT_VERSION=11.2 | ||
|
|
||
| SHELL ["/bin/bash", "-c"] | ||
| ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/lib:/repos/dist/lib | ||
|
|
||
| ENV DEBIAN_FRONTEND=noninteractive | ||
|
|
||
| ENV CUDA_HOME=/usr/local/cuda | ||
| ENV CUDA_PATH=$CUDA_HOME | ||
| ENV PATH=${CUDA_HOME}/lib64/:${PATH}:${CUDA_HOME}/bin | ||
|
|
||
| # Build env variables for arrow | ||
| ENV CMAKE_BUILD_TYPE=release | ||
| ENV PYARROW_WITH_PARQUET=1 | ||
| ENV PYARROW_WITH_CUDA=1 | ||
| ENV PYARROW_WITH_ORC=1 | ||
| ENV PYARROW_WITH_DATASET=1 | ||
|
|
||
| ENV ARROW_HOME=/repos/dist | ||
|
|
||
| # Build env variables for rmm | ||
| ENV INSTALL_PREFIX=/usr | ||
|
|
||
|
|
||
| RUN apt update -y --fix-missing && \ | ||
| apt upgrade -y && \ | ||
| apt install -y --no-install-recommends software-properties-common && \ | ||
| add-apt-repository ppa:deadsnakes/ppa && \ | ||
| apt update -y --fix-missing | ||
|
|
||
| RUN apt install -y --no-install-recommends \ | ||
| git \ | ||
| libboost-all-dev \ | ||
| python3.8-dev \ | ||
| build-essential \ | ||
| autoconf \ | ||
| bison \ | ||
| flex \ | ||
| libboost-filesystem-dev \ | ||
| libboost-system-dev \ | ||
| libboost-regex-dev \ | ||
| libjemalloc-dev \ | ||
| wget \ | ||
| libssl-dev \ | ||
| protobuf-compiler && \ | ||
| apt-get autoremove -y && \ | ||
| apt-get clean && \ | ||
| rm -rf /var/lib/apt/lists/* && \ | ||
| update-alternatives --install /usr/bin/python python /usr/bin/python3.8 1 && \ | ||
| wget https://bootstrap.pypa.io/get-pip.py && \ | ||
| python get-pip.py | ||
|
|
||
| # Install cmake | ||
| RUN version=3.18 && build=5 && mkdir ~/temp && cd ~/temp && wget https://cmake.org/files/v$version/cmake-$version.$build.tar.gz && \ | ||
| tar -xzvf cmake-$version.$build.tar.gz && cd cmake-$version.$build/ && ./bootstrap && make -j$(nproc) && make install | ||
|
|
||
| # Install arrow from source | ||
| RUN git clone https://github.com/apache/arrow.git /repos/arrow && mkdir /repos/dist/ && cd /repos/arrow && git checkout apache-arrow-1.0.1 && git submodule init && \ | ||
| git submodule update && export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data" && export ARROW_TEST_DATA="${PWD}/testing/data" && \ | ||
| cd /repos/arrow/cpp && mkdir release && cd /repos/arrow/cpp/release && pip install -r /repos/arrow/python/requirements-build.txt && \ | ||
| cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib -DARROW_FLIGHT=ON -DARROW_GANDIVA=OFF -DARROW_ORC=ON -DARROW_WITH_BZ2=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_BROTLI=ON -DARROW_PARQUET=ON -DARROW_PYTHON=ON -DARROW_PLASMA=ON -DARROW_BUILD_TESTS=ON -DARROW_CUDA=ON -DARROW_DATASET=ON .. && \ | ||
| make -j$(nproc) && make install && cd /repos/arrow/python/ && python setup.py build_ext --build-type=release bdist_wheel && pip install /repos/arrow/python/dist/*.whl | ||
|
|
||
|
|
||
| # Install rmm from source | ||
| RUN cd /repos/ && git clone https://github.com/rapidsai/rmm.git && cd /repos/rmm/ && ./build.sh librmm && pip install /repos/rmm/python/. | ||
|
|
||
| ADD . /repos/cudf/ | ||
|
|
||
| # Build env for CUDF build | ||
| ENV CUDF_ROOT=/repos/cudf/cpp/build/ | ||
|
|
||
| # Install cudf from source | ||
| RUN cd /repos/cudf/ && git submodule update --init --recursive && ./build.sh libcudf && \ | ||
| pip install /repos/cudf/python/cudf/. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # Copyright (c) 2021, NVIDIA CORPORATION. | ||
|
|
||
| [build-system] | ||
|
|
||
| requires = [ | ||
| "wheel", | ||
| "setuptools", | ||
| "cython>=0.29,<0.30", | ||
| ] | ||
|
|
||
| [tool.black] | ||
| line-length = 79 | ||
| target-version = ["py36"] | ||
| include = '\.py?$' | ||
| exclude = ''' | ||
| /( | ||
| thirdparty | | ||
| \.eggs | | ||
| \.git | | ||
| \.hg | | ||
| \.mypy_cache | | ||
| \.tox | | ||
| \.venv | | ||
| _build | | ||
| buck-out | | ||
| build | | ||
| dist | ||
| )/ | ||
| ''' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # Copyright (c) 2021, NVIDIA CORPORATION. | ||
|
|
||
| # pyarrow gpu package will have to be built from source : | ||
| # https://arrow.apache.org/docs/python/install.html#installing-from-source | ||
|
|
||
| cupy-cuda101 | ||
| cachetools | ||
| cmake | ||
| cmake-setuptools>=0.1.3 | ||
| cython>=0.29,<0.30 | ||
| dlpack | ||
| fastavro>=0.22.9 | ||
| flatbuffers | ||
| fsspec>=0.6.0 | ||
| hypothesis | ||
| mimesis | ||
| mypy==0.782 | ||
| nbsphinx | ||
| numba>=0.49.0,!=0.51.0 | ||
| numpy | ||
| numpydoc | ||
| nvtx>=0.2.1 | ||
| packaging | ||
| pandas>=1.0,<1.3.0dev0 | ||
| pandoc==2.0a4 | ||
| protobuf | ||
| pyorc | ||
| pytest | ||
| pytest-benchmark | ||
| pytest-xdist | ||
| rapidjson | ||
| recommonmark | ||
| setuptools | ||
| sphinx | ||
| sphinx-copybutton | ||
| sphinx-markdown-tables | ||
| sphinx_rtd_theme | ||
| sphinxcontrib-websupport | ||
| typing_extensions | ||
| typing_extensions | ||
| wheel |
Oops, something went wrong.