Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate Java cudf jar build with statically linked dependencies #10578

Conversation

gerashegalov
Copy link
Contributor

@gerashegalov gerashegalov commented Apr 2, 2022

This PR automates cudf jar build via build.sh . It uses java CI centos7 Docker image to build a portable jar from the source checked out on the host. The Docker container is run as a normal non-root user https://docs.docker.com/engine/install/linux-postinstall/

Examples:

# clean, build libcudf, and use `mvn package` with default parallelism
./build.sh clean cudfjar

# build libcudf, and use `mvn install` 
PARALLEL_LEVEL=4  MVN_PHASES="install" ./build.sh cudfjar
  1. allow overriding GPU architectures with ALL, NATIVE, some subset.

  2. Drop GDS from the image

  3. Adds parallelism to the libcudfjni build, removes make hardcoding

  4. Fix: don't disregard CUDF_CPP_BUILD_DIR passed to Maven

  5. Use a dedicated CUDF_CPP_BUILD_DIR value to avoid corrupting interactions with build.sh libcudf

@gerashegalov gerashegalov requested a review from a team as a code owner April 2, 2022 00:32
@github-actions github-actions bot added the Java Affects Java cuDF API. label Apr 2, 2022
@gerashegalov gerashegalov self-assigned this Apr 2, 2022
@gerashegalov gerashegalov added non-breaking Non-breaking change tech debt improvement Improvement / enhancement to an existing function labels Apr 2, 2022
@codecov
Copy link

codecov bot commented Apr 2, 2022

Codecov Report

Merging #10578 (acc42a8) into branch-22.06 (956c7b5) will increase coverage by 0.03%.
The diff coverage is 88.97%.

@@               Coverage Diff                @@
##           branch-22.06   #10578      +/-   ##
================================================
+ Coverage         86.30%   86.34%   +0.03%     
================================================
  Files               140      140              
  Lines             22255    22280      +25     
================================================
+ Hits              19207    19237      +30     
+ Misses             3048     3043       -5     
Impacted Files Coverage Δ
python/cudf/cudf/core/frame.py 94.75% <ø> (+1.02%) ⬆️
python/dask_cudf/dask_cudf/tests/test_accessor.py 98.41% <ø> (ø)
python/cudf/cudf/core/indexed_frame.py 91.77% <87.93%> (-0.87%) ⬇️
python/cudf/cudf/core/column/lists.py 90.62% <100.00%> (+0.57%) ⬆️
python/cudf/cudf/core/dataframe.py 93.59% <100.00%> (ø)
python/cudf/cudf/core/series.py 95.28% <100.00%> (-0.01%) ⬇️
python/cudf/cudf/core/column/column.py 89.45% <0.00%> (+0.10%) ⬆️
python/cudf/cudf/core/column/string.py 89.10% <0.00%> (+0.12%) ⬆️
python/cudf/cudf/core/groupby/groupby.py 91.72% <0.00%> (+0.22%) ⬆️
python/cudf/cudf/core/tools/datetimes.py 84.49% <0.00%> (+0.30%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb03c8b...acc42a8. Read the comment docs.

build.sh Outdated Show resolved Hide resolved
java/ci/Dockerfile.centos7 Show resolved Hide resolved
java/ci/Dockerfile.centos7 Show resolved Hide resolved
gerashegalov and others added 3 commits April 4, 2022 10:42
Co-authored-by: Jason Lowe <[email protected]>
- and fix bug of CUDF_CPP_BUILD_DIR parameter being ignored

Signed-off-by: Gera Shegalov <[email protected]>
@github-actions github-actions bot added the CMake CMake build issue label Apr 5, 2022
@gerashegalov gerashegalov requested a review from jlowe April 5, 2022 06:25
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov gerashegalov requested a review from nvdbaranec April 7, 2022 01:24
@gerashegalov
Copy link
Contributor Author

Made local host's .m2 customizable via LOCAL_MAVEN_REPO environment. defaulting to the standard location ~/.m2

@gerashegalov
Copy link
Contributor Author

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 09b080d into rapidsai:branch-22.06 Apr 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CMake CMake build issue improvement Improvement / enhancement to an existing function Java Affects Java cuDF API. non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants