Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run compute-sanitizer in nightly build #9641

Merged
merged 16 commits into from
Nov 30, 2021
Merged

Run compute-sanitizer in nightly build #9641

Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f5bc277
add compute-sanitizer-to-junit-xml.py converter
karthikeyann Nov 9, 2021
dbedae9
run compute-sanitizer in ci under env COMPUTE_SANITIZER_ENABLE
karthikeyann Nov 9, 2021
e987897
review comments
karthikeyann Nov 15, 2021
5012e7e
Merge branch 'branch-22.02' of github.com:rapidsai/cudf into fea-sani…
karthikeyann Nov 15, 2021
f1a31b4
nightly check BUILD_MODE=branch, BUILD_TYPE=gpu
karthikeyann Nov 16, 2021
0e34fff
change expected value to true (as per jenkins COMPUTE_SANITIZER_ENABL…
karthikeyann Nov 16, 2021
dbc2750
test commit to see memcheck reports in jenkins
karthikeyann Nov 16, 2021
7635953
fix summing error count
karthikeyann Nov 17, 2021
5d6df02
skip ERROR_TEST, remove writing test xml output again.
karthikeyann Nov 17, 2021
c6c1ddc
skip failure of error mismatch (just to unblock)
karthikeyann Nov 18, 2021
ac0fc75
fix skipping ERROR_TEST, update reporting hierarchy
karthikeyann Nov 18, 2021
b30637a
Merge branch 'branch-22.02' of github.com:rapidsai/cudf into fea-sani…
karthikeyann Nov 18, 2021
18b0042
remove debug comments, enable only in nightly build
karthikeyann Nov 19, 2021
b8812bc
Merge branch 'branch-22.02' of github.com:rapidsai/cudf into fea-sani…
karthikeyann Nov 24, 2021
ec8789c
move to ci utils
karthikeyann Nov 24, 2021
2a7f20d
cs.log report parsing moved to gpuci
karthikeyann Nov 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion ci/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2018-2020, NVIDIA CORPORATION.
# Copyright (c) 2018-2021, NVIDIA CORPORATION.
##############################################
# cuDF GPU build and test script for CI #
##############################################
Expand Down Expand Up @@ -176,6 +176,28 @@ else
${gt} --gtest_output=xml:"$WORKSPACE/test-results/"
done

################################################################################
# MEMCHECK - Run compute-sanitizer on GoogleTest (only in nightly builds)
################################################################################
if [[ "$BUILD_MODE" == "branch" && "$BUILD_TYPE" == "gpu" ]]; then
if [[ "$COMPUTE_SANITIZER_ENABLE" == "true" ]]; then
gpuci_logger "Memcheck on GoogleTests with rmm_mode=cuda"
export GTEST_CUDF_RMM_MODE=cuda
COMPUTE_SANITIZER_CMD="compute-sanitizer --tool memcheck"
mkdir -p "$WORKSPACE/test-results/"
for gt in gtests/*; do
test_name=$(basename ${gt})
if [[ "$test_name" == "ERROR_TEST" ]]; then
continue
fi
echo "Running GoogleTest $test_name"
${COMPUTE_SANITIZER_CMD} ${gt} | tee "$WORKSPACE/test-results/${test_name}.cs.log"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this have been the following:

 ${COMPUTE_SANITIZER_CMD} ${gt} --rmm_mode=cuda | tee "$WORKSPACE/test-results/${test_name}.cs.log"

And then you would not need to set and unset the GTEST_CUDF_RMM_MODE environment variable?

Copy link
Contributor Author

@karthikeyann karthikeyann Dec 1, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use either one for google tests. Initially we wanted to use ctest and cdash to run and report memcheck results. It was not possible to add arguments only for memcheck in ctest. So, we went with environmental variable. Since we want to report to Jenkins, we didn't use ctest/cdash
Hoping to use the environmental variable for both gtests and pytests. (variable name might change)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use the cuda async mode for RMM. That will help the tests run faster by using the CUDA pool but still have memcheck support.

done
unset GTEST_CUDF_RMM_MODE
# test-results/*.cs.log are processed in gpuci
fi
fi

CUDF_CONDA_FILE=`find ${CONDA_ARTIFACT_PATH} -name "libcudf-*.tar.bz2"`
CUDF_CONDA_FILE=`basename "$CUDF_CONDA_FILE" .tar.bz2` #get filename without extension
CUDF_CONDA_FILE=${CUDF_CONDA_FILE//-/=} #convert to conda install
Expand Down