Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build metrics report as artifact to cpp-build workflow #12750

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
367fffc
Add build metrics report as artifact to cpp-build workflow
davidwendt Feb 9, 2023
f895bf8
move upload to conda-cpp-build.yaml
davidwendt Feb 9, 2023
054b3d1
fix uses syntax
davidwendt Feb 9, 2023
57ce841
update pr.yaml with local conda-cpp-build.yaml
davidwendt Feb 9, 2023
7ba8466
add output dir variable for bmr files
davidwendt Feb 9, 2023
561919e
run upload artifact test
davidwendt Feb 10, 2023
9917e27
temporarily hardcode some paths
davidwendt Feb 10, 2023
37c23e2
fix typo in script
davidwendt Feb 10, 2023
28f5951
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 10, 2023
4cd458a
add env var to meta.yaml
davidwendt Feb 10, 2023
a83fed0
fix test.log path
davidwendt Feb 10, 2023
55158e4
experiment with smaller build/test for faster turn around
davidwendt Feb 10, 2023
e5024a7
try again
davidwendt Feb 10, 2023
0241cee
guess again
davidwendt Feb 10, 2023
0720523
try to dump all the env vars
davidwendt Feb 10, 2023
7bc70c5
testing env vars
davidwendt Feb 10, 2023
84ebdc8
path: $RAPIDS_BMR_DIR
davidwendt Feb 10, 2023
2d02299
path: ${RAPIDS_BMR_DIR}
davidwendt Feb 10, 2023
a6ea19c
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 13, 2023
b56207e
Merge branch 'exper-build-metrics-report' of github.com:davidwendt/cu…
davidwendt Feb 13, 2023
6f3611b
check env vars in build_cpp.sh
davidwendt Feb 13, 2023
d16603a
fix copyright
davidwendt Feb 13, 2023
c8269b1
upload report to s3 downloads
davidwendt Feb 13, 2023
f16e5ba
check upload env vars
davidwendt Feb 13, 2023
7678389
ren RAPIDS_BMR_DIR to RAPIDS_ARTIFACTS_DIR
davidwendt Feb 13, 2023
d3f7155
test remove env var from container
davidwendt Feb 13, 2023
0db9b0a
remove conda-cpp-tests.yaml
davidwendt Feb 13, 2023
052e253
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 13, 2023
f5d0067
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 15, 2023
3ed6de2
output text per link
davidwendt Feb 15, 2023
131de7b
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 16, 2023
a64681d
reset temp changes
davidwendt Feb 16, 2023
1c4b139
experiment with test config paramaters
davidwendt Feb 16, 2023
5a8b96c
temp disable some steps
davidwendt Feb 16, 2023
e4b4232
undoing temp change
davidwendt Feb 16, 2023
5551949
fix dependencies
davidwendt Feb 16, 2023
b0247b3
add test for compute-sanitizer run
davidwendt Feb 16, 2023
e2a6582
add conda-cpp-memcheck-tests to pr-builder needs
davidwendt Feb 16, 2023
f25dc48
add pretend compute-sanitizer run
davidwendt Feb 16, 2023
45eec1c
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 16, 2023
7fbca2b
skip STREAM_IDENTIFICATION_TEST
davidwendt Feb 16, 2023
86b1593
reset temp changes to finalize PR
davidwendt Feb 16, 2023
c70d2e7
remove conda-cpp-build.yaml
davidwendt Feb 16, 2023
0795381
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 16, 2023
c797f3a
report text highlighted with colored text
davidwendt Feb 17, 2023
3611032
Merge branch 'branch-23.04' into exper-build-metrics-report
davidwendt Feb 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -315,9 +315,11 @@ if buildAll || hasArg libcudf; then
LIBCUDF_FS=$(ls -lh ${LIB_BUILD_DIR}/libcudf.so | awk '{print $5}')
MSG="${MSG}<br/>libcudf.so size: $LIBCUDF_FS"
fi
echo "$MSG"
python ${REPODIR}/cpp/scripts/sort_ninja_log.py ${LIB_BUILD_DIR}/.ninja_log --fmt html --msg "$MSG" > ${LIB_BUILD_DIR}/ninja_log.html
cp ${LIB_BUILD_DIR}/.ninja_log ${LIB_BUILD_DIR}/ninja.log
BMR_DIR=${RAPIDS_ARTIFACTS_DIR:-"${LIB_BUILD_DIR}"}
echo "Metrics output dir: [$BMR_DIR]"
mkdir -p ${BMR_DIR}
python ${REPODIR}/cpp/scripts/sort_ninja_log.py ${LIB_BUILD_DIR}/.ninja_log --fmt html --msg "$MSG" > ${BMR_DIR}/ninja_log.html
cp ${LIB_BUILD_DIR}/.ninja_log ${BMR_DIR}/ninja.log
ajschmidt8 marked this conversation as resolved.
Show resolved Hide resolved
fi

if [[ ${INSTALL_TARGET} != "" ]]; then
Expand Down
28 changes: 27 additions & 1 deletion ci/build_cpp.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2022, NVIDIA CORPORATION.
# Copyright (c) 2022-2023, NVIDIA CORPORATION.

set -euo pipefail

Expand All @@ -14,3 +14,29 @@ rapids-logger "Begin cpp build"
rapids-mamba-retry mambabuild conda/recipes/libcudf

rapids-upload-conda-to-s3 cpp

echo "++++++++++++++++++++++++++++++++++++++++++++"

if [[ -d $RAPIDS_ARTIFACTS_DIR ]]; then
ls -l ${RAPIDS_ARTIFACTS_DIR}
fi

echo "++++++++++++++++++++++++++++++++++++++++++++"

FILE=${RAPIDS_ARTIFACTS_DIR}/ninja.log
if [[ -f $FILE ]]; then
echo -e "\x1B[33;1m\x1B[48;5;240m Ninja log for this build available at the following link \x1B[0m"
UPLOAD_NAME=cpp_cuda${RAPIDS_CUDA_VERSION%%.*}_$(arch).ninja.log
rapids-upload-to-s3 "${UPLOAD_NAME}" "${FILE}"
fi

echo "++++++++++++++++++++++++++++++++++++++++++++"

FILE=${RAPIDS_ARTIFACTS_DIR}/ninja_log.html
if [[ -f $FILE ]]; then
echo -e "\x1B[33;1m\x1B[48;5;240m Build Metrics Report for this build available at the following link \x1B[0m"
UPLOAD_NAME=cpp_cuda${RAPIDS_CUDA_VERSION%%.*}_$(arch).BuildMetricsReport.html
rapids-upload-to-s3 "${UPLOAD_NAME}" "${FILE}"
fi

echo "++++++++++++++++++++++++++++++++++++++++++++"
16 changes: 0 additions & 16 deletions ci/test_cpp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,21 +66,5 @@ for gt in "$CONDA_PREFIX"/bin/gtests/{libcudf,libcudf_kafka}/* ; do
fi
done

if [[ "${RAPIDS_BUILD_TYPE}" == "nightly" ]]; then
rapids-logger "Memcheck gtests with rmm_mode=cuda"
export GTEST_CUDF_RMM_MODE=cuda
COMPUTE_SANITIZER_CMD="compute-sanitizer --tool memcheck"
for gt in "$CONDA_PREFIX"/bin/gtests/{libcudf,libcudf_kafka}/* ; do
test_name=$(basename ${gt})
if [[ "$test_name" == "ERROR_TEST" ]]; then
continue
fi
echo "Running gtest $test_name"
${COMPUTE_SANITIZER_CMD} ${gt} | tee "${RAPIDS_TESTS_DIR}${test_name}.cs.log"
done
unset GTEST_CUDF_RMM_MODE
# TODO: test-results/*.cs.log are processed in CI
fi

ajschmidt8 marked this conversation as resolved.
Show resolved Hide resolved
rapids-logger "Test script exiting with value: $EXITCODE"
exit ${EXITCODE}
1 change: 1 addition & 0 deletions conda/recipes/libcudf/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ build:
- SCCACHE_IDLE_TIMEOUT
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- RAPIDS_ARTIFACTS_DIR

requirements:
build:
Expand Down