Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build-time publish step to cpu build script #9927

Merged
merged 38 commits into from
Jan 10, 2022
Merged

Add build-time publish step to cpu build script #9927

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
f72218e
Add build-time publish step to cpu build script
davidwendt Dec 17, 2021
fc00e28
remove unneeded xml copy
davidwendt Dec 17, 2021
8b0b888
temporarily clear ccache
davidwendt Dec 17, 2021
92825d0
remove ccache clear
davidwendt Dec 17, 2021
d390db1
broke something; trying again
davidwendt Dec 17, 2021
b636402
add some debug output messages
davidwendt Dec 17, 2021
e1222f2
remove copy metrics after libcudf pkg
davidwendt Dec 17, 2021
aec614f
searching for log html
davidwendt Dec 17, 2021
1aa0ff5
found ninja-log html file
davidwendt Dec 17, 2021
fe3c2e7
temp clear ccache again
davidwendt Dec 18, 2021
7db99ac
add style to table elements
davidwendt Dec 18, 2021
f6f328b
testing output table style
davidwendt Dec 18, 2021
9b36a49
going old school since jenkins does not support custom html styles
davidwendt Dec 18, 2021
5cb6848
retry with cleared cache
davidwendt Dec 18, 2021
f5e38d2
fix summary table
davidwendt Dec 19, 2021
5e1360e
add some build info to the output
davidwendt Dec 19, 2021
17adcf7
convert 4 ranges to 3
davidwendt Dec 19, 2021
e2826a0
remove ccache clear
davidwendt Dec 19, 2021
82ce7c3
fix env var cmd
davidwendt Dec 19, 2021
62291f0
try getting env vars again
davidwendt Dec 20, 2021
c92ade8
fix script again
davidwendt Dec 20, 2021
0f2bd3b
remove unresolved message vars
davidwendt Dec 20, 2021
ce222a9
test recompile-everything technique
davidwendt Dec 20, 2021
f91343e
report cache hit rate
davidwendt Dec 20, 2021
fad7cf2
check cache hit rate
davidwendt Dec 20, 2021
8033c58
Merge branch 'branch-22.02' into build-time-publish-html
davidwendt Dec 20, 2021
e0666d1
remove temporary recompile test
davidwendt Dec 21, 2021
192e0f1
remove commented out code
davidwendt Dec 22, 2021
42d37a8
Merge branch 'branch-22.02' into build-time-publish-html
davidwendt Jan 4, 2022
226d355
Merge branch 'branch-22.02' into build-time-publish-html
davidwendt Jan 4, 2022
40d0479
add libcudf.so size to metrics output
davidwendt Jan 5, 2022
01c0a3b
report libcudf.so size only if found
davidwendt Jan 5, 2022
0a03c23
add build-metrics parameters to build.sh
davidwendt Jan 6, 2022
8c1f89a
Merge branch 'branch-22.02' into build-time-publish-html
davidwendt Jan 6, 2022
3b840d4
reset metrics defaults to OFF
davidwendt Jan 7, 2022
18f2990
pass new parameters to build.sh
davidwendt Jan 7, 2022
06f9d50
remove debugging echo statements
davidwendt Jan 7, 2022
6e1b79b
Merge branch 'branch-22.02' into build-time-publish-html
davidwendt Jan 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ if buildAll || hasArg libcudf; then
FILES_IN_CCACHE=""
if [ -x "$(command -v ccache)" ]; then
FILES_IN_CCACHE=$(ccache -s | grep "files in cache")
echo "$FILES_IN_CCACHE"
fi

cmake -S $REPODIR/cpp -B ${LIB_BUILD_DIR} \
Expand All @@ -189,6 +190,7 @@ if buildAll || hasArg libcudf; then
-DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
${CMAKE_ARGS}

echo "LIB_BUILD_DIR=$LIB_BUILD_DIR"
cd ${LIB_BUILD_DIR}

compile_start=$(date +%s)
Expand All @@ -198,7 +200,7 @@ if buildAll || hasArg libcudf; then

# Record build times
if [[ -f "${LIB_BUILD_DIR}/.ninja_log" ]]; then
echo "Formatting build times"
echo "Formatting build times $LIB_BUILD_DIR"
python ${REPODIR}/cpp/scripts/sort_ninja_log.py ${LIB_BUILD_DIR}/.ninja_log --fmt xml > ${LIB_BUILD_DIR}/ninja_log.xml
message="$FILES_IN_CCACHE <p>$PARALLEL_LEVEL parallel build time is $compile_total seconds"
echo "$message"
Expand Down
16 changes: 16 additions & 0 deletions ci/cpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,27 @@ else
fi

if [ "$BUILD_LIBCUDF" == '1' ]; then

# XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ccache -C

gpuci_logger "Build conda pkg for libcudf"
gpuci_conda_retry build --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/libcudf $CONDA_BUILD_ARGS
mkdir -p ${CONDA_BLD_DIR}/libcudf/work
cp -r ${CONDA_BLD_DIR}/work/* ${CONDA_BLD_DIR}/libcudf/work

# echo "CONDA_BLD_DIR=$CONDA_BLD_DIR"
# echo "searching for ninja_log"
# find . -name "ninja_log.html"

# Copy libcudf build time results
LIBCUDF_BUILD_DIR=$CONDA_BLD_DIR/libcudf/work/cpp/build
echo "Checking for build time log $LIBCUDF_BUILD_DIR/ninja_log.html"
if [[ -f "$LIBCUDF_BUILD_DIR/ninja_log.html" ]]; then
gpuci_logger "Copying build time results"
mkdir -p "$WORKSPACE/build-metrics"
cp "$LIBCUDF_BUILD_DIR/ninja_log.html" "$WORKSPACE/build-metrics/BuildMetrics.html"
fi

gpuci_logger "Build conda pkg for libcudf_kafka"
gpuci_conda_retry build --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/libcudf_kafka $CONDA_BUILD_ARGS
Expand Down
9 changes: 5 additions & 4 deletions ci/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,12 +181,13 @@ else
done

# Copy libcudf build time results
echo "Checking for build time log $LIB_BUILD_DIR/ninja_log.html"
if [[ -f "$LIB_BUILD_DIR/ninja_log.html" ]]; then
echo "Checking for build time log $LIB_BUILD_DIR/ninja_log.xml"
if [[ -f "$LIB_BUILD_DIR/ninja_log.xml" ]]; then
gpuci_logger "Copying build time results"
echo "LIB_BUILD_DIR=$LIB_BUILD_DIR"
cp "$LIB_BUILD_DIR/ninja_log.xml" "$WORKSPACE/test-results/buildtimes-junit.xml"
mkdir -p "$WORKSPACE/build-metrics"
cp "$LIB_BUILD_DIR/ninja_log.html" "$WORKSPACE/build-metrics/BuildMetrics.html"
# mkdir -p "$WORKSPACE/build-metrics"
# cp "$LIB_BUILD_DIR/ninja_log.html" "$WORKSPACE/build-metrics/BuildMetrics.html"
fi

################################################################################
Expand Down
39 changes: 35 additions & 4 deletions cpp/scripts/sort_ninja_log.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,21 +88,42 @@
elif output_fmt == "html":
# output results in HTML format
print("<html><head><title>Sorted Ninja Build Times</title>")
print("<style>", "table, th, td { border:1px solid black; }", "</style>")
# Note: Jenkins does not support style defined in the html
# https://www.jenkins.io/doc/book/security/configuring-content-security-policy/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. This is why fancy tables doesn't work in html page! 😢

print("</head><body>")
if args.msg is not None:
print("<p>", args.msg, "</p>")
print("<table>")
print(
"<tr><th>File</th>",
"<th align='right'>Compile time (ms)</th>",
"<th align='right'>Size (bytes)</th><tr>",
"<th>Compile time<br/>(ms)</th>",
"<th>Size<br/>(bytes)</th><tr>",
sep="",
)
summary = {"red": 0, "yellow": 0, "gray": 0, "green": 0}
red = "bgcolor='#FFBBD0'"
yellow = "bgcolor='#FFFF80'"
gray = "bgcolor='#CCCCCC'"
green = "bgcolor='#AAFFBD'"
for key in sl:
result = entries[key]
elapsed = result[0]
color = green
if elapsed > 600000: # 10 minutes
color = red
summary["red"] += 1
elif elapsed > 300000: # 5 minutes
color = yellow
summary["yellow"] += 1
elif elapsed > 120000: # 2 minutes
color = gray
summary["gray"] += 1
else:
summary["green"] += 1
print(
"<tr><td>",
"<tr ",
color,
"><td>",
key,
"</td><td align='right'>",
result[0],
Expand All @@ -111,6 +132,16 @@
"</td></tr>",
sep="",
)
print("</table><br/><table border='2'>")
# include summary table with color legend
print("<tr><td", red, ">&gt; 10 minutes</td>")
print("<td align='right'>", summary["red"], "</td></tr>")
print("<tr><td", yellow, ">5 minutes &lt; time &lt; 10 minutes</td>")
print("<td align='right'>", summary["yellow"], "</td></tr>")
print("<tr><td", gray, ">2 minutes &lt; time &lt; 5 minutes</td>")
print("<td align='right'>", summary["gray"], "</td></tr>")
print("<tr><td", green, ">&lt; 2 minutes</td>")
print("<td align='right'>", summary["green"], "</td></tr>")
print("</table></body></html>")

else:
Expand Down