Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional text file support to ninja-log utility #12823

Merged
merged 36 commits into from
Mar 20, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6f86d47
Add optional text file support to ninja-log utility
davidwendt Feb 22, 2023
fbd5aa8
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 22, 2023
f1d56ee
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 23, 2023
5ee38e6
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 23, 2023
30285a6
initialize MSG to empty
davidwendt Feb 24, 2023
2e5d3b8
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 24, 2023
a551070
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 27, 2023
e478e7a
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 27, 2023
5be07c9
Merge branch 'bmr-stats-file' of github.com:davidwendt/cudf into bmr-…
davidwendt Feb 27, 2023
3163354
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 28, 2023
3a331a1
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Feb 28, 2023
4f533dc
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 1, 2023
06c8274
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 8, 2023
3f9736e
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 8, 2023
ebbe983
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 9, 2023
1d028ca
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 10, 2023
4ee1365
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 10, 2023
68b41ba
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 10, 2023
9b1cacf
add cmp_log parameter to utility
davidwendt Mar 10, 2023
bb94512
Merge branch 'bmr-stats-file' of github.com:davidwendt/cudf into bmr-…
davidwendt Mar 11, 2023
59f9844
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 11, 2023
6e8ae54
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 13, 2023
afdbc3b
add colors to new diff column
davidwendt Mar 13, 2023
fb3e1f1
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 13, 2023
a9852f7
Merge branch 'bmr-stats-file' of github.com:davidwendt/cudf into bmr-…
davidwendt Mar 14, 2023
efd37bd
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 14, 2023
a505c76
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 14, 2023
4bd68dd
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 15, 2023
990bd3d
Merge branch 'bmr-stats-file' of github.com:davidwendt/cudf into bmr-…
davidwendt Mar 15, 2023
936d211
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 15, 2023
42a2145
fix csv diff column
davidwendt Mar 15, 2023
dbb68ed
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 15, 2023
d396b22
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 17, 2023
b2ac817
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 17, 2023
96350b1
Merge branch 'branch-23.04' into bmr-stats-file
davidwendt Mar 20, 2023
1d0b90b
fix column title
davidwendt Mar 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -300,8 +300,7 @@ if buildAll || hasArg libcudf; then
# Record build times
if [[ "$BUILD_REPORT_METRICS" == "ON" && -f "${LIB_BUILD_DIR}/.ninja_log" ]]; then
echo "Formatting build metrics"
python ${REPODIR}/cpp/scripts/sort_ninja_log.py ${LIB_BUILD_DIR}/.ninja_log --fmt xml > ${LIB_BUILD_DIR}/ninja_log.xml
MSG="<p>"
MSG=""
# get some sccache stats after the compile
if [[ "$BUILD_REPORT_INCL_CACHE_STATS" == "ON" && -x "$(command -v sccache)" ]]; then
COMPILE_REQUESTS=$(sccache -s | grep "Compile requests \+ [0-9]\+$" | awk '{ print $NF }')
Expand All @@ -318,7 +317,8 @@ if buildAll || hasArg libcudf; then
BMR_DIR=${RAPIDS_ARTIFACTS_DIR:-"${LIB_BUILD_DIR}"}
echo "Metrics output dir: [$BMR_DIR]"
mkdir -p ${BMR_DIR}
python ${REPODIR}/cpp/scripts/sort_ninja_log.py ${LIB_BUILD_DIR}/.ninja_log --fmt html --msg "$MSG" > ${BMR_DIR}/ninja_log.html
echo "$MSG" > ${BMR_DIR}/build_stats.txt
davidwendt marked this conversation as resolved.
Show resolved Hide resolved
python ${REPODIR}/cpp/scripts/sort_ninja_log.py ${LIB_BUILD_DIR}/.ninja_log --fmt html --msg "${BMR_DIR}/build_stats.txt" > ${BMR_DIR}/ninja_log.html
cp ${LIB_BUILD_DIR}/.ninja_log ${BMR_DIR}/ninja.log
fi

Expand Down
13 changes: 9 additions & 4 deletions cpp/scripts/sort_ninja_log.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
#
# Copyright (c) 2021-2022, NVIDIA CORPORATION.
# Copyright (c) 2021-2023, NVIDIA CORPORATION.
#
import argparse
import os
import sys
import xml.etree.ElementTree as ET
from pathlib import Path
from xml.dom import minidom

parser = argparse.ArgumentParser()
Expand All @@ -22,7 +23,7 @@
"--msg",
type=str,
default=None,
help="optional message to include in html output",
help="optional text file to include at the top of the html output",
)
args = parser.parse_args()

Expand Down Expand Up @@ -150,11 +151,15 @@ def assign_entries_to_threads(entries):
# output chart results in HTML format
def output_html(entries, sorted_list, args):
print("<html><head><title>Build Metrics Report</title>")
# Note: Jenkins does not support javascript nor style defined in the html
# This creates a standalone html file with no javascript and no styles.
# This was originally to support the Jenkins security policy described here:
davidwendt marked this conversation as resolved.
Show resolved Hide resolved
# https://www.jenkins.io/doc/book/security/configuring-content-security-policy/
print("</head><body>")
if args.msg is not None:
print("<p>", args.msg, "</p>")
msg_file = Path(args.msg)
if msg_file.is_file():
msg = msg_file.read_text()
print("<p>", msg, "</p>")

# map entries to threads
# the end_time is used to scale all the entries to a fixed output width
Expand Down