Merge #238

238: Extend flamegraphs to show alloc diff r=LenkaNovak a=LenkaNovak

Experimental PR that test the ease and usefulness of color coding flame graphs based on allocation changes compared to the previous commit, using a local testing bed. 

## To-do


Co-authored-by: LenkaNovak <[email protected]>

.buildkite/pipeline.yml

@@ -193,13 +193,31 @@ steps:
         command: "julia --color=yes --project=perf perf/flame.jl --run_name 2"
         artifact_paths: "perf/output/perf_coarse_single_modular/*"
         agents:
-
           slurm_mem: 20GB
+
       - label: ":rocket: flame graph and allocation tests: perf_target_amip_n32_shortrun"
         command: "julia --color=yes --project=perf perf/flame.jl --run_name 3"
         artifact_paths: "perf/output/perf_target_amip_n32_shortrun/*"
         agents:
           slurm_mem: 20GB
+
+      - label: ":rocket: performance: flame graph diff: perf_default_modular"
+        command: "julia --color=yes --project=perf perf/flame_diff.jl --run_name 1"
+        artifact_paths: "perf/output/perf_diff_default_modular/*"
+        agents:
+          slurm_mem: 20GB
+
+      - label: ":rocket: performance: flame graph diff: perf_coarse_single_modular"
+        command: "julia --color=yes --project=perf perf/flame_diff.jl --run_name 2"
+        artifact_paths: "perf/output/perf_diff_coarse_single_modular/*"
+        agents:
+          slurm_mem: 20GB
+
+      - label: ":rocket: performance: flame graph diff: perf_target_amip_n32_shortrun"
+        command: "julia --color=yes --project=perf perf/flame_diff.jl --run_name 3"
+        artifact_paths: "perf/output/perf_diff_target_amip_n32_shortrun/*"
+        agents:
+          slurm_mem: 20GB
 
       - wait
 
@@ -208,4 +226,5 @@ steps:
         command:
           - build_history staging # name of branch to plot
         artifact_paths:
-          - "build_history.html"
+          - "build_history.html"
+

docs/src/performance.md

@@ -6,11 +6,12 @@
 
 ## Flame Graph Interpretation
 - use for single-process (un)threaded performance CPU profiling of individual stack traces. It provides a tree representation of a set of backtraces, showing the dependence and CPU cost of each function. 
-- here is an example of a flame graph of ClimaCoupler's AMIP run:
+- here is an example of a flame graph of ClimaCoupler's AMIP run, produced by Buildkite running the `perf/flame.jl` script:
 
 ![canvas](images/canvas_coupler.png)
 
 - each row along the y-axis represents a level of backtraces. In this case the lowermost level is at the top, and the top level represents what is directly being run on the CPU. The stacks in each level are sorted alphabetically (not chronologically, like flame _charts_). The column width is proportional to the presence in samples (related to allocations). The colors are grouped into runtime-dispatch, gc, compilation and default. The intensity is random. 
+- we also have a local beta version of flame graphs (in `perf/ProfileCanvasDiff.jl` and `perf/ProfileViewerDiff.js`), triggered by the `perf/flame_diff.jl` script, which plots the same flame graphs as above but with the color corresponding to whether the stack allocation has been reduced (blue) or increased (red) compared to the last staged runs. The color intensity is proportional to the fractional change, and black signifies untracked traces. 
 
 ## References
 - [Description of flame graphs and their interpretation](https://github.com/CliMA/slurm-buildkite/wiki/Flame-Graphs)