init flamegraphs

transfer + modify from ClimaCore/ClimaAtmos

add alloc benchmarks

add buildkite driver

rev allocs

docs

activate docs

clean

fix

doc fix

doc fix

check in perf Manifest

bkite flag order fix

scope fix

FT redef error

fix

fix

stash

fix

toy example runs

prep for buildkite staging test

buildkite driver

format

bug

plot in bkite

fix

fix

module path

bkite build path

switch from toy to coupler

switch from toy to coupler

rebase test

Manifest up

prep dummy benchmark

fix

original ProfileCanvas, ProfileViewer files

diff

format

diff

race and docs

race and docs

permissions

restart

restart

restart

revs

.buildkite/pipeline.yml

@@ -193,13 +193,31 @@ steps:
         command: "julia --color=yes --project=perf perf/flame.jl --run_name 2"
         artifact_paths: "perf/output/perf_coarse_single_modular/*"
         agents:
-
           slurm_mem: 20GB
+
       - label: ":rocket: flame graph and allocation tests: perf_target_amip_n32_shortrun"
         command: "julia --color=yes --project=perf perf/flame.jl --run_name 3"
         artifact_paths: "perf/output/perf_target_amip_n32_shortrun/*"
         agents:
           slurm_mem: 20GB
+
+      - label: ":rocket: performance: flame graph diff: perf_default_modular"
+        command: "julia --color=yes --project=perf perf/flame_diff.jl --run_name 1"
+        artifact_paths: "perf/output/perf_diff_default_modular/*"
+        agents:
+          slurm_mem: 20GB
+
+      - label: ":rocket: performance: flame graph diff: perf_coarse_single_modular"
+        command: "julia --color=yes --project=perf perf/flame_diff.jl --run_name 2"
+        artifact_paths: "perf/output/perf_diff_coarse_single_modular/*"
+        agents:
+          slurm_mem: 20GB
+
+      - label: ":rocket: performance: flame graph diff: perf_target_amip_n32_shortrun"
+        command: "julia --color=yes --project=perf perf/flame_diff.jl --run_name 3"
+        artifact_paths: "perf/output/perf_diff_target_amip_n32_shortrun/*"
+        agents:
+          slurm_mem: 20GB
 
       - wait
 
@@ -208,4 +226,5 @@ steps:
         command:
           - build_history staging # name of branch to plot
         artifact_paths:
-          - "build_history.html"
+          - "build_history.html"
+

docs/src/performance.md

@@ -6,11 +6,12 @@
 
 ## Flame Graph Interpretation
 - use for single-process (un)threaded performance CPU profiling of individual stack traces. It provides a tree representation of a set of backtraces, showing the dependence and CPU cost of each function. 
-- here is an example of a flame graph of ClimaCoupler's AMIP run:
+- here is an example of a flame graph of ClimaCoupler's AMIP run, produced by Buildkite running the `perf/flame.jl` script:
 
 ![canvas](images/canvas_coupler.png)
 
 - each row along the y-axis represents a level of backtraces. In this case the lowermost level is at the top, and the top level represents what is directly being run on the CPU. The stacks in each level are sorted alphabetically (not chronologically, like flame _charts_). The column width is proportional to the presence in samples (related to allocations). The colors are grouped into runtime-dispatch, gc, compilation and default. The intensity is random. 
+- we also have a local beta version of flame graphs (in `perf/ProfileCanvasDiff.jl` and `perf/ProfileViewerDiff.js`), triggered by the `perf/flame_diff.jl` script, which plots the same flame graphs as above but with the color corresponding to whether the stack allocation has been reduced (blue) or increased (red) compared to the last staged runs. The color intensity is proportional to the fractional change, and black signifies untracked traces. 
 
 ## References
 - [Description of flame graphs and their interpretation](https://github.com/CliMA/slurm-buildkite/wiki/Flame-Graphs)