Flame graphs

[tbragin]

Flame graphs can be useful for visualizing output of a profiling tool:
http://www.brendangregg.com/flamegraphs.html
https://github.com/brendangregg/FlameGraph

I remember chatting with @danielmitterdorfer about this a while back. Would be interesting to think through how profiling data can be stored in Elasticsearch and visualized in a native Kibana visualization like this.

cc: @alexfrancoeur @thomasneirynck

[axw]

I've been wanting profiling storage since forever. Now that I'm at Elastic, I thought I'd actually do something about it. I've started on a new Beat which gathers pprof profiles: https://github.com/axw/pprofbeat.

(This is my first real foray into ES and Beats, so the data format is probably not great - but it's something.)

I care mostly about profiling Go programs, which is pretty straightforward: import a standard package, and you can expose CPU and heap profile data via an HTTP endpoint. What I want to see is:

• periodic profiling (done by pprofbeat; other beats would be required for on-machine profiling, i.e. "perf" and friends)
• Kibana visualisations, with the ability to compare profiles (e.g. to see how much memory has grown over time, or how the CPU profile looks at various load levels)

I was intending to look at embedding the "pprof" web UI as a plugin, although I don't really know where to start. Needless to say, the more integrated the better.

A few screenshots of the pprof web UI below in case folks are not aware of it - might serve as inspiration.

Example pprof graph:

Pprof flamegraph:

Pprof top offenders:

Pprof source code annotation:

[pmoust]

A very interesting visualisation approach from the Netflix Perf team, Flamescope

This tool makes it easy to distinguish patterns and pop into flamegraph view in subsecond granularity.

X axis is time (t) in seconds, Y axis is the millisecond split of t. The vector that produces the heatmap color density is the event count for that X->Y time.

[formgeist]

I've taken the liberty of creating a new visualization type request in Elastic Charts elastic/elastic-charts#519 to open the conversation with that team and in order to make a qualified assessment on pursuing an MVP implementation for APM and what that should entail.

[formgeist]

Elastic Charts have just released an alpha version of the flame chart elastic/elastic-charts#965

[monfera]

elastic-charts merged focus (in-chart drilldown) with tweening

Mar-02-2021.15-59-31.mp4

[timductive]

Closing as we now have a flame graph in the o11y profiler application