kamon metrics for akka seem totally off

[pnerg]

Getting some really weird results out of the akka metrics.
So I decided to do a really simple Ping/Pong app with two static actors and one that gets created and killed every 5 seconds.
The metrics for the active actor count just keeps increasing which makes no sense
akka_system_active_actors_count{system="area51-kamon-akka"} 2865.0
Same goes go for queued messages which keeps increasing but as I only send a Ping and a Pong as response it is impossible to have a queue/inbox growing.
The PingerPonger code example
I can't see what I'm doing wrong...pointers would be appreciated

Using Kamon 2.1.8, Java 11 OpenJDK and Scala 2.13

[pnerg]

Took a new look at my "issue" now realising the code is using a range-sampler which is represented as a Histogram.

# TYPE akka_system_active_actors histogram
akka_system_active_actors_bucket{le="10.0",system="application"} 1.0
akka_system_active_actors_bucket{le="30.0",system="application"} 13.0
akka_system_active_actors_bucket{le="100.0",system="application"} 68235.0
akka_system_active_actors_bucket{le="300.0",system="application"} 68235.0
akka_system_active_actors_bucket{le="1000.0",system="application"} 68235.0
akka_system_active_actors_bucket{le="3000.0",system="application"} 68235.0
akka_system_active_actors_bucket{le="10000.0",system="application"} 68235.0
akka_system_active_actors_bucket{le="30000.0",system="application"} 68235.0
akka_system_active_actors_bucket{le="100000.0",system="application"} 68235.0
akka_system_active_actors_bucket{le="+Inf",system="application"} 68235.0
akka_system_active_actors_count{system="application"} 68235.0
akka_system_active_actors_sum{system="application"} 2416078.0

So of course the akka_system_active_actors_count{system="application"} 68235.0 is just a sample counter i.e. how many measurements have been made, not actual active actors. Hence it always grows.
Silly of me not to realise that.

However due to representing it this way the actual use is rather limited. I can't get an accurate snapshot of what is the current status as it's just historical data over time.

[Pooriya-Shokri]

Your question is surprisingly the only related issue/discussion about weird Kamon metrics.
I can manage to understand some of metrics by focusing on '_sum' part and not on '_bucket' or '_count' parts.
But in some of the metrics understanding the metric even by '_sum' part is quite tricky.
For example take a look on "AKKA_GROUP_MEMBERS_SUM" metric on my local running project with about 10 actors.
Do you have any clues to explain this metric?

[pnerg]

I've done a PR #1011 allowing distribution based metrics to be generated at snapshot gauges as well.
In this PR Ivan explained the '_sum' metric and also provided a link to a blog he is writing on the topic of understanding range sampler metrics.

[SimunKaracic]

https://kamon.io/blog/monitoring-queues-and-resources-with-kamon-range-samplers/?utm_source=githubdiscussion

The aforementioned blog post! I hope it clears some things up!