Stats overhaul

[pwojcikdev]

Stats overhaul

The goal of this PR is to address some of the issues that have been increasingly cropping up with the current stats system. I believe that having a more flexible and performant system will allow us to experiment and iterate faster.

The immediate problem that is fixed is the fact that we ran out of space for declaring new stat enums:

In file included from /nano-node/nano/lib/stats.hpp:5:
/nano-node/nano/lib/stats_enums.hpp:407:2: error: enumerator value 256 is not representable in the underlying type 'uint8_t' (aka 'unsigned char')

However, this PR contains a bit more than that. The main changes are:

• Do not require stat::detail enum to have uint8 underlying type. This allows for practically unlimited number of custom stats.
• Offload heavy IO operations (saving stats to disk) to a dedicated thread. Previously this was done inside the call to stat update functions, which introduced latency spikes.
• Rework the way histograms are handled. Previously histograms needed to be registered with the stats system beforehand, which could be cumbersome and as a result not used. Now we store samples in a ring buffer and leave histogram creation to the user (e.g. nano-prom-exporter).
  Note: I'm still not sure whether this approach is really better, this needs a bit more experimentation.
• Use read-write locks with atomic integers for counter values. This is a bit experimental, but on my system it seems to give some performance benefits. I'll try to include microbenchmarking suite as part of a separate PR (ongoing work is here https://github.com/pwojcikdev/nano-node/tree/stats-benchmarking )
• Remove special semantics of stat::detail::all enum. Previously we tracked a special all entry that was used to aggregate all stats of a given type. It was used only by unit tests and required special filters to remove from grafana dashboards, where it unnecessarily cluttered the view. Now stat aggregation is handled by a dedicated overload of stats::count () function.

Heatmaps

I added experimental support for histogram generation to my fork of the nano-prom-exporter (https://github.com/pwojcikdev/nano-prom-exporter/commits/histograms/). This allows for generating heatmaps like this one:

[dsiganos]

LGTM, I did a high level review only because a detailed review would take too long.

[gr0vity-dev]

When doing a quick throughput test, the performance seems similar.
One thing I noticed is that the "all" tab from messages remains empty.
(I didn't update the grafana dashboard yet)