Why do internal metrics send deltas instead of increasing counters?

[gunnihinn]

Various pieces of go-carbon expose internal metrics, for example the persister.

The naive way to instrument some of these metrics is with monotone increasing counters. That's how, for example, I'd monitor updateOperations in the link above, which counts how many updates we've seen so far. What we're really interested in here is the rate of change, but (in my experience, which is only worth so much) if we compute that before send the metric we risk losing information in various ways (packets get dropped, whisper's last-write-wins hoses things, krakens are released, ...).

The persister does not do this, but computes and sends out a delta of update operations since the last time it sent an update, that is, the rate of change itself. Why does it do that?

I've been thinking about this on and off for over a year. Does it have something to do with that if we assume the backing store of the metrics is whisper and use a sum aggregation function, then we'd get more accurate measurements of the rate of change this way?

[azhiltsov]

I totally agree with your concern. Every event derived metric need to be exposed as a counter. That is a best practice and that where we need to move to.

[Civil]

As far as I remember some old discussions with @lomik this is done because go-carbon is re-implementation of carbon and this is the way how carbon expose those metrics to consumers. Currently to migrate old dashboards you just need to change the prefix for the metrics and they'll be compatible.

[deniszh]

I think it's quite easy add another set of metrics named .counter with counter semantics.