Should system.net.tcp.retrans_segs actually use a monotonic count?

[jalaziz]

The documentation seems to indicate that system.net.tcp.retrans_segs and similar metrics are measured as gauges. However, it seems that it's actually being measured as a rate in the network check. I understand that rates are stored as gauges, and that a rate is simply storing the difference between the previous and current value, but since it's time normalized it can result in odd values.

For example, we typically see things like 10.62 retransmitted segments even when using a rollup with sum. This could also simply be to due to the fact that the measured interval using for normalization could be different than the flush interval pass into the flush method.

Using a monotonic count seems more appropriate here. Although, I could also see the rate being useful as well.

[olivielpeau]

@jalaziz Indeed, currently system.net.tcp.retrans_segs is sent from the network check using the rate function, which means that each point of a system.net.tcp.retrans_segs time series represents the number of retransmitted segments per second, as an average since the previous point (i.e. since the previous collection).

So from Datadog there's no easy way to query, for instance, the number of segment retransmits that occurred over a certain timeframe. If I understand correctly this would be the kind of query that you'd like to be able to make? In that case a monotonic counter would make sense yes.

As an aside, since changing the metric type would not be backwards-compatible, if we decide to make the change we'll need to either use another metric name for the counter or wait for a major release of the Agent.

[jalaziz]

Totally understand that we'd have to use a different name.

Some things that could be done now though are:

1. Fix the documentation here to indicate that these metrics are actually rates. Interestingly, the documentation for the system.net.udp metrics correctly identifies them as rates.
2. Fix the default units for these metrics. Again, the system.net.udp series of metrics have the correct units. I can always fix it myself for my account at least.

For being able to query the number of segments (and datagrams for udp), I agree that a new set of metrics using a monotonic counter would be ideal. I'm happy to to submit a PR for this. I'll try to come up with a reasonable name for the new metrics. Maybe simply suffix them with _count?

[olivielpeau]

@jalaziz : The default units for the system.net.udp metrics are correct now, and the units in the documentation have also been updated.

Feel free to open a PR for the new metrics. A _count suffix sounds reasonable to me. Thanks!

[ejholmes]

Could it also be a flag in network.yaml? I'd prefer not to have the rates at all, since they aren't as useful as counts, and _count postfix metrics polute the namespace a bit.

[jalaziz]

@ejholmes Are you suggesting that the flag would disable the rates and not use the suffix? Or it would just disable the rates?

The reason for the suffix is to allow these metrics to be backwards compatible. If we remove the suffix, you'd probably have to fix the units in your account manually. @olivielpeau thoughts?

[ejholmes]

@jalaziz yeah, I'm advocating for the default to use rate, but to be able to use a monotonic_count if configured in network.yaml. Doing this, then changing the metric values in DataDog would be preferable IMO (but I can go either way).