tune histogram buckets for wsgi and requests latency (milliseconds)

[verterok]

This is more a exploratory thing, I'm not entirely sure about the correctness.
I based the new values in that we don't usually care if something takes 10 or 50ms, so the first bucket is 50.
the rest of the buckets are based on the same idea and heavily based on our timeouts...which might not be suitable for others(or maybe it is?)

[bloodearnest]

oh lord, how I dislike prometheus.

I did wonder about providing away for the WSGI app to define it's expected latency profile. But prometheus_client's design makes it horrid, as you have to declare everything as classes at import.

Sorry for the mess :(

Over here in non-C land, OpenTelemetry has grown up, and whilst not perfect, is much nicer for this kind of thing. Would be worth looking at, IMO.

[maxiberta]

I wonder if/how this affects existing histograms, i.e. is this backwards compatible? Can existing histogram metrics in prometheus suddenly change their bucket definition without affecting queries?

Also, do we really need to explicitly add and entry for "infinity"? The online docs show the use of <basename>_bucket{le="+Inf"}.

[verterok]

I wonder if/how this affects existing histograms, i.e. is this backwards compatible? Can existing histogram metrics in prometheus suddenly change their bucket definition without affecting queries?

if it breaks existing metrics, we still have the statsd based metrics.

Also, do we really need to explicitly add and entry for "infinity"? The online docs show the use of <basename>_bucket{le="+Inf"}.
No, I was just being explicit, but can remove it.

My question was more about the values makes sense or we should be more granular?

[verterok]

Hey!

Sorry for the mess :(

Nothing to be sorry about!

Over here in non-C land, OpenTelemetry has grown up, and whilst not perfect, is much nicer for this kind of thing. Would be worth looking at, IMO.

Oh, the endless possibilities of non-C land, how's out there in the wild?
Have OpenTelemetry in my todo/look-at list, but there is always something else to take care of (J)

Cheers!