Reduce the operator latency metric cardinality

[jianoaix]

Why are these changes needed?

Currently attestation_latency_ms metric has a high cardinality, because each operator will have 12 time series (and there are many hundreds of operators), due to:

• breakdown by failure/success
• breakdown by percentiles, due to the use of latency summary
• extra stream sum/count due to the use of latency summary

This PR changes to:

• use gauge, not latency/distribution: this avoids the percentiles, as well as sum/count
• ignore failed requests: just success requests latency should give us enough understanding

As a result, there is 1 time series per operator.

12 time series per operator:

eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="failure",quantile="0.5"} NaN
eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="failure",quantile="0.9"} NaN
eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="failure",quantile="0.95"} NaN
eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="failure",quantile="0.99"} NaN
eigenda_batcher_attestation_latency_ms_sum{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="failure"} 30669
eigenda_batcher_attestation_latency_ms_count{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="failure"} 1
eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="success",quantile="0.5"} 832
eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="success",quantile="0.9"} 857
eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="success",quantile="0.95"} 857
eigenda_batcher_attestation_latency_ms{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="success",quantile="0.99"} 857
eigenda_batcher_attestation_latency_ms_sum{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="success"} 3.465767e+06
eigenda_batcher_attestation_latency_ms_count{operator_id="ff7ca6eb4373179d6d3fb35a62911d7dda217b44d3e2db42dc6b912c393d0663",status="success"} 3288

Before:

After:

Checks

• I've made sure the lint is passing in this PR.
• I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
• I've checked the new test coverage and the coverage percentage didn't drop.
• Testing Strategy
  • Unit tests
  • Integration tests
  • This PR is not tested :(

[dmanc]

do we need the label if we are just going to null it out?

[jianoaix]

Haven't evaluated it much, is it safe to delete the label for this?

[dmanc]

I don't see anything obvious that would go wrong. Maybe test in preprod?

[jianoaix]

It'd be better to keep it for a while, we may want to look back the operator latency in past month for example, if this label is gone, we cannot do that.