Add specific stream protocol counters to track protocol errors

[dcorbacho]

Types of Changes

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)

Checklist

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• All tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in related repositories

[gerhard]

This can now be rebased onto master, #3136 is in. Thank you @michaelklishin 👍🏻

[gerhard]

If these are counters, then Prometheus format requires them to end with _total :

Note that an accumulating count has total as a suffix, in addition to the unit if applicable.

https://prometheus.io/docs/practices/naming/

[gerhard]

Apart from the above _total change, this is missing before we can merge:

• add metric to deps/rabbitmq_prometheus/metrics.md under GLOBAL COUNTERS

@Gsantomaggio do you want to test pivotalrabbitmq/rabbitmq:stream-protocol-counters-otp-max with https://github.com/rabbitmq/rabbitmq-stream-go-client/blob/main/examples/publishersError/publisherError.go ?

After that, if you want Gabriele, we can add these to the new RabbitMQ-Streams dashboard 😉

[Gsantomaggio]

Sure !

[dcorbacho]

@gerhard Those test suites can't check for the new metrics, as they only show up when the stream protocol is used. There is a check for them on rabbitmq_stream/test/rabbit_stream_SUITE

[gerhard]

All looks good, starting QA in 3.9 LRE

[gerhard]

While this works, writing queries for these metrics is really difficult. This is what a single query looks like:

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_access_refused_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_authentication_failure_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_frame_too_large_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_internal_error_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_precondition_failed_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_publisher_does_not_exist_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_sasl_authentication_failure_loopback_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_sasl_challenge_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_sasl_error_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_sasl_mechanism_not_supported_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_stream_already_exists_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_stream_does_not_exist_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_stream_not_available_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_subscription_id_already_exists_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_subscription_id_does_not_exist_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_unknown_frame_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}) +

sum by(rabbitmq_cluster) (rabbitmq_global_stream_error_vhost_access_failure_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"})

Could we capture the error type in a label, so that we would end up with metrics like these:

# TYPE rabbitmq_global_stream_error_total counter
# HELP rabbitmq_global_stream_error_total Total number of errors
rabbitmq_global_stream_error_total{protocol="stream",error_type="access_refused"} 0
rabbitmq_global_stream_error_total{protocol="stream",error_type="sasl_error"} 0
rabbitmq_global_stream_error_total{protocol="stream",error_type="authentication_failure"} 0
...

This would transform the above query into this:

sum by(rabbitmq_cluster, error_type) (rabbitmq_global_stream_error_total{protocol="stream"} * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"})

As I mentioned above, while queries are difficult, this works and produces the following end-result:

[gerhard]

Having discussed this offline, it would be too difficult to capture the error types as labels. Leaving as is & merging. Thank you @dcorbacho!

[gerhard]

Backported to v3.9.x