Add a "CQL Connections (creation) rate" graph

[vladzcloudius]

System information

• Scylla version (you are using): relevant to every scylla version
• Are you willing to contribute it (Yes/No): No

Describe the feature and the current behavior/state.
Many times the graph showing the rate with which new TCP (CQL) connections are being opened is crucial for understanding certain class of issues. In particular issues related to Client-DB connection establishment.

Today we only have a "(Total) CQL Clients connections" graph
We should add a "New CQL Connections Rate" graph alongside the one above.

Who will benefit with this feature?
Everybody who's debugging issues related to connections establishment or tries to analyze the amount of new connections attempts during a particular time period.

[mykaul]

I'm not sure we have such a metric in Scylla core.

[vladzcloudius]

I'm not sure we have such a metric in Scylla core.

scylla_transport_cql_connections

[amnonh]

@vladzcloudius I think what you want to see is the rate of connections that are being created (e.g., the total number of opened connection stay the same, but connections are being open and closed all the time) not the growth in the number of connections (we can do the later, we don't have a metric for the first and it should be added to Scylla).

[vladzcloudius]

@vladzcloudius I think what you want to see is the rate of connections that are being created (e.g., the total number of opened connection stay the same, but connections are being open and closed all the time) not the growth in the number of connections (we can do the later, we don't have a metric for the first and it should be added to Scylla).

Correct, @amnonh. This is exactly what the GH issue name states ;)
And we do have a metric for that (we always had) - I referenced it here: #2053 (comment)

The PromQL expression can be something like:
rate(scylla_transport_cql_connections{<filters go here>}[2m])