rpc: retain information about failed connections

[tbg]

Prior to this change, rpc.Context did not remember a connection after it
failed. A connection attempt would be created whenever we didn't have a
functioning connection to the given (nodeID, targetAddress, class) and
callers would be multiplexed onto it as long as that attempt was not known to
have failed.

The main problem with this setup was that it makes it difficult to provide good
observability because a remote node that just hasn't been dialed yet and one
that is down but between attempts looks the same. We could write code that
periodically tries to fully connect all nodes in the cluster to each other, but
even then exporting good metrics is challenging and while we're currently
comfortable with a densely connected cluster, that may change over time.

An additional challenge was presented by circuit breaking and logging of
attempts. Without state retained, we were limited to a simple retry scheme with
lots of redundant logging. It wasn't easy to log how long a connection had been
unhealthy (as an unhealthy connection was represented as an absent connection),
so in effect folks had to trawl through logs to grab the corresponding
timestamps of first and last failure.

Another piece of complexity were the RPC circuit breakers. These were
implemented at the NodeDialer-level (i.e. one circuit breaker per
(NodeID,Class)) but kept in the rpc.Context (which generally operates on
(NodeID, Addr, Class) because gossip also used them. The library they were
using uses recruitment of client traffic, which also isn't ideal as it could
periodically subject a SQL query to a timeout failure. We'd really prefer it
if the breakers probed without user traffic recruitment.

This PR addresses the above shortcomings:

• state is now retained across restarts on a (NodeID, Addr, Class) basis in
  the `rpc.Context. This has a few complications, for example we need to handle
  decommissioned nodes, as well as nodes that restart under a new listening
  address.
• the NodeDialer-level circuit breakers are removed.
• we're no longer recruiting user traffic for probes. Instead, we adopt the
  util/circuit package already used for Replica-level circuit breaking. This
  library uses an async background probe to determine when to heal the breaker.

I also think the amount of incidental complexity in the rpc package is greatly
reduced as a result of this work.

I ran the failover/non-system/ suite of tests to validate this change. The results
look good¹.

Epic: CRDB-21710
Release note (ui change): a "Networking" metrics dashboard was added.
Release note (ops change): CockroachDB now exposes the following metrics:

• rpc.connection.{un,}healthy: Gauge of healthy/unhealthy connections,
• rpc.connection.{un,}healthy_nanos: Gauge of aggregate duration the currently {un,}healthy connections have been {,un}healthy for,
• grpc.connection.avg_round_trip_latency: sum of weighted moving averages of round-trip latencies for all peers (it really only makes sense to consume these on a per-peer basis via prometheus)
• Counters rpc.connection.heartbeats and rpc.connection.failures.
  When the server.child_metrics.enabled cluster setting is set, these metrics export per-peer values, meaning they can be used to derive a connectivity graph.
  Release note (ops change): logging for intra-node RPC connection failures has improved by reducing frequently and improving information content.

Footnotes

1. https://gist.github.com/tbg/5f5c716fa2fe5a21eca4a0c3a4a35069 ↩

[cockroach-teamcity]

This change is 

[data-matt]

Folks, any chance of backporting this flag to 22.2? thanks

[tbg]

No, this is a major change. We won't be able to backport it.

[data-matt]

@tbg , the cluster setting server.child_metrics.enabled will be in 23.1+ ? I see references to 23.1 here.

Thanks cc @florence-crl

[erikgrinaker]

@tbg , the cluster setting server.child_metrics.enabled will be in 23.1+ ? I see references to 23.1 here.

The setting server.child_metrics.enabled has existed for years, and will enable additional Prometheus metrics for possibly-high-cardinality metrics. For example, changefeeds appear to use them to expose additional per-changefeed metrics.

However, prior to 23.2, this setting does not expose additional RPC metrics. In 23.2, it will also expose per-peer connection metrics as described in the release notes in this PR. This does not happen in 23.1, and can not since the necessary changes are too invasive.

The 23.1 reference you see here was for a minor bugfix that was extracted out of this PR, see #105626. It is not related to the functionality you're interested in.

[data-matt]

Thanks for the clear answer @erikgrinaker.