Database connections don't correctly fallback to healthy agent in leaf clusters

[r0mant]

Description

What happened:

After a database agent in a leaf cluster has been shut down and another one brought up (by AWS ASG), a user was getting the following error trying to connect to the database:

$ tsh db connect --db-user=alice example
ERROR 1105 (HY000): connection rejected: failed dialing through tunnel (dial tcp: address @local-node: missing port in address) or directly (no tunnel connection found: no db reverse tunnel for ef6cd978-f703-4a92-9621-f0f8fbc326c0.example found)
ERROR: exit status 1

This only affects HA fallback for agents connected to a leaf cluster. When connecting to agents in the local (root) cluster, it falls back properly.

What you expected to happen:

Teleport proxy to retry the connection on a healthy database agent according to the following logic:

https://github.com/gravitational/teleport/blob/v8.3.1/lib/srv/db/proxyserver.go#L309-L322

Reproduction Steps

1. Have root cluster, leaf cluster and 2 database agents proxying the same database in a leaf cluster.
2. Shut down one of the leaf's database agents.
3. Run tsh db connect a few times to connect to a database in a leaf cluster.

Debug Logs

Feb 25 00:12:18 node teleport[696]: 2022-02-25T00:12:18Z ERRO [DB:PROXY]  Failed to handle MySQL client connection. error:[
Feb 25 00:12:18 node teleport[696]: ERROR REPORT:
Feb 25 00:12:18 node teleport[696]: Original Error: *errors.errorString connection rejected: failed dialing through tunnel (dial tcp: address @local-node: missing port in address) or directly (no tunnel connection found: no db reverse tunnel for 3d5a6e8a-2b15-4dd6-bf51-d2db7c3fff98.example found)
Feb 25 00:12:18 node teleport[696]: Stack Trace:
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/teleport/api/utils/sshutils/conn.go:84 github.com/gravitational/teleport/api/utils/sshutils.ConnectProxyTransport
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/remotesite.go:719 github.com/gravitational/teleport/lib/reversetunnel.(*remoteSite).chanTransportConn
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/remotesite.go:692 github.com/gravitational/teleport/lib/reversetunnel.(*remoteSite).connThroughTunnel
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/remotesite.go:576 github.com/gravitational/teleport/lib/reversetunnel.(*remoteSite).DialTCP
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/remotesite.go:570 github.com/gravitational/teleport/lib/reversetunnel.(*remoteSite).Dial
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/lib/srv/db/proxyserver.go:309 github.com/gravitational/teleport/lib/srv/db.(*ProxyServer).Connect
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/lib/srv/db/mysql/proxy.go:84 github.com/gravitational/teleport/lib/srv/db/mysql.(*Proxy).HandleConnection
Feb 25 00:12:18 node teleport[696]:         /go/src/github.com/gravitational/teleport/lib/srv/db/proxyserver.go:188 github.com/gravitational/teleport/lib/srv/db.(*ProxyServer).ServeMySQL.func1
Feb 25 00:12:18 node teleport[696]:         /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
Feb 25 00:12:18 node teleport[696]: User Message: connection rejected: failed dialing through tunnel (dial tcp: address @local-node: missing port in address) or directly (no tunnel connection found: no db reverse tunnel for 3d5a6e8a-2b15-4dd6-bf51-d2db7c3fff98.example found)] db/proxyserver.go:190