Add "limiter" support to database service

[jakule]

This PR implements DB limiting (part of #7893).

Rate/connection-limiting can be enabled by overriding the values in a configuration YAML:

teleport:
    connection_limits:
      max_connections: 2

Example error reported when a connection has been rejected when the connection to Postgres:

tsh db connect --db-user=postgres --db-name=postgres prod
psql: error: connection to server at "localhost" (::1), port 58128 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 58128 failed: exceeded connection limit
ERROR: exit status 2

[smallinsky]

Right now the limit is implemented only in db/app agent and the connection in DB/APP passed to HandleConnection func: svc/db/Server.HandleConnection(conn net.Conn)
is actually part of reverse tunnel connection established from db/app agent to the teleport proxy.

Thus the conn.RemoteAddr().String() call in reverse tunnel connection handler conn.RemoteAddr() will return teleport proxy reverse tunnel address instead of factual remote DB client address.

https://github.com/gravitational/teleport/blob/master/lib/reversetunnel/agent.go#L382

 (a *Agent) run() {
  // reverse tunnel connection to teleport proxy
  conn, err := a.connect()
  ...
  err = a.processRequests(conn)
}

(a *Agent) processRequests(conn *ssh.Client) error {
...
t := &transport{
   ...
    sconn:               conn.Conn,
    server:              a.Server,
   ...
  }
  go t.start()
}

(p *transport) start() {
...
case LocalNode: 
   p.server.HandleConnection(sshutils.NewChConn(p.sconn, p.channel))
}

where p.server.HandleConnection is case of db agent is svc/db/Server.HandleConnection

If we want to limit external db/app users based on IP the limit should be implemented in proxy app/db handlers:

teleport/lib/srv/db/proxyserver.go

Line 55 in 5cd7c3c

type ProxyServer struct {

[r0mant]

@smallinsky @jakule We have client IP in the identity so it should be available to app/db agents for connections coming out of reverse tunnel: https://github.com/gravitational/teleport/blob/v8.0.0/lib/tlsca/ca.go#L126. We should limit by this instead of RemoteAddr. @jakule Also, please verify this with a real cluster (in addition to unit tests).

[greedy52]

@smallinsky @jakule We have client IP in the identity so it should be available to app/db agents for connections coming out of reverse tunnel: https://github.com/gravitational/teleport/blob/v8.0.0/lib/tlsca/ca.go#L126. We should limit by this instead of RemoteAddr. @jakule Also, please verify this with a real cluster (in addition to unit tests).

Just curious what address shows up as client IP. Is it the public IP that server receives (for a regular production setup)?

[smallinsky]

Just curious what address shows up as client IP. Is it the public IP that server receives (for a regular production setup)?

I think that right now the identity clientIP is not filled in case of db and app cert:
DB Cert
AppSession Cert

And is only used in MFA flow:

teleport/lib/auth/grpcserver.go

Line 2085 in c5b2da1

clientPeer, ok := peer.FromContext(ctx)

[jakule]

@r0mant I think that @smallinsky is right. I also don't see clientIP being set anywhere in the DB module.

Should I set that field and use it for the limiting?

[r0mant]

@jakule Yeah, I think we should do it. I would also see what the current behavior is, to make sure. I.e. for connections coming out of reverse tunnel - what is the remote addr? It should be that of the proxy but wouldn't hurt to verify.

@greedy52

Just curious what address shows up as client IP.

The client IP is the remote address of the connecting client (tsh, psql, etc) from the perspective of the server. As to what it is exactly, it depends a lot on the network configuration and how client's connection reaches the server.

If client and server are on the same network, then in most cases it would just be the client machine's IP address. If, say, client is on a private network and dials a Teleport server on a different network through a gateway, it would most likely be NAT'd address of the gateway.

If there's a some sort of load balancer in front of Teleport proxy, then the client address will be the IP of the load balancer, unless Proxy Protocol is enabled on the LB - @jakule would be nice to test this scenario with proxy protocols as well once we make this change (database proxy service supports proxy protocol so should be just a matter of using something like AWS NLB or even HAProxy locally).

[greedy52]

The client IP is the remote address of the connecting client (tsh, psql, etc) from the perspective of the server. As to what it is exactly, it depends a lot on the network configuration and how client's connection reaches the server.

Thanks for the explanation!

The reason I was asking about this is that many corps use VPN so their clients may share a few (or one) IPs to the public.

[jakule]

@r0mant As we talked I reduced changes here to DB only. I tested the rate limiting in a few different configurations (including Postgres and MongoDB). As you also asked I checked proxy protocol with Nginx and I was able to see the real IP address.

[smallinsky]

The logic LGTM though I have left some minor comment and question if limiter.NewLimiter should be used instead of limiter. NewConnectionsLimiter to support rate limiting.

[r0mant]

Should this be moved above right after the engine is created so if an error happens during monitorConn or InitializeConnection, it would also be sent to the client properly?

[jakule]

As I mentioned below, an error cannot be sent before InitializeConnection() is called.

[r0mant]

Is this separate InitializeConnection method needed? Can't we set clientConn to the engine e.g. in dispatch method when creating an engine? Or is it needed to perform some initialization actions (e.g. handleStartup for Postgres), otherwise error won't be sent properly?

[jakule]

@r0mant Answering your comments. InitializeConnection() is needed (mainly for postgres) to initialize connection with a client otherwise the communication blocks as client and DB agent tries to send data at the same time. Because of that we cannot send an error that comes from InitializeConnection() as connection is not ready.
I think that the best what I can do is to move InitializeConnection() into dispatch(), so after an engine is created, the connection is ready and we're able to send a message.