Optimize tsh db ls performance

[greedy52]

fix #14075

• tsh db ls execution time depends on number of roles assigned to a user #14075

Changes:

• Added new API GetCurrentUserRoles that streams all user roles
• tsh db ls to reuse proxy client
• tsh db ls --all to run fetch in parallel per profile

[smallinsky]

I wonder if we can cover this flow somehow in the tsh test.
For example, we can use our own dialer with custom timeout and inject this in tsh test to simulate low latency between the tsh client and teleport proxy.

[espadolini]

Have we considered the idea of having the allowed database users retrieved and returned as part of ListDatabases instead, to avoid the extra roundtrips to also fetch the rolesets?

sidenote: we don't seem to care about showing the list of database users when we're outputting databases in json format, perhaps we could skip the extra fetches in that case?

[espadolini]

additional question: do we absolutely need all the changes to parallelize ListDatabases and ConnectToProxy/FetchAllClusterRoles instead of opening one less ProxyClient and using the same one for databases and roles?

[espadolini]

Additional point raised by @rosstimothy: we should be careful about parallelizing connections - a cluster with a lot of leaf clusters can cause a massive amount of outbound connections when doing listDatabasesAllClusters.

[smallinsky]

Nice, I have left one comment but aport from that the fix looks good for me.

[rosstimothy]

Looks good to me. We probably want to consider adding some tracing bits to the new stuff added here. It might also be worthwhile to capture traces before and after this change. That would really show the performance gains and also help identify any areas that could still be improved upon.

[greedy52]

Looks good to me. We probably want to consider adding some tracing bits to the new stuff added here. It might also be worthwhile to capture traces before and after this change. That would really show the performance gains and also help identify any areas that could still be improved upon.

@rosstimothy traceability for new changes works out of the box. Old code services.FetchRoles does not take the context so I have to do some local changes to capture. I will leave instrumenting old code for a separate change.

Here is the comparison. Tested against my cluster in AWS.

Before this change

• ConnectToProxy is the most expansive call and we were doing that 2 times.
• GetRole n times for each role my user has. In the original issue, GetRole takes 100ms~200ms against TeleportCloud so they add up quick.

With this change

• ConnectToProxy once
• No longer calling GetSites to figure out current cluster name.
• A single GetCurrentUserRoles

Really love the tracing!

[rosstimothy]

Looks good!

I think that listDatabasesAllClusters might potentially benefit from a span per profile, but that could be added later too I suppose.

[github-actions]

@greedy52 See the table below for backport results.

Branch	Result
branch/v10	Create PR