cluster-autoscaler: allow setting kuberentes client burst and qps to avoid rate limiting

[grosser]

Which component this PR applies to?

cluster-autoscaler

What type of PR is this?

/kind feature

What this PR does / why we need it:

CA Waited for 191.285161ms due to client-side throttling, not priority and fairness, request: POST:[https://172.29.4.1:443/api/v1/namespaces/kube-system/events

• we currently see lots of rate limiting from the kubernetes client of CA, we'd like to give it more QPS to avoid that
• users might want to reduce the number of queries that CA does against their cluster

Does this PR introduce a user-facing change?

- add `--kube-client-burst` and `--kube-client-qps` to control how many requests CA can do against kubernetes

[x13n]

With this change as-is, the rate limiting will actually be 2x than whatever is configured via flags (assuming there's a lot of load on both events and non-events and rate limiting is the bottleneck). Maybe it would be cleaner to use separate flags for events client?

[grosser]

ideally we'd use a single client, since atm it already uses 2x the configured rate, if we hardcode the "there are 2 different clients" into the cli that could be an issue later too

idk if the event or the regular client produces our throttling issues
also if a user sees throttling it would be nice if they only have to turn a single knob and not try 2 different ones

[grosser]

ah the throttling is from /eviction so I could only apply it to non-events if that makes this PR merge-able

[x13n]

Sorry, I just noticed I missed this one. Assigning so I won't miss it again:

/assign

I think setting the rate limits on a single (non-event) client makes sense. The clients were split on purpose so I think merging them back might be problematic. Perhaps nowadays with API priority and fairness it is doable, but I'd need to better understand the implications first. I'll look into this. Meanwhile, let's apply the limits to just one client.

[grosser]

updated it as requested
... not a big fan of each having their own rate limit, but should be fine

[x13n]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: grosser, x13n

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS [x13n]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment