Allow configuration of target CPU Utilization for HPA

[moh-osman3]

Currently the HPA code hard codes 90% as the target cpu utilization. I am currently running tests with HPA and it would be helpful to be able to change the target utilization to a custom value. Opening this issue to track my change to add this field to the collector config.

[pavolloffay]

cc) @kevinearls

[ajsaclayan]

Can I piggy back off this request to scale on memory as well? Otherwise I can create a new issue.

[pavolloffay]

@moh-osman3 could you please as well comment why do you need a custom value fo HPA? What is the use-case?

[moh-osman3]

@pavolloffay The purpose of adding TargetCPUUtilization to the CRD is because we wish to tune this value and provide a recommended setting to users. Previously with 90% we are experiencing a lot of flapping when load testing. If targets are uneven and one of the collectors has significantly fewer targets than other collectors, it becomes less likely that scale up will happen (average utilization will be too low, this can lead to throttling in collectors with too many targets which can cause scaling down). Giving users the control over this value might allow them to tune the HPA to their specific needs, if the default is unhelpful.

Previously talked with @jaronoff97 about adding the option to scale on memory for workloads with fluctuating memory needs. I can work on that in a followup PR in the next few weeks, if you don't mind creating an issue for this @ajsaclayan and specifying the use case?

[moh-osman3]

Resolved by #1066