Emit Resource Limits in Metrics Controller

[suket22]

Tell us about your request
Emit metrics for resource usage by the provisioner.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
There's a new counter controller that this PR introduces - #817
It sets the resources being consumed by the provisioner in the provisioner.status.Resources but we should also emit this and create a Grafana dashboard off those metrics.

Are you currently working around this issue?
N/A

Additional context
N/A

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[rajakshay]

Adding my use-case here: Im trying to achieve an equivalent of High ASG utilization monitor in Karpenter. This feature will be useful to get the percentage capacity in use, and proactively update the limits on a provisioner before pods get stuck in a Pending state forever.