POC: Evaluate using a proxy kube-scheduler to run simulations

[unmarshall]

How to categorize this issue?
/area control-plane
/kind poc
/priority 2

What would you like to be added:
We would like to do a POC on if its possible to delegate the entire simulation to be done by a proxy kube-scheduler which is run solely for taking cluster autoscaling decisions. This is deployed in addtion to the actual kube-scheduler. It will operate on virtual nodes and not real nodes. The simulated decision should then be used by the cluster autoscaling module.

Why is this needed:
The upstream k8s cluster autoscaler uses kube-scheduler packages/APIs to run pod scheduling simulations. As part of that the simulation it only runs the filters and not the scorers. It also has gaps with kube-scheduler (e.g. kubernetes/autoscaler#6227) due to which decision taken by kube-scheduler is not in sync with the one taken by CA which results in pending pods not getting scheduled for longer times.
We see the same issues also in karpenter (e.g. see comment) where simulations done by karpenter and kube-scheduler differ and is always a catch-up game that needs to be played.

[ashwani2k]

/close as completed.