Flag for exclusive usage of template infos (with warnings)

[bpineau]

This is a second attempt at #1021 (might also provides an alternate solution for #2892 and #3608).

Per #1021 discussion, a flag might be acceptable, if defaulting to false and describing the limitations (not all cloud providers) and the risk of using it (loss of accuracy, risk of upscaling unusable nodes or leaving pending pods).

Some uses cases includes:

• Balance Similar when uspscaling from zero (but I believe Fix BalanceSimilarNodeGroups when scaling from zero #3608 is a better approach)
• Edited/updated ASGs/MIGs taints and labels
• Updated instance type

Per previous discussion, the later two cases could be covered in the long term by custom node-shape discovery Processors (open to discuss that option too).

[bpineau]

@losipiuk and @MaciekPytel as you took part to the previous attempt discussion.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bpineau
To complete the pull request process, please assign feiskyer after the PR has been reviewed.
You can assign the PR to them by writing /assign @feiskyer in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mwielgus]

Do we still need this PR? If yes, please resolve conflicts.

[bpineau]

@mwielgus yes, rebased

[MaciekPytel]

I wonder what's the relationship between this and #3761? It seems that you'd want to use one or the other? If so maybe we can consider packing this into another NodeInfo processor? I think the implementation would be pretty much a cut-paste of the change in this PR.
I know this one doesn't add as much complexity as #3761, but core/ is already one huge code spaghetti and I'm trying to limit changes to it as much as possible to avoid bloating it even further.

[bpineau]

Yes, using #3671 should progressively replace this feature for us (once find alternatives to the habit of updating instance types in place, which came as a side effect of this initially "fix balancing + scale from zero" change, ie. as for #1021 original motivation, but as a second effect here). I can definitely look at making this a Nodeinfo processor.

[mwielgus]

Ok then let's use #3671 and do it as NodeInfo processor/function. Closing this PR in wait for the new one, but feel free to reopen if you prefer to refactor this one.

[bpineau]

Re-did it it as NodeInfo processor/function, in PR #4000, would you mind having a look?