Kubernetes Cluster Autoscaler (VMSS) addon

[sozercan]

What this PR does / why we need it:

This PR adds Azure Virtual Machine Scale Set cluster autoscaler (https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/azure) as an add-on for acs-engine

It automatically pulls in and converts to base64 all required values for credentials (such as clientId, clientSecret, subscriptionId, tentantId, etc) and other required values (scale set name, resource group, etc) from the deployment.

User can override minNodes (default to 1), maxNodes (default to 5), image, limits, and requests.

Documentation: https://github.com/sozercan/acs-engine/blob/autoscaler-vmss/examples/addons/cluster-autoscaler/README.md

Special notes for your reviewer:
Merge after #2620

If applicable:

• documentation
• unit tests
• tested backward compatibility (ie. deploy with previous version, upgrade with this branch)
• managed identity support

Release note:

Added cluster-autoscaler addon for Virtual Machine Scale Sets

[khenidak]

@kkmsft who is working on Autoscaler for AKS RP. Can we align both work streams?

[codecov]

Codecov Report

Merging #2637 into master will decrease coverage by 0.09%.
The diff coverage is 56.09%.

@@            Coverage Diff            @@
##           master    #2637     +/-   ##
=========================================
- Coverage   49.61%   49.51%   -0.1%     
=========================================
  Files          91       91             
  Lines       13854    13939     +85     
=========================================
+ Hits         6873     6902     +29     
- Misses       6344     6394     +50     
- Partials      637      643      +6

Impacted Files	Coverage Δ
pkg/acsengine/addons.go	100% <100%> (ø)	⬆️
pkg/api/types.go	30.2% <100%> (+2.92%)	⬆️
pkg/acsengine/k8s_versions.go	100% <100%> (ø)	⬆️
pkg/api/vlabs/validate.go	43.96% <16.66%> (-0.54%)	⬇️
pkg/acsengine/engine.go	63.22% <22.22%> (-1.11%)	⬇️
pkg/acsengine/defaults.go	73.7% <71.42%> (-0.04%)	⬇️
pkg/operations/kubernetesupgrade/upgrader.go	56.66% <0%> (-0.84%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25049c1...90a3bcd. Read the comment docs.

[mpalumbo7]

What would it take to extend this to support auto-scaling multiple agent pools? At the moment it only scales the first one, but the cluster-autoscaler supports multiple agent pools.

[feiskyer]

@ewok2030 You can add more agent pools by adding

- --nodes=<kubernetesClusterAutoscalerMinNodes>:<kubernetesClusterAutoscalerMaxNodes>:<kubernetesClusterAutoscalerVMSSName>
- --nodes=<kubernetesClusterAutoscalerMinNodes>:<kubernetesClusterAutoscalerMaxNodes>:<kubernetesClusterAutoscalerVMSSName>

[sozercan]

@ewok2030 like @feiskyer said, you can edit the yaml (in /etc/kubernetes/addons/ on the master) to include multiple agent pools, but that functionality isn't there in the add-on definition logic yet (it'll automatically include the primaryScaleSet which is the first agent pool). I have been thinking about adding some extra configuration options to include multiple agent pools.

[sozercan]

@jackfrancis ready for review when you get a chance

[jackfrancis]

Did we want to delete this const?

[sozercan]

yea, this one is no longer used (since it moved to k8s_versions), thought i would clean it up

[jackfrancis]

I assume we're downgrading addon-resizer to 1.7 because 1.8.1 is not mutually compatible with cluster-autoscaler?

Can we be confident that we won't re-introduce this bug:

#2430

Which was fixed here:

#2753

(the addon-resizer manfest definition was changed with the above PR, not just the version; perhaps the changes in how addon-resizer was expressed is the important part and not the upgrade to 1.8.1?)

[sozercan]

didn't mean to downgrade addon-resizer, probably didn't notice when rebasing. reverted that change.

[jackfrancis]

This looks great, couple questions. Let's add validation unit tests (see pkg/api/vlabs/validate_test.go) to cover your new validations.

Thanks so much for this!!

[sozercan]

@jackfrancis thanks! added validation unit tests. should be good to review when you get a chance

[jackfrancis]

lgtm pending E2E

[robsonpeixoto]

How to change the mix/max after the cluster been created?

[mpalumbo7]

At this time, it’ll have to be done in the same way a second agent pool may be added: by SSH’ing into each master node an editing the addon YAML.