VMSS is broken on 0.17.1

[omartin2010]

Is this a request for help?:yes

---

Is this an ISSUE or FEATURE REQUEST? (choose one):ISSUE

---

What version of acs-engine?:0.17.1

---

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) : Kubernetes

What happened:Can't deploy following the instructions here and following the template included in examples/kubernetes-vmss-low-priority/kubernetes.json

What you expected to happen:cluster to deploy

How to reproduce it (as minimally and precisely as possible):run the deployment using this command:
acs-engine deploy --subscription-id <my_sub> --location eastus --api-model kubernetes-vmss-low-priority/kubernetes.json
I modified kubernetes.json to put the dns name in there, and didn't put the dns name in the deploy command line because it creates a bug as I've seen in issue 3051.

Anything else we need to know:
I'm was able to repro one time. Happy to try again, now I'm using the "longer" version of the process which creates the actual template.

[omartin2010]

i fount out this also seems to be broken with the "long way" version of the process. A missing parameter in the generated parameter file (azuredeploy.parameters.json) seems to be the source of the problem. Here's an output of the error message I forgot to put above :
{
"error": {
"code": "InvalidTemplate",
"message": "Unable to process template language expressions for resource '/subscriptions/321cae6f-e4d3-40bf-824f-c07493a62af5/resourceGroups/acs-cluster-1/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-24266323-vmss' at line '1' and column '56819'. 'The template variable 'agentpool1Offset' is not found. Please see https://aka.ms/arm-template/#variables for usage details.'"
}
Tried adding manually the parameter (agentpool1Offset) in the parameter file, but I get an error message because that's not part of the allowed parameters.

[CecileRobertMichon]

@omartin2010 please see #3064. Please use version v0.17.0 in the meantime.

[omartin2010]

I did try using v0.17.0 and experimented another issue :
➜ az group deployment create --name acs-cluster-1-deployment -g acs-cluster-1 --template-file ./azuredeploy.json --parameters ./azuredeploy.parameters.json
Deployment failed. Correlation ID: 365b4205-2ac2-4a54-841b-086d7b7c2706. {
"error": {
"code": "PropertyChangeNotAllowed",
"target": "customData",
"message": "Changing property 'customData' is not allowed."

So I deployed with 0.16.2 and it seems to be working!

[CecileRobertMichon]

@omartin2010 try using a different dns prefix

[navicore]

same here, The template variable 'agentpool1Offset' is not found. with vmss and pre-existing vnet and v0.17.1 but works with v0.17.0.

[LaurentLesle]

I managed to get it working by using ACS-engine version 0.17.0 and kubernetes 1.10.2. I deployed the VMSS into existing vnet.
Any version greater than 0.17.0 fails with same errors as described above

[mwieczorek]

I also got this bug today when trying to deploy k8s on vmss with cluster autoscaler addon.
Unfortunately I need autoscaler addon in v1.2.2 (kubernetes/autoscaler#881), which is in 0.17.1...so I can wait only for 0.17.2 :)

[feiskyer]

The issue has been fixed in master branch, it needs a while for next v0.17.2 release

[CecileRobertMichon]

@mwieczorek 893c2c0 is in master, not in 0.17.1, which only added a new k8s version and fixed a bug with extensions.

[CecileRobertMichon]

This is fixed in 0.18.x

[sshepel]

@CecileRobertMichon not working in 0.19.2

[CecileRobertMichon]

@sshepel our tests suggest that VMSS is working on the latest acs-engine version, can you give more details regarding what exactly is not working / what error message you are getting / with what apimodel?

[sshepel]

@CecileRobertMichon I'm getting same error as @omartin2010 when I'm trying to update cluster:

az group deployment create --name acs-cluster-deployment -g acs-cluster --template-file ./azuredeploy.json --parameters ./azuredeploy.parameters.json
Deployment failed. Correlation ID: 705c5201-2zk3-4a54-151b-086d8g5q5096. {
"error": {
"code": "PropertyChangeNotAllowed",
"target": "customData",
"message": "Changing property 'customData' is not allowed."

[CecileRobertMichon]

@sshepel this is a separate issue, you are trying to modify an existing cluster using a different template than it was created with. You should either a) create a new cluster using a dns prefix / name b) use acs-engine upgrade or scale if you are trying to change an existing cluster or c) re-deploy using the az group deployment create command but without re-generating the template file or modifying it.

[sshepel]

@CecileRobertMichon I'm sorry, I just want to provide the log error (because it was the same for me) not the command.

I faced with this problem duding executing az group deployment create command for updating the cluster... And looks like when cluster was deployed with "useManagedIdentity": true and you wan to re-deploy some node pool (for example without "scaleSetPriority": "Low") the role assignment for the pool can not be updated...