VMSS setting doesn't work with aks-engine v0.28.1 and kubernetes v1.12

[gowthamboddapati]

Is this a request for help?
yes

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine/aks-engine?
v0.28.1

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes

What happened ?
Can't deploy a cluster using agent availability profile "VirtualMachineScaleSets"

What you expected to happen?
cluster to deploy

How to reproduce it (as minimally and precisely as possible)?

run the deployment using this command:
aks-engine generate $k8s_json_file_path --output-directory $outputDirectoryPath

$template_file= Join-Path -Path $outputDirectoryPath -ChildPath "azuredeploy.json"
$parameter_file = Join-Path -Path $outputDirectoryPath -ChildPath "azuredeploy.parameters.json"

az group deployment create -name $ClusterName --resource-group $ResourceGroupName --template-file $template_file --parameters $parameter_file

Anything else we need to know:
I am trying to deploy windows clusters with agent availability profile VirtualMachineScaleSets. But I couldn't get it working. I am using aks-engine v0.28.1 and kubernetes v1.12. I get the below mentioned error. This looks similar to the incident that is resolved.

Error:
Azure Error: InvalidTemplate
Message: Deployment template validation failed: 'The template variable 'windowspool2Offset' is not found. Please see https://aka.ms/arm-template/#variables for usage details.'.

kubernetes.json used to generate deployment templates with aks-engine v0.28.1.
{
"apiVersion":"vlabs",
"properties":{
"orchestratorProfile":{
"orchestratorType":"Kubernetes",
"orchestratorRelease":"1.12",
"kubernetesConfig":{
"networkPolicy":"azure",
"useManagedIdentity":true
}
},
"masterProfile":{
"count":1,
"dnsPrefix":"clustername",
"vmSize":"Standard_D2_v2"
},
"agentPoolProfiles":[
{
"name":"windowspool2",
"count":2,
"vmSize":"Standard_D2_v2",
"availabilityProfile":"AvailabilitySet",
"osType":"Windows",
"osDiskSizeGB":127
}
],
"windowsProfile":{
"adminUsername":"place-holder",
"adminPassword":"pwd-placeholder",
"windowsPublisher":"MicrosoftWindowsServer",
"windowsOffer":"WindowsServerSemiAnnual",
"windowsSku":"Datacenter-Core-1803-with-Containers-smalldisk"
},
"linuxProfile":{
"adminUsername":"azureuser",
"ssh":{
"publicKeys":[
{
"keyData":"placeholder"
}
]
}
},
"servicePrincipalProfile":{
"clientId":"placeholder",
"secret":"placeholder"
}
}
}

Could some one let me know if I am doing something wrong in kubernetes.json (mentioned above)?

[CecileRobertMichon]

@gowthamboddapati this apimodel has "availabilityProfile":"AvailabilitySet", is that meant to be VirtualMachineScaleSets? Can you please double check that the apimodel that you provided is what you used to deploy so we can get an accurate repro?

[CecileRobertMichon]

I switched the availabilityProfile to VirtualMachineScaleSets but was not able to repro with 0.28.1, my deployment was successful.

[gowthamboddapati]

@CecileRobertMichon Yes the availabilityProfile is VirtualMachineScaleSets. I retried again I get the same error:

INFO[0000] Generating assets into E:\output-01-23-2019-07-19-34...
[BaseFile] E:\output-01-23-2019-07-19-34\azuredeploy.json
[PatchFile] E:\output-01-23-2019-07-19-34\geneva.azuredeploy.json
[ResultFile] E:\output-01-23-2019-07-19-34\azuredeploy.json
[Patch:Selector] JsonPath:parameters
[Patch:JsonMerge] Node: parameters
[Patch:Selector] JsonPath:resources
[Patch:JsonMerge] Node: resources
[Patch:Selector] JsonPath:resources[?(@.tags.poolName == 'windowspool2')].properties.osProfile
[BaseFile] E:\output-01-23-2019-07-19-34\azuredeploy.parameters.json
[PatchFile] E:\01-23-2019-07-19-34\geneva.azuredeploy.parameters.json
[ResultFile] E:\output-01-23-2019-07-19-34\azuredeploy.parameters.json
[Patch:Selector] JsonPath:parameters
[Patch:JsonMerge] Node: parameters
Azure Error: InvalidTemplate
Message: Deployment template validation failed: 'The template variable 'windowspool2Offset' is not found. Please see https://aka.ms/arm-template/#variables for usage details.'.

Do you know if there is an error log that might indicate what might be wrong besides saying windowspool2offset is not found ?

[CecileRobertMichon]

Can you please take a look at the azuredeploy.json file that was generated in your _output directory and search for instances of windowspool2Offset? Please let me know which resourceType references it.

[gowthamboddapati]

This is the section is azuredeploy.json that has some reference to windowspool2offset.

{
"apiVersion": "2018-06-01",
"name": "[concat(variables('windowspool2VMNamePrefix'), copyIndex(variables('windowspool2Offset')), '/node-monitor')]",
"type": "Microsoft.Compute/virtualMachines/extensions",
"copy": {
"count": "[sub(variables('windowspool2Count'), variables('windowspool2Offset'))]",
"name": "loop"
},
"location": "[variables('location')]",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', variables('windowspool2VMNamePrefix'), copyIndex(variables('windowspool2Offset')))]"
],
"tags": {
"displayName": "dscExtension"
},
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "[parameters('DSCVersion')]",
"autoUpgradeMinorVersion": "[parameters('DSCAutoUpdate')]",
"settings": {
"configuration": {
"url": "[parameters('DSCModulesUrl')]",
"script": "[parameters('DSCScript')]",
"function": "[parameters('DSCFunction')]"
},
"WmfVersion": "latest",
"Privacy": {
"DataCollection": "Enable"
}
},
"protectedSettings": {
.
.
.
.
}
}
}
}

[CecileRobertMichon]

@gowthamboddapati what's the dscExtension? It's not in your apimodel... That's what's causing the issue.

[gowthamboddapati]

Closing the ticket. Sorry haven't mentioned it earlier. We are installing dsc as windows extension to have some software installed on the node. This settings gets appended to the azuredeploy.json before deploying to cluster. The settings we are using for DSC is not comaptiable with VMSS. We have to use different settings : https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-dsc.

Thanks for all the help Cecile.