Cannot scale a private cluster

[ams0]

Is this a request for help?:

Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

What version of acs-engine?:

Version: v0.15.0

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes 1.10 private cluster with a jumpbox

What happened:

Scaling operation fails with:

FATA[0004] error tranforming the template for scaling template /Users/alessandro/Desktop/x10/git/acs-engine-templates/ahold/_output/privk8s/apimodel.json: Found 2 resources with type Microsoft.Network/networkSecurityGroups in the template. There should only be 1

Because there are two instances of networkSecurityGroups in the template (one for the jumpbox and one for everything else).

What you expected to happen:

Cluster scales.

How to reproduce it (as minimally and precisely as possible):

Deploy a private cluster with custom vnet and multiple pools and a jumpbox with this template:

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.10",
      "kubernetesConfig": {
        "networkPolicy": "calico",
        "privateCluster": {
          "enabled": true,
          "jumpboxProfile": {
            "name": "acs-engine-jb",
            "vmSize": "Standard_D4s_v3",
            "osDiskSizeGB": 30,
            "storageProfile": "ManagedDisks",
            "username": "azureuser",
            "publicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIOxg+goSYoCIND3IIAjPoPGr7gsux9OQjE5IP2wEU8eMsywgGXBwZXUVjh8NgFHVWZEMTQCAM52P2ipYBup9QhuqWVjH4v0hrj1X/rx7tzlZh2wk3kgVPQwMKCyacQLifqus4quJLSQAPu1ksgxaWEBWnSa0e+DM2D0PYs/j284qOO9T9ULqpb/ZJK9gySa+AfSMhGCskcT/EfE8g1iqC96PajFxGHOBxqiDFtIKPhNiqKYruDhVJYmhAXG6ScHadiXzP3BdiPR66eyCOQtSeIxjnEeJcrZ7vZLFpWQvaaZw+JfPkGGFCsBTn39dfr1awrMtPIPvkj4iU1jkGKzUD [email protected]"
          }
      }
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "privk8s",
      "vnetSubnetId": "/subscriptions/12c7e9d6-967e-40c8-8b3e-4659a4ada3ef/resourceGroups/ACS-Engine-Test-Rg/providers/Microsoft.Network/virtualNetworks/ACS-Engine-Test-vNet/subnets/ACS-Engine-Test-Master",
      "firstConsecutiveStaticIP": "192.168.168.4",
      "vnetCidr": "192.168.168.0/24",
      "storageProfile": "ManagedDisks",
      "vmSize": "Standard_D2_v2"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool1",
        "vnetSubnetId": "/subscriptions/12c7e9d6-967e-40c8-8b3e-4659a4ada3ef/resourceGroups/ACS-Engine-Test-Rg/providers/Microsoft.Network/virtualNetworks/ACS-Engine-Test-vNet/subnets/ACS-Engine-Test-AgentPool1",
        "count": 2,
        "vmSize": "Standard_D2_v2",
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "AvailabilitySet"
      },
      {
        "name": "agentpool2",
        "vnetSubnetId": "/subscriptions/12c7e9d6-967e-40c8-8b3e-4659a4ada3ef/resourceGroups/ACS-Engine-Test-Rg/providers/Microsoft.Network/virtualNetworks/ACS-Engine-Test-vNet/subnets/ACS-Engine-Test-AgentPool2",
        "count": 2,
        "vmSize": "Standard_D2_v2",
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIOxg+goSYoCIND3IIAjPoPGr7gsux9OQjE5IP2wEU8eMsywgGXBwZXUVjh8NgFHVWZEMTQCAM52P2ipYBup9QhuqWVjH4v0hrj1X/rx7tzlZh2wk3kgVPQwMKCyacQLifqus4quJLSQAPu1ksgxaWEBWnSa0e+DM2D0PYs/j284qOO9T9ULqpb/ZJK9gySa+AfSMhGCskcT/EfE8g1iqC96PajFxGHOBxqiDFtIKPhNiqKYruDhVJYmhAXG6ScHadiXzP3BdiPR66eyCOQtSeIxjnEeJcrZ7vZLFpWQvaaZw+JfPkGGFCsBTn39dfr1awrMtPIPvkj4iU1jkGKzUD [email protected]"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    }
  }
}

and the custom vnet with this other:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {  },
  "variables": {  },
  "resources": [
    {
      "apiVersion": "2016-03-30",
      "location": "[resourceGroup().location]",
      "name": "ACS-Engine-Test-vNet",
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "192.168.168.0/22"
          ]
        },
        "subnets": [
          {
            "name": "ACS-Engine-Test-Master",
            "properties": {
              "addressPrefix": "192.168.168.0/24"
            }
          },
          {
            "name": "ACS-Engine-Test-AgentPool1",
            "properties": {
              "addressPrefix": "192.168.169.0/24"
            }
          },
          {
            "name": "ACS-Engine-Test-AgentPool2",
            "properties": {
              "addressPrefix": "192.168.170.0/24"
            }
          }
        ]
      },
      "type": "Microsoft.Network/virtualNetworks"
    }
  ]
}

Anything else we need to know:

[CecileRobertMichon]

Thanks for reporting @ams0, this is indeed a bug... If you want feel free to open a PR to change

acs-engine/pkg/acsengine/transform/transform.go

Line 117 in 4649c8d

if nsgIndex != -1 {

if there is a jumpbox.

[CecileRobertMichon]

As @marrobi pointed out, we also need to figure out how to allow the user to scale a private cluster if there is no jumpbox... The scale command relies on the fqdn to access the cluster and drain the nodes. If the cluster is completely private, how do we drain the nodes?