Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Add availability zone support for masters #3864

Merged
merged 4 commits into from
Sep 20, 2018
Merged

Add availability zone support for masters #3864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
55837cf
Add availability zone support for master
ritazh Sep 18, 2018
3a6821c
Address comments
ritazh Sep 18, 2018
ee22109
Address more comments
ritazh Sep 19, 2018
7151e10
defaults n validates
ritazh Sep 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/clusterdefinition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ To learn more about supported orchestrators and versions, run the orchestrators
| gcLowThreshold | no | Sets the --image-gc-low-threshold value on the kublet configuration. Default is 80. [See kubelet Garbage Collection](https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/) |
| kubeletConfig | no | Configure various runtime configuration for kubelet. See `kubeletConfig` [below](#feat-kubelet-config) |
| kubernetesImageBase | no | Specifies the base URL (everything preceding the actual image filename) of the kubernetes hyperkube image to use for cluster deployment, e.g., `k8s.gcr.io/` |
| loadBalancerSku | no | Sku of Load Balancer and Public IP. Candidate values are: `basic` and `standard`. If not set, it will be default to basic. Requires Kubernetes 1.11 or newer. NOTE: VMs behind ILB standard SKU will not be able to access the internet without ELB configured with at least one frontend IP as described in the [standard loadbalancer outbound connectivity doc](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview#control-outbound-connectivity). For Kubernetes 1.11, We have created an external loadbalancer service in the kube-system namespace as a workaround to this issue. Starting k8s 1.12, instead of creating an ELB service, we will setup outbound rules in ARM template once the API is available. |
| loadBalancerSku | no | Sku of Load Balancer and Public IP. Candidate values are: `basic` and `standard`. If not set, it will be default to basic. Requires Kubernetes 1.11 or newer. NOTE: VMs behind ILB standard SKU will not be able to access the internet without ELB configured with at least one frontend IP as described in the [standard loadbalancer outbound connectivity doc](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview#control-outbound-connectivity). For Kubernetes 1.11 and 1.12, We have created an external loadbalancer service in the kube-system namespace as a workaround to this issue. Starting k8s 1.13, instead of creating an ELB service, we will setup outbound rules in ARM template once the API is available. |
| networkPlugin | no | Specifies the network plugin implementation for the cluster. Valid values are:<br>`"azure"` (default), which provides an Azure native networking experience <br>`"kubenet"` for k8s software networking implementation. <br> `"flannel"` for using CoreOS Flannel <br> `"cilium"` for using the default Cilium CNI IPAM |
| networkPolicy | no | Specifies the network policy enforcement tool for the cluster (currently Linux-only). Valid values are:<br>`"calico"` for Calico network policy.<br>`"cilium"` for cilium network policy (Lin), and `"azure"` (experimental) for Azure CNI-compliant network policy (note: Azure CNI-compliant network policy requires explicit `"networkPlugin": "azure"` configuration as well).<br>See [network policy examples](../examples/networkpolicy) for more information. |
| privateCluster | no | Build a cluster without public addresses assigned. See `privateClusters` [below](#feat-private-cluster). |
Expand Down Expand Up @@ -508,7 +508,7 @@ A cluster can have 0 to 12 agent pool profiles. Agent Pool Profiles are used for
| ---------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| availabilityProfile | no | Supported values are `VirtualMachineScaleSets` (default, except for Kubernetes clusters before version 1.10) and `AvailabilitySet`. |
| count | yes | Describes the node count |
| availabilityZones | no | To protect your cluster from datacenter-level failures, you can provide Availability Zones for each of your agentPool. Only applies to Kubernetes clusters version 1.12+. Supported values are arrays of strings, each representing a supported availability zone in a region for your subscription. e.g. `"availabilityZones": ["1","2"]` represents zone 1 and zone 2 can be used. To get supported zones for a region in your subscription, run `az vm list-skus --location centralus --query "[?name=='Standard_DS2_v2'].[locationInfo, restrictions"] -o table`. You should see values like `'zones': ['2', '3', '1']` appear in the first column. If `NotAvailableForSubscription` appears in the output, then you need to create an Azure support ticket to enable zones for that region. Note: For availability zones, only standard load balancer is supported. ([Availability zone example](../examples/e2e-tests/kubernetes/zones)). |
| availabilityZones | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring `"availabilityZones"` for the master profile and all of the agentPool profiles in the cluster definition. This feature only applies to Kubernetes clusters version 1.12+. Supported values are arrays of strings, each representing a supported availability zone in a region for your subscription. e.g. `"availabilityZones": ["1","2"]` represents zone 1 and zone 2 can be used. To get supported zones for a region in your subscription, run `az vm list-skus --location centralus --query "[?name=='Standard_DS2_v2'].[locationInfo, restrictions"] -o table`. You should see values like `'zones': ['2', '3', '1']` appear in the first column. If `NotAvailableForSubscription` appears in the output, then create an Azure support ticket to enable zones for that region. NOTE: To ensure high availability, each profile must define at least two nodes per zone. e.g. An agent pool profile with 2 zones: `"availabilityZones": ["1","2"]` must have at least 4 nodes total with `"count": 4`. When `"availabilityZones"` is configured, the `"loadBalancerSku"` will default to `Standard` as Standard LoadBalancer is required for availability zones. |
| singlePlacementGroup | no | Supported values are `true` (default) and `false`. Only applies to clusters with availabilityProfile `VirtualMachineScaleSets`. `true`: A VMSS with a single placement group and has a range of 0-100 VMs. `false`: A VMSS with multiple placement groups and has a range of 0-1,000 VMs. For more information, check out [virtual machine scale sets placement groups](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-placement-groups). |
| scaleSetPriority | no | Supported values are `Regular` (default) and `Low`. Only applies to clusters with availabilityProfile `VirtualMachineScaleSets`. Enables the usage of [Low-priority VMs on Scale Sets](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-use-low-priority). |
| scaleSetEvictionPolicy | no | Supported values are `Delete` (default) and `Deallocate`. Only applies to clusters with availabilityProfile of `VirtualMachineScaleSets` and scaleSetPriority of `Low`. |
Expand Down
10 changes: 8 additions & 2 deletions examples/e2e-tests/kubernetes/zones/definition.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,21 @@
"orchestratorRelease": "1.12"
},
"masterProfile": {
"count": 1,
"count": 5,
"dnsPrefix": "",
"vmSize": "Standard_DS2_v2"
"vmSize": "Standard_DS2_v2",
"availabilityProfile": "VirtualMachineScaleSets",
"availabilityZones": [
"1",
"2"
]
},
"agentPoolProfiles": [
{
"name": "agentpool",
"count": 4,
"vmSize": "Standard_DS2_v2",
"availabilityProfile": "VirtualMachineScaleSets",
"availabilityZones": [
"1",
"2"
Expand Down
18 changes: 16 additions & 2 deletions parts/k8s/kubernetesmasterresourcesvmss.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,22 +186,29 @@
},
{{end}}
{
"apiVersion": "[variables('apiVersionDefault')]",
"apiVersion": "2018-04-01",
"location": "[variables('location')]",
"name": "[variables('masterPublicIPAddressName')]",
"properties": {
"dnsSettings": {
"domainNameLabel": "[variables('masterFqdnPrefix')]"
},
{{ if .MasterProfile.HasAvailabilityZones}}
"publicIPAllocationMethod": "Static"
{{else}}
"publicIPAllocationMethod": "Dynamic"
{{end}}
},
"sku": {
"name": "[variables('loadBalancerSku')]"
},
"type": "Microsoft.Network/publicIPAddresses"
},
{
"type": "Microsoft.Network/loadBalancers",
"name": "[variables('masterLbName')]",
"location": "[variables('location')]",
"apiVersion": "[variables('apiVersionDefault')]",
"apiVersion": "2018-04-01",
"dependsOn": [
"[concat('Microsoft.Network/publicIPAddresses/', variables('masterPublicIPAddressName'))]"
],
Expand Down Expand Up @@ -269,6 +276,9 @@
}
}
]
},
"sku": {
"name": "[variables('loadBalancerSku')]"
}
},
{
Expand All @@ -290,6 +300,9 @@
"poolName": "master"
},
"location": "[variables('location')]",
{{ if .MasterProfile.HasAvailabilityZones}}
"zones": "[parameters('availabilityZones')]",
{{ end }}
"name": "[concat(variables('masterVMNamePrefix'), 'vmss')]",
{{if UseManagedIdentity}}
{{if UserAssignedIDEnabled}}
Expand All @@ -311,6 +324,7 @@
"name": "[parameters('masterVMSize')]"
},
"properties": {
"singlePlacementGroup": {{ .MasterProfile.SinglePlacementGroup}},
"overprovision": false,
"upgradePolicy": {
"mode": "Manual"
Expand Down
2 changes: 1 addition & 1 deletion parts/k8s/kubernetesmastervars.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
{{if .HasManagedDisks}}
"apiVersionStorageManagedDisks": "2016-04-30-preview",
{{end}}
{{if .HasVirtualMachineScaleSets}}
{{if .HasVMSSAgentPool}}
"apiVersionVirtualMachineScaleSets": "2017-12-01",
{{end}}
{{if not IsHostedMaster}}
Expand Down
2 changes: 0 additions & 2 deletions parts/k8s/kubernetesmastervarsvmss.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,7 @@
{{if .HasManagedDisks}}
"apiVersionStorageManagedDisks": "2016-04-30-preview",
{{end}}
{{if .HasVirtualMachineScaleSets}}
"apiVersionVirtualMachineScaleSets": "2017-12-01",
{{end}}
{{if not IsHostedMaster}}
{{if .MasterProfile.IsStorageAccount}}
"masterStorageAccountName": "[concat(variables('storageAccountBaseName'), 'mstr0')]",
Expand Down
8 changes: 8 additions & 0 deletions parts/masterparams.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,14 @@
"type": "string"
},
{{end}}
{{if .MasterProfile.HasAvailabilityZones}}
"availabilityZones": {
"metadata": {
"description": "Master availability zones"
},
"type": "array"
},
{{end}}
{{end}}
{{if IsHostedMaster}}
"masterSubnet": {
Expand Down
33 changes: 24 additions & 9 deletions pkg/acsengine/defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,12 +237,19 @@ func setPropertiesDefaults(cs *api.ContainerService, isUpgrade, isScale bool) (b
if cs.Properties.MasterProfile != nil {
setMasterProfileDefaults(properties, isUpgrade)
}
// Set VMSS Defaults for Masters
if cs.Properties.MasterProfile != nil && cs.Properties.MasterProfile.IsVirtualMachineScaleSets() {
setVMSSDefaultsForMasters(properties)
}

setAgentProfileDefaults(properties, isUpgrade, isScale)

setStorageDefaults(properties)
setExtensionDefaults(properties)
setVMSSDefaults(properties)
// Set VMSS Defaults for Agents
if cs.Properties.HasVMSSAgentPool() {
setVMSSDefaultsForAgents(properties)
}

// Set hosted master profile defaults if this cluster configuration has a hosted control plane
if cs.Properties.HostedMasterProfile != nil {
Expand Down Expand Up @@ -422,11 +429,11 @@ func setOrchestratorDefaults(cs *api.ContainerService, isUpdate bool) {
a.OrchestratorProfile.KubernetesConfig.UseInstanceMetadata = helpers.PointerToBool(api.DefaultUseInstanceMetadata)
}

if a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "" {
if !a.HasAvailabilityZones() && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "" {
CecileRobertMichon marked this conversation as resolved.
Show resolved Hide resolved
a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku = api.DefaultLoadBalancerSku
}

if common.IsKubernetesVersionGe(a.OrchestratorProfile.OrchestratorVersion, "1.11.0") && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "Standard" {
if common.IsKubernetesVersionGe(a.OrchestratorProfile.OrchestratorVersion, "1.11.0") && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "Standard" && a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB == nil {
a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB = helpers.PointerToBool(api.DefaultExcludeMasterFromStandardLB)
}

Expand Down Expand Up @@ -584,8 +591,19 @@ func setMasterProfileDefaults(a *api.Properties, isUpgrade bool) {
}
}

// setVMSSDefaults
func setVMSSDefaults(a *api.Properties) {
// setVMSSDefaultsForMasters
func setVMSSDefaultsForMasters(a *api.Properties) {
if a.MasterProfile.SinglePlacementGroup == nil {
a.MasterProfile.SinglePlacementGroup = helpers.PointerToBool(api.DefaultSinglePlacementGroup)
}
if a.MasterProfile.HasAvailabilityZones() && (a.OrchestratorProfile.KubernetesConfig != nil && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "") {
a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku = "Standard"
a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB = helpers.PointerToBool(api.DefaultExcludeMasterFromStandardLB)
}
}

// setVMSSDefaultsForAgents
func setVMSSDefaultsForAgents(a *api.Properties) {
for _, profile := range a.AgentPoolProfiles {
if profile.AvailabilityProfile == api.VirtualMachineScaleSets {
if profile.Count > 100 {
Expand All @@ -594,10 +612,7 @@ func setVMSSDefaults(a *api.Properties) {
if profile.SinglePlacementGroup == nil {
profile.SinglePlacementGroup = helpers.PointerToBool(api.DefaultSinglePlacementGroup)
}
if profile.SinglePlacementGroup == helpers.PointerToBool(false) {
profile.StorageProfile = api.ManagedDisks
}
if profile.HasAvailabilityZones() {
if profile.HasAvailabilityZones() && (a.OrchestratorProfile.KubernetesConfig != nil && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "") {
a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku = "Standard"
a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB = helpers.PointerToBool(api.DefaultExcludeMasterFromStandardLB)
}
Expand Down
Loading