Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Commit

Permalink
Add availability zone support for masters (#3864)
Browse files Browse the repository at this point in the history
  • Loading branch information
@ritazh @jackfrancis
ritazh authored and jackfrancis committed Sep 20, 2018
1 parent 277dafb commit 13c91da
Show file tree
Hide file tree
Showing 19 changed files with 643 additions and 105 deletions.
4 changes: 2 additions & 2 deletions docs/clusterdefinition.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ To learn more about supported orchestrators and versions, run the orchestrators
| gcLowThreshold | no | Sets the --image-gc-low-threshold value on the kublet configuration. Default is 80. [See kubelet Garbage Collection](https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/) |
| kubeletConfig | no | Configure various runtime configuration for kubelet. See `kubeletConfig` [below](#feat-kubelet-config) |
| kubernetesImageBase | no | Specifies the base URL (everything preceding the actual image filename) of the kubernetes hyperkube image to use for cluster deployment, e.g., `k8s.gcr.io/` |
| loadBalancerSku | no | Sku of Load Balancer and Public IP. Candidate values are: `basic` and `standard`. If not set, it will be default to basic. Requires Kubernetes 1.11 or newer. NOTE: VMs behind ILB standard SKU will not be able to access the internet without ELB configured with at least one frontend IP as described in the [standard loadbalancer outbound connectivity doc](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview#control-outbound-connectivity). For Kubernetes 1.11, We have created an external loadbalancer service in the kube-system namespace as a workaround to this issue. Starting k8s 1.12, instead of creating an ELB service, we will setup outbound rules in ARM template once the API is available. |
| loadBalancerSku | no | Sku of Load Balancer and Public IP. Candidate values are: `basic` and `standard`. If not set, it will be default to basic. Requires Kubernetes 1.11 or newer. NOTE: VMs behind ILB standard SKU will not be able to access the internet without ELB configured with at least one frontend IP as described in the [standard loadbalancer outbound connectivity doc](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview#control-outbound-connectivity). For Kubernetes 1.11 and 1.12, We have created an external loadbalancer service in the kube-system namespace as a workaround to this issue. Starting k8s 1.13, instead of creating an ELB service, we will setup outbound rules in ARM template once the API is available. |
| networkPlugin | no | Specifies the network plugin implementation for the cluster. Valid values are:<br>`"azure"` (default), which provides an Azure native networking experience <br>`"kubenet"` for k8s software networking implementation. <br> `"flannel"` for using CoreOS Flannel <br> `"cilium"` for using the default Cilium CNI IPAM |
| networkPolicy | no | Specifies the network policy enforcement tool for the cluster (currently Linux-only). Valid values are:<br>`"calico"` for Calico network policy.<br>`"cilium"` for cilium network policy (Lin), and `"azure"` (experimental) for Azure CNI-compliant network policy (note: Azure CNI-compliant network policy requires explicit `"networkPlugin": "azure"` configuration as well).<br>See [network policy examples](../examples/networkpolicy) for more information. |
| privateCluster | no | Build a cluster without public addresses assigned. See `privateClusters` [below](#feat-private-cluster). |
Expand Down Expand Up @@ -508,7 +508,7 @@ A cluster can have 0 to 12 agent pool profiles. Agent Pool Profiles are used for
| ---------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| availabilityProfile | no | Supported values are `VirtualMachineScaleSets` (default, except for Kubernetes clusters before version 1.10) and `AvailabilitySet`. |
| count | yes | Describes the node count |
| availabilityZones | no | To protect your cluster from datacenter-level failures, you can provide Availability Zones for each of your agentPool. Only applies to Kubernetes clusters version 1.12+. Supported values are arrays of strings, each representing a supported availability zone in a region for your subscription. e.g. `"availabilityZones": ["1","2"]` represents zone 1 and zone 2 can be used. To get supported zones for a region in your subscription, run `az vm list-skus --location centralus --query "[?name=='Standard_DS2_v2'].[locationInfo, restrictions"] -o table`. You should see values like `'zones': ['2', '3', '1']` appear in the first column. If `NotAvailableForSubscription` appears in the output, then you need to create an Azure support ticket to enable zones for that region. Note: For availability zones, only standard load balancer is supported. ([Availability zone example](../examples/e2e-tests/kubernetes/zones)). |
| availabilityZones | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring `"availabilityZones"` for the master profile and all of the agentPool profiles in the cluster definition. This feature only applies to Kubernetes clusters version 1.12+. Supported values are arrays of strings, each representing a supported availability zone in a region for your subscription. e.g. `"availabilityZones": ["1","2"]` represents zone 1 and zone 2 can be used. To get supported zones for a region in your subscription, run `az vm list-skus --location centralus --query "[?name=='Standard_DS2_v2'].[locationInfo, restrictions"] -o table`. You should see values like `'zones': ['2', '3', '1']` appear in the first column. If `NotAvailableForSubscription` appears in the output, then create an Azure support ticket to enable zones for that region. NOTE: To ensure high availability, each profile must define at least two nodes per zone. e.g. An agent pool profile with 2 zones: `"availabilityZones": ["1","2"]` must have at least 4 nodes total with `"count": 4`. When `"availabilityZones"` is configured, the `"loadBalancerSku"` will default to `Standard` as Standard LoadBalancer is required for availability zones. |
| singlePlacementGroup | no | Supported values are `true` (default) and `false`. Only applies to clusters with availabilityProfile `VirtualMachineScaleSets`. `true`: A VMSS with a single placement group and has a range of 0-100 VMs. `false`: A VMSS with multiple placement groups and has a range of 0-1,000 VMs. For more information, check out [virtual machine scale sets placement groups](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-placement-groups). |
| scaleSetPriority | no | Supported values are `Regular` (default) and `Low`. Only applies to clusters with availabilityProfile `VirtualMachineScaleSets`. Enables the usage of [Low-priority VMs on Scale Sets](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-use-low-priority). |
| scaleSetEvictionPolicy | no | Supported values are `Delete` (default) and `Deallocate`. Only applies to clusters with availabilityProfile of `VirtualMachineScaleSets` and scaleSetPriority of `Low`. |
Expand Down
10 changes: 8 additions & 2 deletions examples/e2e-tests/kubernetes/zones/definition.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,21 @@
"orchestratorRelease": "1.12"
},
"masterProfile": {
"count": 1,
"count": 5,
"dnsPrefix": "",
"vmSize": "Standard_DS2_v2"
"vmSize": "Standard_DS2_v2",
"availabilityProfile": "VirtualMachineScaleSets",
"availabilityZones": [
"1",
"2"
]
},
"agentPoolProfiles": [
{
"name": "agentpool",
"count": 4,
"vmSize": "Standard_DS2_v2",
"availabilityProfile": "VirtualMachineScaleSets",
"availabilityZones": [
"1",
"2"
Expand Down
18 changes: 16 additions & 2 deletions parts/k8s/kubernetesmasterresourcesvmss.t
View file
Original file line number Diff line number Diff line change
Expand Up @@ -186,22 +186,29 @@
},
{{end}}
{
"apiVersion": "[variables('apiVersionDefault')]",
"apiVersion": "2018-04-01",
"location": "[variables('location')]",
"name": "[variables('masterPublicIPAddressName')]",
"properties": {
"dnsSettings": {
"domainNameLabel": "[variables('masterFqdnPrefix')]"
},
{{ if .MasterProfile.HasAvailabilityZones}}
"publicIPAllocationMethod": "Static"
{{else}}
"publicIPAllocationMethod": "Dynamic"
{{end}}
},
"sku": {
"name": "[variables('loadBalancerSku')]"
},
"type": "Microsoft.Network/publicIPAddresses"
},
{
"type": "Microsoft.Network/loadBalancers",
"name": "[variables('masterLbName')]",
"location": "[variables('location')]",
"apiVersion": "[variables('apiVersionDefault')]",
"apiVersion": "2018-04-01",
"dependsOn": [
"[concat('Microsoft.Network/publicIPAddresses/', variables('masterPublicIPAddressName'))]"
],
Expand Down Expand Up @@ -269,6 +276,9 @@
}
}
]
},
"sku": {
"name": "[variables('loadBalancerSku')]"
}
},
{
Expand All @@ -290,6 +300,9 @@
"poolName": "master"
},
"location": "[variables('location')]",
{{ if .MasterProfile.HasAvailabilityZones}}
"zones": "[parameters('availabilityZones')]",
{{ end }}
"name": "[concat(variables('masterVMNamePrefix'), 'vmss')]",
{{if UseManagedIdentity}}
{{if UserAssignedIDEnabled}}
Expand All @@ -311,6 +324,7 @@
"name": "[parameters('masterVMSize')]"
},
"properties": {
"singlePlacementGroup": {{ .MasterProfile.SinglePlacementGroup}},
"overprovision": false,
"upgradePolicy": {
"mode": "Manual"
Expand Down
2 changes: 1 addition & 1 deletion parts/k8s/kubernetesmastervars.t
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
{{if .HasManagedDisks}}
"apiVersionStorageManagedDisks": "2016-04-30-preview",
{{end}}
{{if .HasVirtualMachineScaleSets}}
{{if .HasVMSSAgentPool}}
"apiVersionVirtualMachineScaleSets": "2017-12-01",
{{end}}
{{if not IsHostedMaster}}
Expand Down
2 changes: 0 additions & 2 deletions parts/k8s/kubernetesmastervarsvmss.t
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,7 @@
{{if .HasManagedDisks}}
"apiVersionStorageManagedDisks": "2016-04-30-preview",
{{end}}
{{if .HasVirtualMachineScaleSets}}
"apiVersionVirtualMachineScaleSets": "2017-12-01",
{{end}}
{{if not IsHostedMaster}}
{{if .MasterProfile.IsStorageAccount}}
"masterStorageAccountName": "[concat(variables('storageAccountBaseName'), 'mstr0')]",
Expand Down
8 changes: 8 additions & 0 deletions parts/masterparams.t
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,14 @@
"type": "string"
},
{{end}}
{{if .MasterProfile.HasAvailabilityZones}}
"availabilityZones": {
"metadata": {
"description": "Master availability zones"
},
"type": "array"
},
{{end}}
{{end}}
{{if IsHostedMaster}}
"masterSubnet": {
Expand Down
33 changes: 24 additions & 9 deletions pkg/acsengine/defaults.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -241,12 +241,19 @@ func setPropertiesDefaults(cs *api.ContainerService, isUpgrade, isScale bool) (b
if cs.Properties.MasterProfile != nil {
setMasterProfileDefaults(properties, isUpgrade)
}
// Set VMSS Defaults for Masters
if cs.Properties.MasterProfile != nil && cs.Properties.MasterProfile.IsVirtualMachineScaleSets() {
setVMSSDefaultsForMasters(properties)
}

setAgentProfileDefaults(properties, isUpgrade, isScale)

setStorageDefaults(properties)
setExtensionDefaults(properties)
setVMSSDefaults(properties)
// Set VMSS Defaults for Agents
if cs.Properties.HasVMSSAgentPool() {
setVMSSDefaultsForAgents(properties)
}

// Set hosted master profile defaults if this cluster configuration has a hosted control plane
if cs.Properties.HostedMasterProfile != nil {
Expand Down Expand Up @@ -426,11 +433,11 @@ func setOrchestratorDefaults(cs *api.ContainerService, isUpdate bool) {
a.OrchestratorProfile.KubernetesConfig.UseInstanceMetadata = helpers.PointerToBool(api.DefaultUseInstanceMetadata)
}

if a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "" {
if !a.HasAvailabilityZones() && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "" {
a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku = api.DefaultLoadBalancerSku
}

if common.IsKubernetesVersionGe(a.OrchestratorProfile.OrchestratorVersion, "1.11.0") && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "Standard" {
if common.IsKubernetesVersionGe(a.OrchestratorProfile.OrchestratorVersion, "1.11.0") && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "Standard" && a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB == nil {
a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB = helpers.PointerToBool(api.DefaultExcludeMasterFromStandardLB)
}

Expand Down Expand Up @@ -588,8 +595,19 @@ func setMasterProfileDefaults(a *api.Properties, isUpgrade bool) {
}
}

// setVMSSDefaults
func setVMSSDefaults(a *api.Properties) {
// setVMSSDefaultsForMasters
func setVMSSDefaultsForMasters(a *api.Properties) {
if a.MasterProfile.SinglePlacementGroup == nil {
a.MasterProfile.SinglePlacementGroup = helpers.PointerToBool(api.DefaultSinglePlacementGroup)
}
if a.MasterProfile.HasAvailabilityZones() && (a.OrchestratorProfile.KubernetesConfig != nil && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "") {
a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku = "Standard"
a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB = helpers.PointerToBool(api.DefaultExcludeMasterFromStandardLB)
}
}

// setVMSSDefaultsForAgents
func setVMSSDefaultsForAgents(a *api.Properties) {
for _, profile := range a.AgentPoolProfiles {
if profile.AvailabilityProfile == api.VirtualMachineScaleSets {
if profile.Count > 100 {
Expand All @@ -598,10 +616,7 @@ func setVMSSDefaults(a *api.Properties) {
if profile.SinglePlacementGroup == nil {
profile.SinglePlacementGroup = helpers.PointerToBool(api.DefaultSinglePlacementGroup)
}
if profile.SinglePlacementGroup == helpers.PointerToBool(false) {
profile.StorageProfile = api.ManagedDisks
}
if profile.HasAvailabilityZones() {
if profile.HasAvailabilityZones() && (a.OrchestratorProfile.KubernetesConfig != nil && a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku == "") {
a.OrchestratorProfile.KubernetesConfig.LoadBalancerSku = "Standard"
a.OrchestratorProfile.KubernetesConfig.ExcludeMasterFromStandardLB = helpers.PointerToBool(api.DefaultExcludeMasterFromStandardLB)
}
Expand Down
Loading

0 comments on commit 13c91da

Please sign in to comment.