Skip to content
This repository has been archived by the owner on Oct 24, 2023. It is now read-only.

feat: enable multiple frontend IPs in Standard LB #3085

Merged
merged 6 commits into from
Apr 20, 2020

Conversation

jackfrancis
Copy link
Member

@jackfrancis jackfrancis commented Apr 16, 2020

Reason for Change:

This PR enables a configurable number of frontend IP addresses to use for Standard LoadBalancer outbound rules, in order to increase available port allocation for large clusters.

The new interface to use this configuration is a property in the kubernetesConfig object. E.g.:

...
"kubernetesConfig": {
  ...
  "loadBalancerOutboundIPs": 3,
  ...
}
...

// TODO

  • documentation

Outbound SNAT limitations are described here:

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections#multifesnat

Issue Fixed:

Fixes #2377

Requirements:

Notes:

@codecov
Copy link

codecov bot commented Apr 17, 2020

Codecov Report

Merging #3085 into master will increase coverage by 0.22%.
The diff coverage is 92.72%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3085      +/-   ##
==========================================
+ Coverage   70.64%   70.87%   +0.22%     
==========================================
  Files         147      147              
  Lines       25430    25472      +42     
==========================================
+ Hits        17965    18053      +88     
+ Misses       6342     6290      -52     
- Partials     1123     1129       +6     
Impacted Files Coverage Δ
pkg/api/common/const.go 0.00% <ø> (ø)
pkg/api/types.go 93.98% <ø> (ø)
pkg/api/vlabs/types.go 73.30% <ø> (ø)
pkg/engine/armvariables.go 86.08% <65.21%> (-1.10%) ⬇️
pkg/api/converterfromapi.go 93.88% <100.00%> (+0.01%) ⬆️
pkg/api/convertertoapi.go 93.68% <100.00%> (+0.01%) ⬆️
pkg/api/vlabs/validate.go 79.31% <100.00%> (+0.11%) ⬆️
pkg/engine/armresources.go 79.83% <100.00%> (+1.65%) ⬆️
pkg/engine/ipaddresses.go 100.00% <100.00%> (ø)
pkg/engine/loadbalancers.go 100.00% <100.00%> (ø)
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3cb0105...f81bcf7. Read the comment docs.

@jackfrancis jackfrancis changed the title [WIP] feat: enable multiple frontend IPs in Standard LB feat: enable multiple frontend IPs in Standard LB Apr 17, 2020
@@ -41,6 +41,8 @@ const (
DefaultInternalLbStaticIPOffset = 10
// DefaultEnableCSIProxyWindows determines if CSI proxy should be enabled by default for Windows nodes
DefaultEnableCSIProxyWindows = false
// MaxLoadBalancerOutboundIPs is the maximum number of outbound IPs in a Standard LoadBalancer frontend configuration
MaxLoadBalancerOutboundIPs = 16
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmeytin are you aware of a sensible limit we should put here?

@jackfrancis jackfrancis force-pushed the lb-multiple-frontend-ips branch from 324d937 to 1f333a3 Compare April 17, 2020 22:58
@jackfrancis jackfrancis force-pushed the lb-multiple-frontend-ips branch from 1f333a3 to 594fa52 Compare April 17, 2020 23:07
@@ -65,7 +65,8 @@ $ aks-engine get-versions
| gcLowThreshold | no | Sets the --image-gc-low-threshold value on the kublet configuration. Default is 80. [See kubelet Garbage Collection](https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/) |
| kubeletConfig | no | Configure various runtime configuration for kubelet. See `kubeletConfig` [below](#feat-kubelet-config) |
| kubernetesImageBase | no | Specifies the default image base URL (everything preceding the actual image filename) to be used for all kubernetes-related containers such as hyperkube, cloud-controller-manager, pause, addon-manager, heapster, exechealthz etc. e.g., `k8s.gcr.io/` |
| loadBalancerSku | no | Sku of Load Balancer and Public IP. Candidate values are: `basic` and `standard`. If not set, it will be default to basic. Requires Kubernetes 1.11 or newer. NOTE: VMs behind standard SKU load balancer will not be able to access the internet without an outbound rule configured with at least one frontend IP. We have created a loadbalancer with an outbound rule and with agent nodes added to the backend pool, as described in the [Outbound NAT for internal Standard Load Balancer scenarios doc](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-rules-overview#outbound-nat-for-internal-standard-load-balancer-scenarios) |
| loadBalancerSku | no | Sku of Load Balancer and Public IP. Candidate values are: `basic` and `standard`. If not set, it will be default to "standard". NOTE: Because VMs behind standard SKU load balancer will not be able to access the internet without an outbound rule configured with at least one frontend IP, AKS Engine creates a Load Balancer with an outbound rule and with agent nodes added to the backend pool during cluster creation, as described in the [Outbound NAT for internal Standard Load Balancer scenarios doc](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-rules-overview#outbound-nat-for-internal-standard-load-balancer-scenarios) |
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This documentation was stale, updated here

Copy link
Member

@mboersma mboersma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@jackfrancis jackfrancis merged commit 499b54b into Azure:master Apr 20, 2020
@jackfrancis jackfrancis deleted the lb-multiple-frontend-ips branch April 20, 2020 18:44
alexeldeib pushed a commit to alexeldeib/aks-engine that referenced this pull request Apr 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Be able to configure both the number of IPs or the allocatedOutboundPorts
2 participants