Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for getting SNAT Ports from Load Balancer/AKS (data "azurerm_lb_outbound_rule") #19301

Closed
1 task done
ElvenSpellmaker opened this issue Nov 15, 2022 · 6 comments · Fixed by #19345
Closed
1 task done

Comments

@ElvenSpellmaker
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Due to the low SNAT port defaults of 1024 (even when you have a few nodes) when using the default Load Balancer configuration in Kubernetes and ran into SNAT port exhaustion.
(Moving to NAT Gateway isn't an option at the present time but still.)

We've since done this in our K8s to up the amount of allocated ports, allowing us to have 10 nodes:

...
    load_balancer_profile {
      outbound_ports_allocated = "5816"
      idle_timeout_in_minutes  = "4"
    }
...

However there's no way for us to retrieve the number, which we'd love so we may alert on X% of SNAT ports used for example, as there's no data source for Outbound Rules.

It'd be nice to expose this from the outputs of azurerm_kubernetes_cluster as well seeing as we set this from this resource (but no way of retrieving it), but at least some way of getting them (i.e. data "azurerm_lb_outbound_rule) would be great.

image

New or Affected Resource(s)/Data Source(s)

azurerm_lb_outbound_rule (new data), azurerm_kubernetes_cluster (affected data)

Potential Terraform Configuration

No response

References

https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections

@aristosvo
Copy link
Collaborator

aristosvo commented Nov 15, 2022

Hi @ElvenSpellmaker! Thanks for raising this.

Is there at this moment a way to get the number you are looking for? I mean, in your example you are setting a value, is this the same value expected as an output, like:

NODE_RG=$(az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv)
az network lb outbound-rule list --resource-group $NODE_RG --lb-name kubernetes -o table

An example how to solve this via az or Azure Portal might help get your issue resolved sooner. If this is not possible at the moment, I would advise to raise this at Microsoft side first.

@ElvenSpellmaker
Copy link
Contributor Author

Hi @ElvenSpellmaker! Thanks for raising this.

Is there at this moment a way to get the number you are looking for? Via az or the portal maybe? An example might help get your issue resolved sooner.

If this is not the case, I would advise to raise this at Microsoft side first.

Hey, you can definitely see the number in the portal (from my screenshot above)
I'll check az when I can.

https://user-images.githubusercontent.com/2286713/201972120-5758db8f-7753-4f0a-a84c-6b83b0023cc9.png

@aristosvo
Copy link
Collaborator

So to understand what you're looking for, correct me if I'm wrong:

  • You set a value for the AKS resource
  • You want to get that same value as an output from a data source

Can you explain what the difference is between the two or why this helps?

@ElvenSpellmaker
Copy link
Contributor Author

ElvenSpellmaker commented Nov 15, 2022

The use-case is so we can set an alert up on SNAT ports used as a % of the total allocated without having to hardcode it.

It's the same value, just it can't currently be retrieved in TF, but can be seen in the portal and set by both the AKS resource and by standalone outbound_rule resources.

See:

allocated_outbound_ports - (Optional) The number of outbound ports to be used for NAT. Defaults to 1024.

@ElvenSpellmaker
Copy link
Contributor Author

Also for completeness to your original question, yes this can be obtained by az:

$ az network lb outbound-rule list -g MC-foo-bar-baz --lb-name kubernetes --subscription <sub>
[
  {
    "allocatedOutboundPorts": 5816,
...

katbyte pushed a commit that referenced this issue Dec 15, 2022
favoretti pushed a commit to favoretti/terraform-provider-azurerm that referenced this issue Jan 12, 2023
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants