Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to version 2.2.0 or higher causes "Address family not supported by protocol" #2970

Closed
jppitout opened this issue Aug 25, 2022 · 8 comments

Comments

@jppitout
Copy link

jppitout commented Aug 25, 2022

Describe the bug
On Kubernetes clusters with IPv6 disabled an upgrade to version 2.2.0 and higher causes:
[::]:80 failed (97: Address family not supported by protocol

To Reproduce
Steps to reproduce the behavior:

  1. Disable IPv6 on Kubernetes cluster
  2. Upgrade kubernetes-ingress to v2.2.0
  3. Pods CrashLoopBackOff
  4. Check pod logs:
    2022/08/25 12:58:31 [emerg] 12#12: socket() [::]:80 failed (97: Address family not supported by protocol)

Expected behavior
A way to cater for Kubernetes clusters without IPv6 either in the configmap or server-snippets.
We cannot find a way to disable NGINX's IPv6 implementation currently.

Environment

  • Ingress Controller: v2.3.0
  • Kubernetes: v1.23.7
  • Kubernetes platform: TKGi
  • Using NGINX (no PLUS)
@github-actions
Copy link

Hi @jppitout thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this 🙂

Cheers!

@brianehlert
Copy link
Collaborator

Beginning with the 2.2 release a default IPv6 listener is created for nginx. This was to align with K8s making IPv6 a first class citizen and giving controls to administrators to define both IPv4 or IPv6 address pools.

Since the determination of whether or not a pod receives an IPv6 address was fully exposed through the K8s cluster configuration, and a pod would not receive an IPv6 address unless an IPv6 address pool was defined, and that that you have to go back quite a few years into unsupported distros to not have an IPv6 stack present - we considered it safe to default to defining an IPv6 listener.

If I am understanding your scenario, you are not blocking IPv6 address assignment through the normal networking configuration files, but rather through some method of blocking the IPv6 stack from loading at all at the kernel level of the K8s node.
And since pods share the node of the kernel of the system they run on, this has the downstream impact of the pods being blocked from loading their IPv6 stack.
And thus, when nginx attempts to start, the IPv6 stack is not present in the pod and thus the binding for the listener cannot be made.

@jppitout, Can you help me understand two things?

  1. How have you disabled IPv6 on your Nodes (in the image you use for your nodes)?
  2. Why have you disabled it the way that you did?

That said, and those questions asked, we will be introducing a way to give you more control over the listeners in an upcoming release. And thus fully disable the IPv6 listener if you need to.

@jppitout
Copy link
Author

jppitout commented Aug 25, 2022

@brianehlert thank you for your response.

Some prerequisite knowledge:
VMware TKGi's implementation of Kubernetes is deployed using BOSH. The base OS of the Kubernetes nodes are known as
BOSH stemcells and are basically prepackaged OS images.

  1. IPV6 support on BOSH stemcells are baked into the kernel but disabled via GRUB bootloader option, module blacklist, and sysctl disabling.
  2. This is specific to VMware TKGi. It has been disabled upstream and we cannot easily enable it nor do we have reason to at this time.

For now we'll stick to version NGINX 2.1.2, thanks for clarifying!

@brianehlert
Copy link
Collaborator

@jppitout Thank you very much for that most excellent explanation. I didn't realize that.
We have Tanzu customers that have not reported this. So I obviously need to learn more about the options and limitations there.

@jppitout
Copy link
Author

@brianehlert No problem, I suspect that TKG which uses Cluster API will not have this problem only TKGi which uses BOSH.

@jppitout
Copy link
Author

jppitout commented Oct 10, 2022

Hi @brianehlert

Any feed back on this? We'd like to upgrade to 2.4.0 for other bug fixes but cannot due to this bug.

UPDATE
Apologies I see the was an argument added to 2.4.0 #3040

@brianehlert
Copy link
Collaborator

New command line option added in 2.4

@jppitout
Copy link
Author

jppitout commented Oct 11, 2022

Hi @brianehlert,

With version 2.4.0 we're still seeing below in the logs even with -disable-ipv6=true argument set and we're getting 404s to the upstreams:
2022/10/11 07:04:53 [emerg] 12#12: socket() [::]:80 failed (97: Address family not supported by protocol)

On the pods themselves we're still seeing ipv6 listeners configured:

nginx@nginx-ingress-57c678c956-9bd2k:/etc/nginx$ grep -ir 'listen \[::\]:80;' *
conf.d/default-dex-k8s-authenticator.conf:      listen [::]:80;
conf.d/default-dex.conf:        listen [::]:80;
conf.d/vault-vault.conf:        listen [::]:80;
...

Any ideas what could be causing this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants