Wrong number of open file calculated

[dawez]

The current 0.11 version changed the way the calculation of maximum open files is done. #2050

Now the value is coming from the kernel parameter fs.file-max and this is probably not what we want. The value is shared among all containers if you run many multiple containers only a small fraction of the "open files" might be available to a single container. It make no sense to calculate the open files limit per nginx worker based on this parameter as if nginx would be only one running on the node.

Furthermore each container can have a specific limit that is capped by the kernel limit. If you set a limit for the "open files" in the Docker daemon that is way lower than the kernel parameter, the nginx ingress tries to setup a value [worker_rlimit_nofile] outside the pod's limits and causes an error in the log:

2018/02/28 08:37:57 [alert] 2415#2415: setrlimit(RLIMIT_NOFILE, 201374) failed (1: Operation not permitted)

When running ingress in non priviledged mode it is also not possible to change the kernel value anyway when calling 'sysctl -w fs.file-max=10000 in the initContainer.

What would make sense is to use instead the value coming from the userlimit as this is the real limit for the process open files.

[maxlaverse]

Actually, I think it was done right since #177 but #2050 broke the "open files" limit determination.

There was a discussion about it in the issue #2048 that led to PR #2050. The example given in that issue is a very specific use-cases. The calculation only makes sense if the Ingress is alone on the Kubernetes node and if no lower limit is set at a different level

Where the limit for the number of open files should be set is a tricky discussion as seen in kubernetes/kubernetes#3595.

But having the Nginx Ingress relying on the global kernel parameter fs.file-max to calculate how many open files there can be per Nginx worker sounds wrong. fs.file-max is "just" the limit on the kernel level and can be far away from how many files a process is allowed to open.

[bismarck]

I just hit this issue. Any plans to revert #2050? Maybe add a configuration flag to opt into using fs.file-max instead of making it the default?

[aledbf]

@gyliu513 please check this

[shenshouer]

No effect on sysctl -w fs.file-max=2097152 in initContainers, cannot change the value of worker_rlimit_nofile on 0.21.0 and 0.20.0

      initContainers:
      - command:
        - /bin/sh
        - -c
        - |
            sysctl -w net.netfilter.nf_conntrack_max=6553600
            sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=120
            sysctl -w net.ipv4.tcp_max_syn_backlog=2621440
            sysctl -w net.ipv4.tcp_max_tw_buckets=8000
            sysctl -w net.core.netdev_max_backlog=2621440
            sysctl -w net.core.rmem_default=33554432
            sysctl -w net.core.wmem_default=33554432
            sysctl -w net.core.rmem_max=33554432
            sysctl -w net.core.wmem_max=33554432
            sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432"
            sysctl -w net.ipv4.tcp_wmem="4096 87380 33554432"
            sysctl -w net.ipv4.tcp_mem="94500000 915000000 927000000"
            sysctl -w net.ipv4.tcp_timestamps=0
            sysctl -w net.ipv4.tcp_synack_retries=1
            sysctl -w net.ipv4.tcp_syn_retries=1
            sysctl -w net.ipv4.tcp_tw_reuse=1
            sysctl -w net.ipv4.tcp_tw_recycle=1
            sysctl -w net.ipv4.ip_forward=1
            sysctl -w net.ipv4.tcp_fin_timeout=15
            sysctl -w net.ipv4.tcp_keepalive_time=60
            sysctl -w net.bridge.bridge-nf-call-iptables=1
            sysctl -w net.bridge.bridge-nf-call-ip6tables=1
            sysctl -w net.ipv4.conf.all.forwarding=1
            sysctl -w net.ipv4.ip_forward=1
            sysctl -w net.core.somaxconn=65535
            sysctl -w net.ipv4.ip_local_port_range="1024 65535"
            sysctl -w fs.file-max=2097152

            sysctl -w fs.inotify.max_user_instances=16384
            sysctl -w fs.inotify.max_user_watches=524288
            sysctl -w fs.inotify.max_queued_events=16384
            sysctl -w fs.suid_dumpable=2
            ulimit -c unlimited
            ulimit -n 1024000
            chmod -R 777 -R /var/log/nginx
        securityContext:
          privileged: true

and logs as:

# docker logs  0ce5e64d9c21
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.21.0
  Build:      git-b65b85cd9
  Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------

nginx version: nginx/1.15.6
built by gcc 8.2.0 (Debian 8.2.0-9)
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
...
I1218 20:08:40.207903       8 status.go:148] new leader elected: nginx-ingress-controller-75d8d
I1218 20:08:40.208786       8 nginx.go:510] Adjusting ServerNameHashBucketSize variable to 64
I1218 20:08:40.208809       8 nginx.go:522] Number of worker processes: 48
I1218 20:08:40.208821       8 util.go:76] rlimit.max=65536
I1218 20:08:40.208833       8 nginx.go:527] Maximum number of open file descriptors: 341
...

when I exec to pods:

# kubectl -n ingress-nginx exec -it nginx-ingress-controller-8zld2 bash
Error from server (NotFound): pods "nginx-ingress-controller-8zld2" not found
# kubectl -n ingress-nginx exec -it nginx-ingress-controller-7tj8d bash
www-data@nginx-ingress-controller-7tj8d:/etc/nginx$ sysctl -a |grep file-max
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
fs.file-max = 2097152
...

[k8s-ci-robot]

@shenshouer: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.