Panic: invalid memory address or nil pointer dereference in getBackendServers

[andscho-msft]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):
Bug report

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):
nil pointer, dereference

---

Is this a BUG REPORT or FEATURE REQUEST?: Bug

NGINX Ingress controller version: 0.16.2

Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:38:10Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Azure
• OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus)
• Kernel (e.g. uname -a): Linux k8s-master 4.13.0-1018-azure Refactor default backend handling and add better events #21-Ubuntu SMP Thu May 17 13:58:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Azure Container Service

What happened:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.16.2
  Build:      git-26eacf4
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

nginx version: nginx/1.13.12
W0702 19:08:59.614870       7 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0702 19:08:59.615046       7 main.go:183] Creating API client for https://10.0.0.1:443
I0702 19:08:59.640343       7 main.go:227] Running in Kubernetes cluster version v1.8 (v1.8.2) - git (clean) commit bdaeafa71f6c7c04636251031f93464384d54963 - platform linux/amd64
I0702 19:08:59.650198       7 main.go:100] Validated shared/default-http-backend as the default backend.
I0702 19:09:00.098130       7 nginx.go:250] Starting NGINX Ingress controller
I0702 19:09:00.109057       7 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"shared", Name:"nginx-ingress-controller-conf", UID:"70874c8d-69be-11e8-abe0-000d3a0682dd", APIVersion:"v1", ResourceVersion:"16005580", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap shared/nginx-ingress-controller-conf
I0702 19:09:00.125498       7 backend_ssl.go:69] adding secret shared/nginx-controller-default-ssl to the local store

.... some lines removed here....

E0702 19:09:01.308589       7 controller.go:369] Unexpected error reading annotations for Ingress "qstest-ingress" from local store: ingress annotations <redacted>/qstest-ingress was not found
E0702 19:09:01.310541       7 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/asm_amd64.s:573
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/panic.go:502
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/panic.go:63
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/signal_unix.go:388
/home/travis/gopath/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:401
/home/travis/gopath/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:128
/home/travis/gopath/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:120
/home/travis/gopath/src/k8s.io/ingress-nginx/internal/task/queue.go:129
/home/travis/gopath/src/k8s.io/ingress-nginx/internal/task/queue.go:61
/home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/home/travis/gopath/src/k8s.io/ingress-nginx/internal/task/queue.go:61
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1172f8a]

goroutine 174 [running]:
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x132a420, 0x1edd460)
        /home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/panic.go:502 +0x229
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).getBackendServers(0xc420021740, 0xc421636600, 0x37, 0x40, 0x80, 0x800, 0x20, 0x14fcbd7, 0x13, 0x14ec927)
        /home/travis/gopath/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:401 +0x1dba
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress(0xc420021740, 0x13c1900, 0xc4216de060, 0xc4124390ab, 0x660139de)
        /home/travis/gopath/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:128 +0x179
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).(k8s.io/ingress-nginx/internal/ingress/controller.syncIngress)-fm(0x13c1900, 0xc4216de060, 0xa, 0xc420b94d58)
        /home/travis/gopath/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:120 +0x3e
k8s.io/ingress-nginx/internal/task.(*Queue).worker(0xc4200af650)
        /home/travis/gopath/src/k8s.io/ingress-nginx/internal/task/queue.go:129 +0x35a
k8s.io/ingress-nginx/internal/task.(*Queue).(k8s.io/ingress-nginx/internal/task.worker)-fm()
        /home/travis/gopath/src/k8s.io/ingress-nginx/internal/task/queue.go:61 +0x2a
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420c0a7a8)
        /home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420fe3fa8, 0x3b9aca00, 0x0, 0x7f1ce15ce401, 0xc4200a0360)
        /home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420c0a7a8, 0x3b9aca00, 0xc4200a0360)
        /home/travis/gopath/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
k8s.io/ingress-nginx/internal/task.(*Queue).Run(0xc4200af650, 0x3b9aca00, 0xc4200a0360)
        /home/travis/gopath/src/k8s.io/ingress-nginx/internal/task/queue.go:61 +0x55
created by k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).Start
        /home/travis/gopath/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:274 +0x1d0

What you expected to happen:
Should generate nginx.conf and start up

How to reproduce it (as minimally and precisely as possible):
Not sure, same configuration works on a mirror-image cluster

Anything else we need to know:
There are some previous issues about the secret not containing data for some secrets. This is not the case here, our secrets have tls key, ca, and crt, all with data.

[aledbf]

@andscho-msft can you share the ingress rule you are using?

[andscho-msft]

We did have an ingress with an empty rules section, thanks.