nil pointer dereference in syncIngress()

[jessesuen]

time="2020-06-15T11:00:51Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 206 [running]:
runtime/debug.Stack(0xc000764888, 0x16de020, 0x282f0c0)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1.1.1(0xc000474070, 0xc000764b08)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:147 +0x57
panic(0x16de020, 0x282f0c0)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/argoproj/argo-rollouts/ingress.(*Controller).syncIngress(0xc00048e400, 0xc000826750, 0x2a, 0x18fb227, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/ingress/ingress.go:134 +0xfd
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1.1(0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:151 +0xa0
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1(0x1bb4a20, 0xc000435da0, 0x18f95f3, 0x7, 0xc000764e50, 0xc000084f00, 0x16660e0, 0xc0003db0f0, 0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:155 +0x30a
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem(0x1bb4a20, 0xc000435da0, 0x18f95f3, 0x7, 0xc000764e50, 0xc000084f00, 0x1)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:166 +0x9a
github.com/argoproj/argo-rollouts/utils/controller.RunWorker(0x1bb4a20, 0xc000435da0, 0x18f95f3, 0x7, 0xc000764e50, 0xc000084f00)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:102 +0x5d
github.com/argoproj/argo-rollouts/ingress.(*Controller).Run.func1()
    /go/src/github.com/argoproj/argo-rollouts/ingress/ingress.go:104 +0x78
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000c1a1e0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000c1a1e0, 0x3b9aca00, 0x0, 0x1, 0xc00028a540)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc000c1a1e0, 0x3b9aca00, 0xc00028a540)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by github.com/argoproj/argo-rollouts/ingress.(*Controller).Run
    /go/src/github.com/argoproj/argo-rollouts/ingress/ingress.go:103 +0xac
" ingress=rollouts-demo-rollouts-demo-canary namespace=default

[smitthakkar96]

@jessesuen how to reproduce this problem and why is this happening? I got similar issue today

time="2020-07-21T07:10:33Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 242 [running]:
runtime/debug.Stack(0xc00082c7a0, 0x16de020, 0x282f0a0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1.1.1(0xc0025f6930, 0xc00082db08)
	/go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:147 +0x57
panic(0x16de020, 0x282f0a0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/argoproj/argo-rollouts/rollout.(*RolloutPodRestarter).Reconcile(0xc00021bbf8, 0x1bbc960, 0xc003a8a240, 0x4, 0x4)
	/go/src/github.com/argoproj/argo-rollouts/rollout/restart.go:60 +0x287
github.com/argoproj/argo-rollouts/rollout.(*RolloutController).syncReplicasOnly(0xc00021bb00, 0xc0025a7800, 0xc0025f96c0, 0x4, 0x4, 0x1, 0x0, 0x0)
	/go/src/github.com/argoproj/argo-rollouts/rollout/sync.go:256 +0x2c9
github.com/argoproj/argo-rollouts/rollout.(*RolloutController).syncHandler(0xc00021bb00, 0xc0015a7e60, 0x16, 0x0, 0x0)
	/go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:338 +0xa25
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1.1(0x0, 0x0)
	/go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:151 +0xa0
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1(0x1bb4960, 0xc0003b7fa0, 0x18f97d6, 0x7, 0xc00082de50, 0xc0005f05c0, 0x16660e0, 0xc00396fd50, 0x0, 0x0)
	/go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:155 +0x30a
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem(0x1bb4960, 0xc0003b7fa0, 0x18f97d6, 0x7, 0xc00082de50, 0xc0005f05c0, 0x1)
	/go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:166 +0x9a
github.com/argoproj/argo-rollouts/utils/controller.RunWorker(0x1bb4960, 0xc0003b7fa0, 0x18f97d6, 0x7, 0xc00082de50, 0xc0005f05c0)
	/go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:102 +0x5d
github.com/argoproj/argo-rollouts/rollout.(*RolloutController).Run.func1()
	/go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:235 +0x81
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0010529e0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0010529e0, 0x3b9aca00, 0x0, 0x1, 0xc000214540)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0010529e0, 0x3b9aca00, 0xc000214540)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by github.com/argoproj/argo-rollouts/rollout.(*RolloutController).Run
	/go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:234 +0xc9
" namespace=beta-bid rollout=authenticator
E0721 07:10:33.870028       1 controller.go:169] Recovered from Panic

[smitthakkar96]

This issue is causing rollouts to be stuck in progressing state due to which scaling doesn't happen. It was reason a recent incident in our company. If such a critical operation can failure due to rollouts acting weird I guess we should use deployment controller and have rollouts manage it like flagger, atleast it will not fuck up the autoscaling. Guys need help on this urgently

[dthomson25]

Closed with #603