[BUG]: CSI-provisioner container panic issue

[karthikk92]

Bug Description

Issue Description:
[Basic info]

• CSI Driver version: v1.5
• Kubernetes version (use kubectl version): v1.20.5
• OS (e.g. from /etc/os-release): CentOS 7.9
• Kernel (e.g. uname -a): 3.10.0-1160.el7.x86_64

E0127 03:00:00.954142 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

Logs

goroutine 181 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1775c20, 0x25e09d0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1775c20, 0x25e09d0)
/go/pkg/csiprow.XXXXfMOPAH/go-1.15/src/runtime/panic.go:969 +0x175
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/workspace/vendor/k8s.io/client-go/tools/cache/shared_informer.go:774 +0x194
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0009c3760)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0012b1f60, 0x1b66080, 0xc000a02840, 0x1741401, 0xc0006c69c0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009c3760, 0x3b9aca00, 0x0, 0x1, 0xc0006c69c0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a8880)
/workspace/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0007023e0, 0xc000cf26c0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1411d14]
goroutine 181 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x1775c20, 0x25e09d0)
/go/pkg/csiprow.XXXXfMOPAH/go-1.15/src/runtime/panic.go:969 +0x175
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/workspace/vendor/k8s.io/client-go/tools/cache/shared_informer.go:774 +0x194
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0009c3760)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0018f3f60, 0x1b66080, 0xc000a02840, 0x1741401, 0xc0006c69c0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009c3760, 0x3b9aca00, 0x0, 0x1, 0xc0006c69c0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0001a8880)
/workspace/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0007023e0, 0xc000cf26c0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

Troubleshooting Steps Performed:
We only have the log file, but don't find any dump file.

Current Impact:
Customer want to confirm if it is a software issue, and if the CSI driver version need to be upgraded.

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

Install csi Driver 1.5

Expected Behavior

driver pods should not throw "invalid memory address or nil pointer dereference"

CSM Driver(s)

CSI Driver for Unity XT

Installation Type

No response

Container Storage Modules Enabled

No response

Container Orchestrator

Kubernetes 1.20.5

Operating System

CentOS 7.9

[karthikk92]

Could you please share the driver logs for both node and controller with log level as "debug" in myvalues.yaml file.. Which helps further debugging

Like kubectl logs pod_name -c driver -c unity

[karthikk92]

Got customer feedback：

We will report the contents confirmed through this SR to customers and recommend driver upgrades.
You've worked hard during that time. Now you can archive the SR. After discussion with Fred, this issue is resolved. Hence closing this issue.