[backend] Panic in API Server

[TobiasGoerke]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  Using Kubeflow v1.8.0 Release.
• KFP version:
  v2.0.3

Steps to reproduce

The pipelines API (ml-pipeline-Pod) periodically crashes.
While I can't give an example reproducing the error, here's the --previous log with the panic:

I1127 14:21:13.257864       7 resource_manager.go:1702] Authorizing request
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1d7362c]

goroutine 591944 [running]:
[github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).Error(..](http://github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).Error(..).)
        /go/src/[github.com/kubeflow/pipelines/backend/src/common/util/error.go:249](http://github.com/kubeflow/pipelines/backend/src/common/util/error.go:249)
[github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).IsAuthorized(0xc00035f900](http://github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).IsAuthorized(0xc00035f900), {0x28b0890, 0xc001e659b0}, 0xc0003c5d50)
        /go/src/[github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1730](http://github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1730) +0x9cc
[github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment(0xc00024cea0](http://github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment(0xc00024cea0), {0x28b0890, 0xc001e659b0}, {0xc00059ef60, 0x24}, 0xc0003c5d50)
        /go/src/[github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:348](http://github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:348) +0x1e8
[github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).getExperiment(0xc00024cea0](http://github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).getExperiment(0xc00024cea0), {0x28b0890, 0xc001e659b0}, {0xc00059ef60, 0x24})
        /go/src/[github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:155](http://github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:155) +0x71
[github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).GetExperiment(0xc00024cea0](http://github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).GetExperiment(0xc00024cea0)?, {0x28b0890?, 0xc001e659b0?}, 0xc001c1b2c0?)
        /go/src/[github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:188](http://github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:188) +0x8c
[github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._ExperimentService_GetExperiment_Handler.func1({0x28b0890](http://github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._ExperimentService_GetExperiment_Handler.func1(%7B0x28b0890), 0xc001e659b0}, {0x21c8060?, 0xc001c1b2c0})
        /go/src/[github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/experiment.pb.go:1086](http://github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/experiment.pb.go:1086) +0x78
main.apiServerInterceptor({0x28b0890, 0xc001e659b0}, {0x21c8060, 0xc001c1b2c0}, 0xc0005ea2e0, 0xc001b38fc0)
        /go/src/[github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30](http://github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30) +0xd8
[github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._ExperimentService_GetExperiment_Handler({0x2360700](http://github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._ExperimentService_GetExperiment_Handler(%7B0x2360700)?, 0xc00024cea0}, {0x28b0890, 0xc001e659b0}, 0xc00177b7a0, 0x25498a0)
        /go/src/[github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/experiment.pb.go:1088](http://github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/experiment.pb.go:1088) +0x138
[google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ba000](http://google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ba000), {0x28b8ae0, 0xc000683d40}, 0xc000956ea0, 0xc0005486c0, 0x3a03d38, 0x0)
        /go/pkg/mod/[google.golang.org/[email protected]/server.go:1282](http://google.golang.org/[email protected]/server.go:1282) +0xcef
[google.golang.org/grpc.(*Server).handleStream(0xc0001ba000](http://google.golang.org/grpc.(*Server).handleStream(0xc0001ba000), {0x28b8ae0, 0xc000683d40}, 0xc000956ea0, 0x0)
        /go/pkg/mod/[google.golang.org/[email protected]/server.go:1616](http://google.golang.org/[email protected]/server.go:1616) +0xa36
[google.golang.org/grpc.(*Server).serveStreams.func1.2()](http://google.golang.org/grpc.(*Server).serveStreams.func1.2())
        /go/pkg/mod/[google.golang.org/[email protected]/server.go:921](http://google.golang.org/[email protected]/server.go:921) +0x98
created by [google.golang.org/grpc.(*Server).serveStreams.func1](http://google.golang.org/grpc.(*Server).serveStreams.func1)
        /go/pkg/mod/[google.golang.org/[email protected]/server.go:919](http://google.golang.org/[email protected]/server.go:919) +0x28a

---

Impacted by this bug? Give it a 👍.

[TobiasGoerke]

Looks like it paniced while trying to log an error, as it can't access the internalError.Error()