panic during startup

[gregwebs]

I saw this during startup. Note that it was able to restart successfully without seeing this error again. So I think it is an issue that occurs during cluster startup.

It seems like the line oldSet, err := tkmm.setLister.StatefulSets(ns).Get(controller.TiKVMemberName(tcName)) must have returned a nil pointer to oldSet?

kubectl logs -n test tidb-controller-manager-7b9bbb46d9-7pkrl
I1018 20:39:12.242431       1 version.go:37] Welcome to TiDB Operator.
I1018 20:39:12.242464       1 version.go:38] Git Commit Hash: e29b757b88848ef318a39ac34627e222b7f16b2e
I1018 20:39:12.242477       1 version.go:39] UTC Build Time:  2018-10-15 11:33:08
I1018 20:39:12.292318       1 leaderelection.go:175] attempting to acquire leader lease  test/tidb-controller-manager...
I1018 20:39:12.405856       1 leaderelection.go:184] successfully acquired lease test/tidb-controller-manager
I1018 20:39:12.405974       1 tidb_cluster_controller.go:200] Starting tidbcluster controller
E1018 20:39:14.591821       1 pd_member_manager.go:206] failed to sync TidbCluster: [test/db]'s status, error: Get http://db-pd.test:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1018 20:39:14.611189       1 pod_control.go:78] Pod: [test/db-pd-0] updated successfully, TidbCluster: [test/db]
I1018 20:39:14.611934       1 event.go:218] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"test", Name:"db", UID:"dd88a87b-d315-11e8-8232-42010a8a0035", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"17675", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update Pod db-pd-0 in TidbCluster db successful
I1018 20:39:14.612303       1 tidb_cluster_control.go:100] tidbcluster: [test/db]'s pd cluster is not running.
I1018 20:39:14.626988       1 tidbcluster_control.go:66] TidbCluster: [test/db] updated successfully
I1018 20:39:14.689320       1 event.go:218] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"test", Name:"db", UID:"dd88a87b-d315-11e8-8232-42010a8a0035", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"17675", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster db successful
E1018 20:39:16.691178       1 pd_member_manager.go:206] failed to sync TidbCluster: [test/db]'s status, error: Get http://db-pd.test:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1018 20:39:16.691879       1 tidb_cluster_control.go:100] tidbcluster: [test/db]'s pd cluster is not running.
E1018 20:39:37.108371       1 pd_member_manager.go:206] failed to sync TidbCluster: [test/db]'s status, error: Get http://db-pd.test:2379/pd/health: dial tcp 10.59.245.234:2379: connect: connection refused
I1018 20:39:37.109280       1 tidb_cluster_control.go:100] tidbcluster: [test/db]'s pd cluster is not running.
I1018 20:39:37.189549       1 tidbcluster_control.go:66] TidbCluster: [test/db] updated successfully
I1018 20:39:37.190368       1 event.go:218] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"test", Name:"db", UID:"dd88a87b-d315-11e8-8232-42010a8a0035", APIVersion:"pingcap.com", ResourceVersion:"17694", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster db successful
E1018 20:39:37.192546       1 pd_member_manager.go:206] failed to sync TidbCluster: [test/db]'s status, error: Get http://db-pd.test:2379/pd/health: dial tcp 10.59.245.234:2379: connect: connection refused
I1018 20:39:37.193548       1 tidb_cluster_control.go:100] tidbcluster: [test/db]'s pd cluster is not running.
I1018 20:39:42.412132       1 event.go:218] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"test", Name:"db", UID:"dd88a87b-d315-11e8-8232-42010a8a0035", APIVersion:"pingcap.com", ResourceVersion:"17750", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service db-tikv-peer in TidbCluster db successful
I1018 20:39:42.425299       1 tidb_cluster_control.go:112] tidbcluster: [test/db]'s tikv cluster is not running.
I1018 20:39:42.425793       1 event.go:218] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"test", Name:"db", UID:"dd88a87b-d315-11e8-8232-42010a8a0035", APIVersion:"pingcap.com", ResourceVersion:"17750", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet db-tikv in TidbCluster db successful
I1018 20:39:42.443350       1 tidbcluster_control.go:66] TidbCluster: [test/db] updated successfully
I1018 20:39:42.445731       1 event.go:218] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"test", Name:"db", UID:"dd88a87b-d315-11e8-8232-42010a8a0035", APIVersion:"pingcap.com", ResourceVersion:"17750", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster db successful
E1018 20:39:42.450730       1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/utils.go:68
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:446
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:173
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:107
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:105
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:77
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:266
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:262
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:229
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:216
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:208
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1309ff8]

goroutine 229 [running]:
github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x14fa1c0, 0x2227f70)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/pingcap/tidb-operator/pkg/manager/member.statefulSetIsUpgrading(...)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/utils.go:68
github.com/pingcap/tidb-operator/pkg/manager/member.(*tikvMemberManager).syncTidbClusterStatus(0xc4203c5760, 0xc420575c00, 0xc42022fb00, 0xc42022fb00, 0x0)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:446 +0xf48
github.com/pingcap/tidb-operator/pkg/manager/member.(*tikvMemberManager).syncStatefulSetForTidbCluster(0xc4203c5760, 0xc420575c00, 0x1707f39, 0x4)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:173 +0x2a6
github.com/pingcap/tidb-operator/pkg/manager/member.(*tikvMemberManager).Sync(0xc4203c5760, 0xc420575c00, 0x1, 0x0)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:107 +0x178
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*defaultTidbClusterControl).updateTidbCluster(0xc4200383f0, 0xc420575c00, 0x0, 0x0)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:105 +0xbb
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*defaultTidbClusterControl).UpdateTidbCluster(0xc4200383f0, 0xc420575c00, 0xc420575c00, 0xc420575c00)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:77 +0x79
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).syncTidbCluster(0xc420038460, 0xc420575c00, 0x0, 0xc4205b7880)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:266 +0x3e
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).sync(0xc420038460, 0xc420938840, 0x7, 0x0, 0x0)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:262 +0x1be
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).processNextWorkItem(0xc420038460, 0xc4204d4a00)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:229 +0xec
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).worker(0xc420038460)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:216 +0x2b
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).(github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.worker)-fm()
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:208 +0x2a
github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42056eee0)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc42056eee0, 0x3b9aca00, 0x0, 0x1, 0xc420731c20)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42056eee0, 0x3b9aca00, 0xc420731c20)
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).Run
	/home/jenkins/workspace/build_tidb_operator_master/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:208 +0x1af

[weekface]

This PR will fixes it: https://github.com/pingcap/tidb-operator/pull/120/files#diff-969d53a49525b7fa111454dd3e70b96fR64-R75

set.Status.ObservedGeneration is a pointer. May be nil.