support multiple replicas for linstor-controller

Running multiple replicas requires special support from the linstor controller. The controller container will start a leader election process when it detects the presence of the K8S_AWAIT_ELECTION_* variables. The election process determines which pod is allowed to start the linstor-controller process. Only this pod will be added as endpoint for the controller service. Should the leader crash or the node its running on goes offline, a new leader will be elected and allowed to start the controller process. Note: in case the full node goes offline, the old pod will still be marked as ready. By using ClusterIP: "" on our service, we ensure we create an actual proxy (which automatically chooses the responding pod) instead of each client having to try multiple DNS responses.
piraeusdatastore · Sep 30, 2020 · 53688fe · 53688fe
1 parent cc3af23
commit 53688fe
Show file tree

Hide file tree

Showing 17 changed files with 251 additions and 77 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,8 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Breaking
+
+* The LINSTOR controller image given in `operator.controller.controllerImage` has to have
+  its entrypoint set to [`k8s-await-election v0.2.0`](https://github.com/LINBIT/k8s-await-election/)
+  or newer. Learn more in the [upgrade guide](./UPGRADE.md#upgrade-from-v10-to-head).
+
 ### Added
 
+* LINSTOR controller can be started with multiple replicas. See [`operator.controller.replicas`](./doc/helm-values.adoc#operatorcontrollerreplicas).
+  NOTE: This requires support from the container. You need `piraeus-server:v1.8.0` or newer.
 * The `pv-hostpath` helper chart automatically sets up permissions for non-root etcd containers.
 * Disable securityContext enforcement by setting `global.setSecurityContext=false`.
 * Add cluster roles to work with OpenShift's SCC system.
@@ -20,6 +28,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+* Default values:
+  * `operator.controller.controllerImage`: `quay.io/piraeusdatastore/piraeus-server:v1.9.0`
+  * `operator.satelliteSet.satelliteImage`: `quay.io/piraeusdatastore/piraeus-server:v1.9.0`
+  * `operator.satelliteSet.kernelModuleInjectionImage`: `quay.io/piraeusdatastore/drbd9-bionic:v9.0.25`
 * linstor-controller no longer starts in a privileged container.
 
 ### Removed

diff --git a/README.md b/README.md
@@ -115,6 +115,24 @@ this [example chart configuration.](./examples/resource-requirements.yaml)
 
 [resource requests and limits]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
 
+### Running multiple replicas
+
+Running multiple replicas of pods is recommended for high availability and fast error recovery.
+The following components can be started with multiple replicas:
+
+* Operator: Set [`operator.replicas`] to the desired number of operator pods.
+* CSI: Set [`csi.controllerReplicas`] to the desired number of CSI Controller pods.
+* Linstor Controller: Set [`operator.controller.replicas`] to the desired number of LINSTOR controller pods.
+* CSI Snapshotter: Set [`csi-snapshotter.replicas`] to the desired number of CSI Snapshot Controller pods.
+* Etcd: Set [`etcd.replicas`] to the desired number of Etcd pods.
+* Stork: Set [`stork.replicas`] to the desired number of both Stork plugin and Stork scheduler pods.
+
+[`operator.replicas`]: ./doc/helm-values.adoc#operatorreplicas
+[`csi.controllerReplicas`]: ./doc/helm-values.adoc#csicontrollerreplicas
+[`operator.controller.replicas`]: ./doc/helm-values.adoc#operatorcontrollerreplicas
+[`csi-snapshotter.replicas`]: ./doc/helm-values.adoc#csi-snapshotterreplicas
+[`etcd.replicas`]: ./doc/helm-values.adoc#etcdreplicas
+[`stork.replicas`]: ./doc/helm-values.adoc#storkreplicas
 
 ### Terminating Helm deployment
 

diff --git a/UPGRADE.md b/UPGRADE.md
@@ -1,4 +1,13 @@
-The following document describes how to convert
+# Upgrade from v1.0 to HEAD
+
+* The LINSTOR controller image given in `operator.controller.controllerImage` has to have
+  its entrypoint set to [`k8s-await-election v0.2.0`](https://github.com/LINBIT/k8s-await-election/)
+  or newer. All images starting with `piraeus-server:v1.8.0` meet this requirement.
+
+  Older images will not work, as the `Service` will not automatically pick up on the active pod.
+
+  To upgrade, first update the deployed LINSTOR image to a compatible version, then upgrade the
+  operator.
 
 # Upgrade to v1.0
 

diff --git a/charts/piraeus/crds/piraeus.linbit.com_linstorcontrollers_crd.yaml b/charts/piraeus/crds/piraeus.linbit.com_linstorcontrollers_crd.yaml
@@ -657,6 +657,11 @@ spec:
               description: priorityClassName is the name of the PriorityClass for
                 the controller pods
               type: string
+            replicas:
+              description: Number of replicas in the controller deployment
+              format: int32
+              nullable: true
+              type: integer
             resources:
               description: Resource requirements for the LINSTOR controller pod
               nullable: true
@@ -676,6 +681,10 @@ spec:
                     value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
                   type: object
               type: object
+            serviceAccountName:
+              description: Name of the service account that runs leader elections
+                for linstor
+              type: string
             sslSecret:
               description: Name of k8s secret that holds the SSL key for a node (called
                 `keystore.jks`) and the trusted certificates (called `certificates.jks`)

diff --git a/charts/piraeus/templates/controller-rbac.yml b/charts/piraeus/templates/controller-rbac.yml
@@ -6,3 +6,29 @@ kind: ServiceAccount
 metadata:
   name: linstor-controller
   namespace: {{ .Release.Namespace }}
+---
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: linstor-leader-elector
+  namespace: {{ .Release.Namespace }}
+rules:
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "watch", "list", "delete", "update", "create"]
+  - apiGroups: [""]
+    resources: ["endpoints"]
+    verbs: ["create", "patch", "update"]
+---
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: linstor-leader-elector
+  namespace: {{ .Release.Namespace }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: linstor-leader-elector
+subjects:
+  - kind: ServiceAccount
+    name: linstor-controller
diff --git a/charts/piraeus/templates/operator-controller.yaml b/charts/piraeus/templates/operator-controller.yaml
@@ -17,3 +17,5 @@ spec:
   affinity: {{ .Values.operator.controller.affinity | toJson }}
   tolerations: {{ .Values.operator.controller.tolerations | toJson}}
   resources: {{ .Values.operator.controller.resources | toJson }}
+  replicas: {{ .Values.operator.controller.replicas }}
+  serviceAccountName: linstor-controller
diff --git a/charts/piraeus/values.cn.yaml b/charts/piraeus/values.cn.yaml
@@ -51,22 +51,23 @@ operator:
   image: daocloud.io/piraeus/piraeus-operator:latest
   resources: {}
   controller:
-    controllerImage: daocloud.io/piraeus/piraeus-server:v1.7.1
+    controllerImage: daocloud.io/piraeus/piraeus-server:v1.9.0
     luksSecret: ""
     dbCertSecret: ""
     dbUseClientCert: false
     sslSecret: ""
     affinity: {}
     tolerations: []
     resources: {}
+    replicas: 1
   satelliteSet:
-    satelliteImage: daocloud.io/piraeus/piraeus-server:v1.7.1
+    satelliteImage: daocloud.io/piraeus/piraeus-server:v1.9.0
     storagePools: null
     sslSecret: ""
     automaticStorageType: None
     affinity: {}
     tolerations: []
     resources: {}
-    kernelModuleInjectionImage: daocloud.io/piraeus/drbd9-bionic:v9.0.24
+    kernelModuleInjectionImage: daocloud.io/piraeus/drbd9-bionic:v9.0.25
     kernelModuleInjectionMode: Compile
     kernelModuleInjectionResources: {}
diff --git a/charts/piraeus/values.yaml b/charts/piraeus/values.yaml
@@ -51,22 +51,23 @@ operator:
   image: quay.io/piraeusdatastore/piraeus-operator:latest
   resources: {}
   controller:
-    controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
+    controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.9.0
     luksSecret: ""
     dbCertSecret: ""
     dbUseClientCert: false
     sslSecret: ""
     affinity: {}
     tolerations: []
     resources: {}
+    replicas: 1
   satelliteSet:
-    satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
+    satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.9.0
     storagePools: null
     sslSecret: ""
     automaticStorageType: None
     affinity: {}
     tolerations: []
     resources: {}
-    kernelModuleInjectionImage: quay.io/piraeusdatastore/drbd9-bionic:v9.0.24
+    kernelModuleInjectionImage: quay.io/piraeusdatastore/drbd9-bionic:v9.0.25
     kernelModuleInjectionMode: Compile
     kernelModuleInjectionResources: {}
diff --git a/deploy/piraeus/templates/controller-rbac.yml b/deploy/piraeus/templates/controller-rbac.yml
@@ -7,5 +7,33 @@ metadata:
   namespace: default
 ---
 # Source: piraeus/templates/controller-rbac.yml
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: linstor-leader-elector
+  namespace: default
+rules:
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "watch", "list", "delete", "update", "create"]
+  - apiGroups: [""]
+    resources: ["endpoints"]
+    verbs: ["create", "patch", "update"]
+---
+# Source: piraeus/templates/controller-rbac.yml
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: linstor-leader-elector
+  namespace: default
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: linstor-leader-elector
+subjects:
+  - kind: ServiceAccount
+    name: linstor-controller
+---
+# Source: piraeus/templates/controller-rbac.yml
 # This YAML file contains all RBAC objects that are necessary to run a
 # LINSTOR controller pod
diff --git a/deploy/piraeus/templates/operator-controller.yaml b/deploy/piraeus/templates/operator-controller.yaml
@@ -12,10 +12,12 @@ spec:
   dbCertSecret: 
   dbUseClientCert: false
   drbdRepoCred: ""
-  controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
+  controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.9.0
   imagePullPolicy: "IfNotPresent"
   linstorHttpsControllerSecret: ""
   linstorHttpsClientSecret: ""
   affinity: {}
   tolerations: []
   resources: {}
+  replicas: 1
+  serviceAccountName: linstor-controller
diff --git a/deploy/piraeus/templates/operator-satelliteset.yaml b/deploy/piraeus/templates/operator-satelliteset.yaml
@@ -9,13 +9,13 @@ spec:
   sslSecret: 
   drbdRepoCred: ""
   imagePullPolicy: "IfNotPresent"
-  satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
+  satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.9.0
   linstorHttpsClientSecret: ""
   controllerEndpoint: http://piraeus-op-cs.default.svc:3370
   automaticStorageType: "None"
   affinity: {}
   tolerations: []
   resources: {}
   kernelModuleInjectionMode: "Compile"
-  kernelModuleInjectionImage: "quay.io/piraeusdatastore/drbd9-bionic:v9.0.24"
+  kernelModuleInjectionImage: "quay.io/piraeusdatastore/drbd9-bionic:v9.0.25"
   kernelModuleInjectionResources: {}
diff --git a/doc/helm-values.adoc b/doc/helm-values.adoc
@@ -208,7 +208,7 @@ Valid values:: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-po
 Description:: Affinity settings for controller pods. Can be used to restrict the pods to specific nodes.
 
 === `operator.controller.controllerImage`
-Default:: `quay.io/piraeusdatastore/piraeus-server:v1.7.1`
+Default:: `quay.io/piraeusdatastore/piraeus-server:v1.9.0`
 Valid values:: image ref
 Description:: Name of the image to use for the controller.
 
@@ -229,6 +229,11 @@ Default:: `""`
 Valid values:: secret name
 Description:: Name of the secret that contains the master passphrase to use for encrypted volumes. Check link:./security.md#automatically-set-the-passphrase-for-encrypted-volumes[the security guide].
 
+=== `operator.controller.replicas`
+Default:: `1`
+Valid values:: number
+Description:: Number of replicas to use for the Linstor controller.
+
 === `operator.controller.resources`
 Default:: `{}`
 Valid values:: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/[resource requests]
@@ -269,7 +274,7 @@ Description::  Automatically create storage pools of the specified type. Check t
 * `ZFS`: create a ZFS based storage pool
 
 === `operator.satelliteSet.kernelModuleInjectionImage`
-Default:: `quay.io/piraeusdatastore/drbd9-bionic:v9.0.24`
+Default:: `quay.io/piraeusdatastore/drbd9-bionic:v9.0.25`
 Valid values:: image ref
 Description:: Name of the image to use for loading kernel modules. This is specific to the nodes host system. Check https://quay.io/organization/piraeusdatastore[the available `drbd9` images]
 
@@ -302,7 +307,7 @@ Description:: Resource requests and limits to apply to the satellite containers.
 Note: at least 750MiB memory is recommended.
 
 === `operator.satelliteSet.satelliteImage`
-Default:: `quay.io/piraeusdatastore/piraeus-server:v1.7.1`
+Default:: `quay.io/piraeusdatastore/piraeus-server:v1.9.0`
 Valid values:: image ref
 Description:: Name of the image to use for the satellites.
 

diff --git a/go.mod b/go.mod
@@ -5,13 +5,13 @@ go 1.13
 require (
 	github.com/BurntSushi/toml v0.3.1
 	github.com/LINBIT/golinstor v0.26.1-0.20200520122514-71747751b6af
+	github.com/linbit/k8s-await-election v0.2.0
 	github.com/operator-framework/operator-sdk v0.16.0
-	github.com/sirupsen/logrus v1.4.2
+	github.com/sirupsen/logrus v1.6.0
 	github.com/spf13/pflag v1.0.5
-	github.com/stretchr/testify v1.4.0
 	gopkg.in/ini.v1 v1.56.0
 	k8s.io/api v0.0.0
-	k8s.io/apimachinery v0.0.0
+	k8s.io/apimachinery v0.18.4
 	k8s.io/client-go v12.0.0+incompatible
 	sigs.k8s.io/controller-runtime v0.4.0
 )

diff --git a/go.sum b/go.sum
@@ -421,6 +421,8 @@ github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgo
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.2 h1:DB17ag19krx9CFsz4o3enTrPXyIXCl+2iCXH/aMAp9s=
 github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8=
+github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
@@ -434,6 +436,8 @@ github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
 github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
 github.com/libopenstorage/openstorage v1.0.0/go.mod h1:Sp1sIObHjat1BeXhfMqLZ14wnOzEhNx2YQedreMcUyc=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE=
+github.com/linbit/k8s-await-election v0.2.0 h1:ICvl2wd4h6mWgsdMumJ6e2/NS0Mp11DvK2cM/a3l2jw=
+github.com/linbit/k8s-await-election v0.2.0/go.mod h1:VCRtUTvVQmfNyqW7OSNyCOCh9mi29fgQ75XtUIfP5WE=
 github.com/lithammer/dedent v1.1.0/go.mod h1:jrXYCQtgg0nJiN+StA2KgR7w6CiQNv9Fd/Z9BP0jIOc=
 github.com/lithammer/shortuuid v3.0.0+incompatible h1:NcD0xWW/MZYXEHa6ITy6kaXN5nwm/V115vj2YXfhS0w=
 github.com/lithammer/shortuuid v3.0.0+incompatible/go.mod h1:FR74pbAuElzOUuenUHTK2Tciko1/vKuIKS9dSkDrA4w=
@@ -623,6 +627,8 @@ github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPx
 github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
 github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
 github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
+github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I=
+github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
 github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a h1:pa8hGb/2YqsZKovtsgrwcDH1RZhVbTKCjLp47XpqCDs=

diff --git a/pkg/apis/piraeus/v1/linstorcontroller_types.go b/pkg/apis/piraeus/v1/linstorcontroller_types.go
@@ -88,6 +88,15 @@ type LinstorControllerSpec struct {
 	// +nullable
 	Tolerations []corev1.Toleration `json:"tolerations"`
 
+	// Number of replicas in the controller deployment
+	// +optional
+	// +nullable
+	Replicas *int32 `json:"replicas"`
+
+	// Name of the service account that runs leader elections for linstor
+	// +optional
+	ServiceAccountName string `json:"serviceAccountName"`
+
 	shared.LinstorClientConfig `json:",inline"`
 }
 

diff --git a/pkg/apis/piraeus/v1/zz_generated.deepcopy.go b/pkg/apis/piraeus/v1/zz_generated.deepcopy.go