Skip to content

Commit

Permalink
support multiple replicas for linstor-controller
Browse files Browse the repository at this point in the history
Running multiple replicas requires special support from the linstor
controller. The controller container will start a leader election
process when it detects the presence of the K8S_AWAIT_ELECTION_*
variables.

The election process determines which pod is allowed to start the
linstor-controller process. Only this pod will be added as endpoint for the
controller service.

Should the leader crash or the node its running on goes offline,
a new leader will be elected and allowed to start the controller process.
Note: in case the full node goes offline, the old pod will still be marked
as ready. By using ClusterIP: "" on our service, we ensure we create an actual
proxy (which automatically chooses the responding pod) instead of each client
having to try multiple DNS responses.
  • Loading branch information
WanzenBug committed Sep 7, 2020
1 parent 8eaa0db commit 02304a2
Show file tree
Hide file tree
Showing 17 changed files with 253 additions and 78 deletions.
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Breaking

* The LINSTOR controller image given in `operator.controller.controllerImage` has to have
its entrypoint set to [`k8s-await-election v0.2.0`](https://github.com/LINBIT/k8s-await-election/)
or newer. Learn more in the [upgrade guide](./UPGRADE.md#upgrade-from-v10-to-head).

### Added

* LINSTOR controller can be started with multiple replicas. See [`operator.controller.replicas`](./doc/helm-values.adoc#operatorcontrollerreplicas).
NOTE: This requires support from the container. You need `piraeus-server:v1.8.0` or newer.
* All pods use a dedicated service account to allow for fine-grained permission control.
* The new [helm section `psp.*`](./doc/helm-values.adoc#psp) can automatically configure the ServiceAccount
of all components to use the appropriate [PSP](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) roles.

### Changed

* Default values:
* `operator.controller.controllerImage`: `quay.io/piraeusdatastore/piraeus-server:v1.8.0`
* `operator.satelliteSet.satelliteImage`: `quay.io/piraeusdatastore/piraeus-server:v1.8.0`
* linstor-controller no longer starts in a privileged container.

### Removed
Expand Down
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,24 @@ this [example chart configuration.](./examples/resource-requirements.yaml)
[resource requests and limits]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
### Running multiple replicas
Running multiple replicas of pods is recommended for high availability and fast error recovery.
The following components can be started with multiple replicas:
* Operator: Set [`operator.replicas`] to the desired number of operator pods.
* CSI: Set [`csi.controllerReplicas`] to the desired number of CSI Controller pods.
* Linstor Controller: Set [`operator.controller.replicas`] to the desired number of LINSTOR controller pods.
* CSI Snapshotter: Set [`csi-snapshotter.replicas`] to the desired number of CSI Snapshot Controller pods.
* Etcd: Set [`etcd.replicas`] to the desired number of Etcd pods.
* Stork: Set [`stork.replicas`] to the desired number of both Stork plugin and Stork scheduler pods.
[`operator.replicas`]: ./doc/helm-values.adoc#operatorreplicas
[`csi.controllerReplicas`]: ./doc/helm-values.adoc#csicontrollerreplicas
[`operator.controller.replicas`]: ./doc/helm-values.adoc#operatorcontrollerreplicas
[`csi-snapshotter.replicas`]: ./doc/helm-values.adoc#csi-snapshotterreplicas
[`etcd.replicas`]: ./doc/helm-values.adoc#etcdreplicas
[`stork.replicas`]: ./doc/helm-values.adoc#storkreplicas
### Terminating Helm deployment
Expand Down
11 changes: 10 additions & 1 deletion UPGRADE.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@
The following document describes how to convert
# Upgrade from v1.0 to HEAD

* The LINSTOR controller image given in `operator.controller.controllerImage` has to have
its entrypoint set to [`k8s-await-election v0.2.0`](https://github.com/LINBIT/k8s-await-election/)
or newer. All images starting with `piraeus-server:v1.8.0` meet this requirement.

Older images will not work, as the `Service` will not automatically pick up on the active pod.

To upgrade, first update the deployed LINSTOR image to a compatible version, then upgrade the
operator.

# Upgrade to v1.0

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -657,6 +657,11 @@ spec:
description: priorityClassName is the name of the PriorityClass for
the controller pods
type: string
replicas:
description: Number of replicas in the controller deployment
format: int32
nullable: true
type: integer
resources:
description: Resource requirements for the LINSTOR controller pod
nullable: true
Expand All @@ -676,6 +681,10 @@ spec:
value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
type: object
type: object
serviceAccountName:
description: Name of the service account that runs leader elections
for linstor
type: string
sslSecret:
description: Name of k8s secret that holds the SSL key for a node (called
`keystore.jks`) and the trusted certificates (called `certificates.jks`)
Expand Down
26 changes: 26 additions & 0 deletions charts/piraeus/templates/controller-rbac.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,29 @@ kind: ServiceAccount
metadata:
name: linstor-controller
namespace: {{ .Release.Namespace }}
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: linstor-leader-elector
namespace: {{ .Release.Namespace }}
rules:
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["create", "patch", "update"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: linstor-leader-elector
namespace: {{ .Release.Namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: linstor-leader-elector
subjects:
- kind: ServiceAccount
name: linstor-controller
2 changes: 2 additions & 0 deletions charts/piraeus/templates/operator-controller.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,5 @@ spec:
affinity: {{ .Values.operator.controller.affinity | toJson }}
tolerations: {{ .Values.operator.controller.tolerations | toJson}}
resources: {{ .Values.operator.controller.resources | toJson }}
replicas: {{ .Values.operator.controller.replicas }}
serviceAccountName: linstor-controller
5 changes: 3 additions & 2 deletions charts/piraeus/values.cn.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,17 @@ operator:
image: daocloud.io/piraeus/piraeus-operator:latest
resources: {}
controller:
controllerImage: daocloud.io/piraeus/piraeus-server:v1.7.1
controllerImage: daocloud.io/piraeus/piraeus-server:v1.8.0
luksSecret: ""
dbCertSecret: ""
dbUseClientCert: false
sslSecret: ""
affinity: {}
tolerations: []
resources: {}
replicas: 1
satelliteSet:
satelliteImage: daocloud.io/piraeus/piraeus-server:v1.7.1
satelliteImage: daocloud.io/piraeus/piraeus-server:v1.8.0
storagePools: null
sslSecret: ""
automaticStorageType: None
Expand Down
5 changes: 3 additions & 2 deletions charts/piraeus/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,17 @@ operator:
image: quay.io/piraeusdatastore/piraeus-operator:latest
resources: {}
controller:
controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.8.0
luksSecret: ""
dbCertSecret: ""
dbUseClientCert: false
sslSecret: ""
affinity: {}
tolerations: []
resources: {}
replicas: 1
satelliteSet:
satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.8.0
storagePools: null
sslSecret: ""
automaticStorageType: None
Expand Down
28 changes: 28 additions & 0 deletions deploy/piraeus/templates/controller-rbac.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,33 @@ metadata:
namespace: default
---
# Source: piraeus/templates/controller-rbac.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: linstor-leader-elector
namespace: default
rules:
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["create", "patch", "update"]
---
# Source: piraeus/templates/controller-rbac.yml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: linstor-leader-elector
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: linstor-leader-elector
subjects:
- kind: ServiceAccount
name: linstor-controller
---
# Source: piraeus/templates/controller-rbac.yml
# This YAML file contains all RBAC objects that are necessary to run a
# LINSTOR controller pod
4 changes: 3 additions & 1 deletion deploy/piraeus/templates/operator-controller.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@ spec:
dbCertSecret:
dbUseClientCert: false
drbdRepoCred: ""
controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.8.0
imagePullPolicy: "IfNotPresent"
linstorHttpsControllerSecret: ""
linstorHttpsClientSecret: ""
affinity: {}
tolerations: []
resources: {}
replicas: 1
serviceAccountName: linstor-controller
2 changes: 1 addition & 1 deletion deploy/piraeus/templates/operator-satelliteset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ spec:
sslSecret:
drbdRepoCred: ""
imagePullPolicy: "IfNotPresent"
satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
satelliteImage: quay.io/piraeusdatastore/piraeus-server:v1.8.0
linstorHttpsClientSecret: ""
controllerEndpoint: http://piraeus-op-cs.default.svc:3370
automaticStorageType: "None"
Expand Down
9 changes: 7 additions & 2 deletions doc/helm-values.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ Valid values:: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-po
Description:: Affinity settings for controller pods. Can be used to restrict the pods to specific nodes.

=== `operator.controller.controllerImage`
Default:: `quay.io/piraeusdatastore/piraeus-server:v1.7.1`
Default:: `quay.io/piraeusdatastore/piraeus-server:v1.8.0`
Valid values:: image ref
Description:: Name of the image to use for the controller.

Expand All @@ -214,6 +214,11 @@ Default:: `""`
Valid values:: secret name
Description:: Name of the secret that contains the master passphrase to use for encrypted volumes. Check link:./security.md#automatically-set-the-passphrase-for-encrypted-volumes[the security guide].

=== `operator.controller.replicas`
Default:: `1`
Valid values:: number
Description:: Number of replicas to use for the Linstor controller.

=== `operator.controller.resources`
Default:: `{}`
Valid values:: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/[resource requests]
Expand Down Expand Up @@ -287,7 +292,7 @@ Description:: Resource requests and limits to apply to the satellite containers.
Note: at least 750MiB memory is recommended.

=== `operator.satelliteSet.satelliteImage`
Default:: `quay.io/piraeusdatastore/piraeus-server:v1.7.1`
Default:: `quay.io/piraeusdatastore/piraeus-server:v1.8.0`
Valid values:: image ref
Description:: Name of the image to use for the satellites.

Expand Down
5 changes: 3 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@ go 1.13
require (
github.com/BurntSushi/toml v0.3.1
github.com/LINBIT/golinstor v0.26.1-0.20200520122514-71747751b6af
github.com/linbit/k8s-await-election v0.2.0
github.com/operator-framework/operator-sdk v0.16.0
github.com/sirupsen/logrus v1.4.2
github.com/sirupsen/logrus v1.6.0
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.4.0
gopkg.in/ini.v1 v1.56.0
k8s.io/api v0.0.0
k8s.io/apimachinery v0.0.0
k8s.io/apimachinery v0.18.4
k8s.io/client-go v12.0.0+incompatible
sigs.k8s.io/controller-runtime v0.4.0
)
Expand Down
6 changes: 6 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,8 @@ github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgo
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.2 h1:DB17ag19krx9CFsz4o3enTrPXyIXCl+2iCXH/aMAp9s=
github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
Expand All @@ -434,6 +436,8 @@ github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/libopenstorage/openstorage v1.0.0/go.mod h1:Sp1sIObHjat1BeXhfMqLZ14wnOzEhNx2YQedreMcUyc=
github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE=
github.com/linbit/k8s-await-election v0.2.0 h1:ICvl2wd4h6mWgsdMumJ6e2/NS0Mp11DvK2cM/a3l2jw=
github.com/linbit/k8s-await-election v0.2.0/go.mod h1:VCRtUTvVQmfNyqW7OSNyCOCh9mi29fgQ75XtUIfP5WE=
github.com/lithammer/dedent v1.1.0/go.mod h1:jrXYCQtgg0nJiN+StA2KgR7w6CiQNv9Fd/Z9BP0jIOc=
github.com/lithammer/shortuuid v3.0.0+incompatible h1:NcD0xWW/MZYXEHa6ITy6kaXN5nwm/V115vj2YXfhS0w=
github.com/lithammer/shortuuid v3.0.0+incompatible/go.mod h1:FR74pbAuElzOUuenUHTK2Tciko1/vKuIKS9dSkDrA4w=
Expand Down Expand Up @@ -623,6 +627,8 @@ github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPx
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I=
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a h1:pa8hGb/2YqsZKovtsgrwcDH1RZhVbTKCjLp47XpqCDs=
Expand Down
9 changes: 9 additions & 0 deletions pkg/apis/piraeus/v1/linstorcontroller_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,15 @@ type LinstorControllerSpec struct {
// +nullable
Tolerations []corev1.Toleration `json:"tolerations"`

// Number of replicas in the controller deployment
// +optional
// +nullable
Replicas *int32 `json:"replicas"`

// Name of the service account that runs leader elections for linstor
// +optional
ServiceAccountName string `json:"serviceAccountName"`

shared.LinstorClientConfig `json:",inline"`
}

Expand Down
5 changes: 5 additions & 0 deletions pkg/apis/piraeus/v1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 02304a2

Please sign in to comment.