Using CRD or annotations to store cluster info in Kubernetes

[ghost]

Submission type

• Request for enhancement (RFE)

Environment

Kubernetes

Stolon version

9fd8393

Enhancement Description

I've been reviewing stolon to set it up in my kubernetes cluster and I happened to see a suggestion in HN:

If your pattern requires you to connect to etcd, then you're not taking the best advantage of Kubernetes.
Yes, you can connect to an etcd instance outside of Kubernetes, or deploy etcd on Kubernetes. CoreOS developed the etcd operator for the purposes of self-hosting the system KV store, but you can of course run a "user" etcd.
A better pattern is relying on the objects provided by Kubernetes, which are themselves backed by a lock service (which happens to be etcd, but that's an implementation detail you shouldn't worry about.)
You could, for example, use an Annotation on an object to do leader election. Look at how HA masters are implemented on Kubernetes, for example:
https://github.com/kubernetes/kubernetes/blob/master/pkg/client/leaderelection/leaderelection.go

His comment seems to fair and convincing, I'm not much aware of implementation details of stolon but using kubernetes native features is very neat and avoids worrying about etcd management and maintenance which is proved to be troublesome.

[sgotti]

@k1-hedayati we could use CRD for the cluster data and locking features for sentinel election but we use the store also for discovery of the keepers and proxies and this relies on key ttl expiration.

So to move to use only the k8s api server as the store there's the need to rearchitect discovery to work with it. In the past we had (see the git history) k8s based keeper's pod discovery using label selectors and exposing an api on the keeper to fetch it's state but we removed it since it wasn't really helpful due to lack, at that time, of crd.

This way of doing discovery could be reintroduced and, if someone wants to propose a design doc on how to do this. I'll happily review and discuss this.

[ghost]

Unfortunately in my current situation I cannot spend much time on this but I found out that patroni has implemented this functionality, architectures may differ but it could be a good place to start. Also there is Postgres Operator which is using CRD too.

I'm aiming for minimum worry and maintenance setup, as you very well know etcd setups are inconsistent and fragile, coreos/etcd-operator#1323 is still WIP too and my knowledge on etcd is very limited so I don't know if it's possible to have resilient setup without etcd opertator. I saw your solution on this, is it reliable for production use or its just for example?

Edit: Thanks to your efforts it seems PV support for operator has been merged recently so I guess now it's possible to have a resilient setup.

[sgotti]

@k1-hedayati I got your points but won't try to copy other solutions since they have different features and architectures. I think we should start from my previous comment and provide an initial design doc. This change will require some effort also to improve the test suite to run the tests also on a k8s cluster.

I'd be really happy if someone in the community jumps in and help us implementing this or wants to sponsor its development.

About the etcd operator vs statefulset let's discuss this in the mailing list to keep this issue clean.

[sgotti]

I did an initial POC. This is how it basically works:

• Use a configmap (`stolon-cluster-$CLUSTERNAME) and use its annotation to save the clusterdata and for sentinel leader election.
• The stolon keepers, proxies, sentinels will update their own pod resources adding an annotation to report their status (keeperInfos, proxyInfos, sentinelInfos). So we'll solve the ttl problem because it'll automatically be removed when a pod is deleted.
• The sentinel/stolonctl will discover keepers and proxies (stolonctl also sentinels) listing pods by a label selector with a label stolon-cluster set to the cluster name and reading the above metadata.
• Additionally the keepers/sentinels/proxies could automatically set the stolon-cluster label of their pod to avoid the user to keep in sync the pod stolon-cluster label and the --cluster-name command line option.
• stolonctl have to be executed inside a k8s cluster (as a second step we could add all the external kubeconfig/cluster/context options).

NOTE: it won't use CRD because CRD is for custom resources managed by a controller/operator and doesn't fit well to save the clusterdata.

[ghost]

For creating right RBAC rules there is audit2rbac which looks at audit logs provided by the api server.

[sgotti]

Created #433 . I'll be really happy if someone could test that PR.

[sgotti]

Implemented in #433 .