Investigate Operator Framework?

[wongma7]

CoreOS has come up with this "operator" concept. https://coreos.com/operators/. What do we think about it? Should we use their SDK, which is pre-alpha in our library? Immediate benefits I see are:

• simplify our codebase. (I am not sure about this one, we have enough expertise as it is, the controller is really not that complicated being ~1000 lines, and it is not that different from upstream controllers)
• get RBAC specs generated for free (this has always been a problem, every provisioner ships their own RBAC yamls, there is a default clusterrole in Kubernetes, etc., it is IMO a mess)
• get HA/leader election for free (see optimize leader election and abuse api server #837)
• better UX when deploying provisioners because we follow what is maybe becoming the "standard" way to deploy for-kube applications (see also RBAC point above)
• some automatic upgrading stuff which I'm not 100% clear on. Pretty important if upstream continues to add things to PV/PVC/StorageClass specs

Downsides:

• we don't need any of the CRD stuff, it seems to me like all operator examples involve CRDs somehow
• it's "pre-alpha"
• UX may actually be worse if operators don't become "standard" and cluster admins are burdened with understanding what an operator is

If we don't use the SDK/"Framework" we should at least take some principles from them and apply them here.

If anybody has expertise in this area, maybe somebody from CoreOS is watching :), please kindly share.

[wongma7]

I still am not sure what the heck an operator is tbh. But the result of my "investigation" is we are going to do this regardless :))

[orainxiong]

kubebuilder may be another not bad option.

I know a little bit about both, I still try to figure out the difference.

IMO we can choose any one of them, kubebuilder is a little closer to raw kubernetes code that could give us flexibility along with complexity, and operator-sdk provide us an abstract layer atop of raw kubernetes library to make writing the operator much easier but limited.

In a word, it is a matter of functionality and convenience.