Add option for setting tolerations

[olemarkus]

In addition to the options added in #19 it would be useful for us to be able to set tolerations.
The use case for us would be to be able to schedule the pods to Kops master nodes.

[aanm]

@olemarkus this is available since v2.0.6 let me know if it works fine for you

https://github.com/cilium/cilium-etcd-operator#optional-deployment-with-an-existing-etcdcluster-custom-resource

[olemarkus]

I don't really think this will work. The etcd-operator still wouldn't be scheduled. The CRD definitions also doesn't exist in the cluster at that time a new cluster is created.

But I am not 100% sure this is the best way to get a dedicated etcd cluster for cilium using kops either.

[olemarkus]

The toleration required for running on kops master nodes is:

  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
    operator: Exists

In addition, one may want to have a selector set:

  nodeSelector:
    kubernetes.io/role: master