manager: make leader election timeouts configurable

[nixpanic]

Adding the following options:

  -leader-election-lease-duration duration
        Duration, in seconds, that non-leader candidates will wait to
        force acquire leadership. Defaults to 15 seconds. (default 15s)
  -leader-election-renew-deadline duration
        Duration, in seconds, that the acting leader will retry
	refreshing leadership before giving up. Defaults to 10 seconds.
        (default 10s)
  -leader-election-retry-period duration
	Duration, in seconds, the LeaderElector clients should wait
        between tries of actions. Defaults to 5 seconds. (default 5s)

Some container platforms (like OpenShift) have recommendations for these
timeout that differ from the Kubernetes defaults. With these options,
deployments can define the timeouts according to their own preferences.

[nixpanic]

@yati1998 , you should be able to test this out by editing the deployment for the controller, and

1. use quay.io/nixpanic/csi-addons-k8s-controller:pr_505 as image

2. add the following arguments to the manager executable:

        - "--leader-election-lease-duration=137s"
        - "--leader-election-renew-deadline=107s"
        - "--leader-election-retry-period=26s"
   

3. make sure the pod comes up again

4. verify that the manager obtained the lease (check its logs)

After that, run must-gather again and see if the controller restarts (or not, ideally).

[Madhu-1]

set below values in the template https://github.com/rook/rook/blob/3c38e04b6d34bbcef45bd3f704254f026efcc991/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml#L34-L36 as this is suggested by openshift storage team for csi sidecars or even make this as default?

[nixpanic]

I've intentionally not modified those, but kept them at the recommendation from the controller-runtime maintainers. OpenShift specific recommendations do not really belong in a Kubernetes focused project, any product/distribution of this operator can patch the options or modify the deployment with settings they prefer.

[yati1998]

I am not sure if taking it in makes much difference, I have tested the image that @nixpanic has shared above. And the issue still persists. @nixpanic did you get the time to check this?

[nixpanic]

I am not sure if taking it in makes much difference, I have tested the image that @nixpanic has shared above. And the issue still persists. @nixpanic did you get the time to check this?

This change is unfortunately not addressing the issue that you reported. But, it is a general enhancement, as the default lease timeout/period/retry is not optimal for some Kubernetes distributions (like OpenShift). Most applications that use leases allow the parameters to be configurable, the kubernetes-csi-addons does not have that at the moment.