Enable leader election in csi-resizer sidecar

[rdpsin]

This commit enables leader election in the csi-resizer sidecar. Without it, a resize request is routed to all instances of the sidecar, which would cause all but one of them to fail.

Is this a bug fix or adding new feature?
Feature

What is this PR about? / Why do we need it?

What testing is done?

Without leader election:

$ k describe pvc ebs-claim                                    
Name:          ebs-claim
Namespace:     default
StorageClass:  resize-sc
Status:        Bound
Volume:        pvc-7c054c55-c253-449c-97f2-09fdf6edcf75
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      100Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       app
Events:
  Type     Reason                      Age                From                                                                                      Message
  ----     ------                      ----               ----                                                                                      -------
  Warning  ProvisioningFailed          22s                persistentvolume-controller                                                               storageclass.storage.k8s.io "resize-sc" not found
  Normal   ExternalProvisioning        20s (x2 over 20s)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
  Normal   Provisioning                20s                ebs.csi.aws.com_ebs-csi-controller-6ccfd5dfb7-qk9f7_7f9a97f2-d5b9-4749-a315-8bb1bb748992  External provisioner is provisioning volume for claim "default/ebs-claim"
  Normal   ProvisioningSucceeded       17s                ebs.csi.aws.com_ebs-csi-controller-6ccfd5dfb7-qk9f7_7f9a97f2-d5b9-4749-a315-8bb1bb748992  Successfully provisioned volume pvc-7c054c55-c253-449c-97f2-09fdf6edcf75
  Warning  ExternalExpanding           8s                 volume_expand                                                                             Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   Resizing                    7s                 external-resizer ebs.csi.aws.com                                                          External resizer is resizing volume pvc-7c054c55-c253-449c-97f2-09fdf6edcf75
  Warning  VolumeResizeFailed          5s                 external-resizer ebs.csi.aws.com                                                          Mark PVC "default/ebs-claim" as file system resize required failed: can't patch status of  PVC default/ebs-claim with Operation cannot be fulfilled on persistentvolumeclaims "ebs-claim": the object has been modified; please apply your changes to the latest version and try again
  Normal   Resizing                    4s (x2 over 8s)    external-resizer ebs.csi.aws.com                                                          External resizer is resizing volume pvc-7c054c55-c253-449c-97f2-09fdf6edcf75
  Normal   FileSystemResizeRequired    4s                 external-resizer ebs.csi.aws.com                                                          Require file system resize of volume on node
  Warning  VolumeResizeFailed          4s                 external-resizer ebs.csi.aws.com                                                          Mark PVC "default/ebs-claim" as file system resize required failed: can't patch status of  PVC default/ebs-claim with Operation cannot be fulfilled on persistentvolumeclaims "ebs-claim": the object has been modified; please apply your changes to the latest version and try again
  Normal   FileSystemResizeSuccessful  0s                 kubelet                                                                                   MountVolume.NodeExpandVolume succeeded for volume "pvc-7c054c55-c253-449c-97f2-09fdf6edcf75" ip-192-168-92-57.us-east-2.compute.internal

With leader election

$ k describe pvc ebs-claim                                     
Name:          ebs-claim
Namespace:     default
StorageClass:  resize-sc
Status:        Bound
Volume:        pvc-2c166604-11ba-46cb-9bce-48567ec883df
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
             pv.kubernetes.io/bound-by-controller: yes
             volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
             volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      150Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       app
Events:
Type     Reason                      Age                From                                                                                      Message
----     ------                      ----               ----                                                                                      -------
Warning  ProvisioningFailed          42s                persistentvolume-controller                                                               storageclass.storage.k8s.io "resize-sc" not found
Normal   Provisioning                41s                ebs.csi.aws.com_ebs-csi-controller-6ccfd5dfb7-qk9f7_7f9a97f2-d5b9-4749-a315-8bb1bb748992  External provisioner is provisioning volume for claim "default/ebs-claim"
Normal   ExternalProvisioning        41s (x2 over 41s)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Normal   ProvisioningSucceeded       38s                ebs.csi.aws.com_ebs-csi-controller-6ccfd5dfb7-qk9f7_7f9a97f2-d5b9-4749-a315-8bb1bb748992  Successfully provisioned volume pvc-2c166604-11ba-46cb-9bce-48567ec883df
Warning  ExternalExpanding           23s                volume_expand                                                                             Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
Normal   Resizing                    23s                external-resizer ebs.csi.aws.com                                                          External resizer is resizing volume pvc-2c166604-11ba-46cb-9bce-48567ec883df
Normal   FileSystemResizeRequired    17s                external-resizer ebs.csi.aws.com                                                          Require file system resize of volume on node
Normal   FileSystemResizeSuccessful  9s                 kubelet                                                                                   MountVolume.NodeExpandVolume succeeded for volume "pvc-2c166604-11ba-46cb-9bce-48567ec883df" ip-192-168-92-57.us-east-2.compute.internal

[hanyuel]

/lgtm

[torredil]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment