Make kubefed webhook replica count configurable

[nitinatgh]

So it can be Highly Available (HA)
Added in description to the README.md
Bumped up the chart version of the controller manager
Bumped up the dependency version

What this PR does / why we need it: This PR gives the users an option to set the number of replicas for the Kubefed Admission Webhook

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1456

Special notes for your reviewer:
Tested it out as follows in a local kind cluster

$ kubectl get pods -A
NAMESPACE                NAME                                             READY   STATUS    RESTARTS   AGE
...
kube-federation-system   kubefed-admission-webhook-67fb48d5f7-kw4ps       1/1     Running   0          4d17h
...

helm upgrade dry-run

$ helm --namespace kube-federation-system upgrade -i kubefed ./kubefed --create-namespace --dry-run
...
# Source: kubefed/charts/controllermanager/templates/deployments.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kube-federation-system
  name: kubefed-admission-webhook
  labels:
    kubefed-admission-webhook: "true"
spec:
  replicas: 2
  selector:
    matchLabels:
...

helm upgrade

$ helm --namespace kube-federation-system upgrade -i kubefed ./kubefed --create-namespace
Release "kubefed" has been upgraded. Happy Helming!
NAME: kubefed
LAST DEPLOYED: Tue Sep 14 10:33:35 2021
NAMESPACE: kube-federation-system
STATUS: deployed
REVISION: 13
TEST SUITE: None

$ kubectl get pods -A
NAMESPACE                NAME                                             READY   STATUS    RESTARTS   AGE
...
kube-federation-system   kubefed-admission-webhook-67fb48d5f7-kw4ps       1/1     Running   0          4d17h
kube-federation-system   kubefed-admission-webhook-67fb48d5f7-r4vf5       1/1     Running   0          93s
...

[nitinatgh]

/assign @xunpan
/assign @makkes

[nitinatgh]

@xunpan @makkes , Do you know how to update the helm charts to reflect these new changes? When the charts gets pulled remotely they still have the old charts and not the new updated ones? Does a new tag need to be created, 0.8.2?
Any help on this would be great as we'd like to start using the charts with the latest updates in our clusters please.
Thanks!

[nitinatgh]

/assign @jimmidyson

[nitinatgh]

/retest

[k8s-ci-robot]

@nitinatgh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nitinatgh]

Can someone take a look at this PR for approval please?
@jimmidyson @xunpan @makkes
Thanks!

[makkes]

/lgtm

[makkes]

/ok-to-test

[makkes]

/retest

[nitinatgh]

/retest

[irfanurrehman]

@nitinatgh Was going through pending issues and PRs and noticed this. Can you please rebase this. I think the unrelated test failure has been fixed by #1500

[nitinatgh]

@irfanurrehman @makkes
Have updated with master and tests seems to be passing, so looks all good to go I hope?
Thanks!

[irfanurrehman]

Thanks @nitinatgh for doing this
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: irfanurrehman, makkes, nitinatgh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [irfanurrehman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment