Enable optional kubernetes features by default

[claudiubelu]

Is this a request for help?: No.

---

Is this an ISSUE or FEATURE REQUEST? (choose one): ISSUE

---

What version of acs-engine?: 0.17.1 or newer.

---

Some Kubernetes features are not enabled by default (LocalStorageCapacityIsolation, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, tokencleaner, bootstrapsigner), which can cause some features not to work and thus, some Kubernetes E2E tests not to pass.

Feature tests that would be passing if the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controls are enabled [1]:

Should be able to deny custom resource creation
Should be able to deny pod and configmap creation
Should mutate configmap
Should mutate crd
Should mutate pod and apply defaults after mutation
Should unconditionally reject operations on fail closed webhook
Should not be able to prevent deleting validating-webhook-configurations or mutating-webhook-configurations
Should mutate custom resource
Should deny crd creation
don't cause replicaset controller creating extra pods if the initializer is not handled
should be invisible to controllers by default
should dynamically register and apply initializers to pods
will be set to nil if a patch removes the last pending initializer

Features that would be passing if the LocalStorageCapacityIsolation feature gate would be enabled [2]:

should create a ResourceQuota and capture the life of a pod
should create a ResourceQuota and capture the life of a replica set
should create a ResourceQuota and capture the life of a replication controller
should create a ResourceQuota and capture the life of a secret
should create a ResourceQuota and capture the life of a service
should create a ResourceQuota and capture the life of a persistent volume claim with a storage class
should create a ResourceQuota and capture the life of a persistent volume claim
should create a ResourceQuota and capture the life of a configMap
should create a ResourceQuota and capture the life of an uninitialized pod
validates local ephemeral storage resource limits of pods that are allowed to run
should provide container's limits.ephemeral-storage and requests.ephemeral-storage as env vars
should provide default limits.ephemeral-storage from node allocatable
should create a LimitRange with default ephemeral storage and ensure pod has the default applied

Features that would be passing if the tokencleaner and bootstrapsigner controllers would be enabled [3]:

should sign the new added bootstrap tokens
should resign the bootstrap tokens when the clusterInfo ConfigMap updated
should not delete the token secret when the secret is not expired
should delete the token secret when the secret expired
should delete the signed bootstrap tokens from clusterInfo ConfigMap when bootstrap token is deleted

These features have to be enabled manually in order to benefit from them.

[1] https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
[2] https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
[3] https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes

What happened:

The tests are skipped / failing. The respective features are disabled.

What you expected to happen:

The tests should pass. The respective features to be enabled.

How to reproduce it (as minimally and precisely as possible):

Create a Kubernetes deployment with minimal configuration [4] and check the various features which are enabled by the optional Kubernetes controllers and features.

[4] https://github.com/Azure/acs-engine/blob/fc56e540144aad66baf769446d1d87ebc86e0911/examples/kubernetes.json

Anything else we need to know:

[PatrickLang]

@ritazh - these are probably needed for Linux conformance too. Can you take a look?