Scale Down may not wait long enough before destroying PODs

[maddisondavid]

The Scale Down logic currently waits for the ConfigMap to be updated and then 6 reconcile loops before updating the Statefulset. The reconcile Loop wait I suspect is a way of waiting until the PODs have received the new ConfigMap environment variables.

The problem is that this sync be up to 1 minute (by default)

As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period (1 minute by default) + ttl of ConfigMaps cache (1 minute by default) in kubelet.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically

Just waiting 6 reconcile loops does not seem a valid indicator of time as the reconcile loop can be called multiple times depending on what watches have fired.

Problem Location
https://github.com/pravega/zookeeper-operator/blob/master/pkg/controller/zookeepercluster/zookeepercluster_controller.go#L207-L210

[pbelgundi]

Fixed by PR: #120