Helm cannot be upgraded in ACS

[squillace]

Is this a request for help?:
No.

Is this an ISSUE or FEATURE REQUEST? (choose one):
Issue

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
k8s: Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

What happened:
ACS cluster created and installed with helm 2.5.1 works fine. Draft installed through helm 2.5.1. Helm 2.6.0 comes out along with draft 0.6. CANNOT REMOVE HELM 2.5.1. Want client features, which depend upon server portion, "tiller".

What you expected to happen:
I created the cluster. I called helm init and it installed tiller. I expect to be able to remove tiller and upgrade it. Instead, I CAN upgrade helm to 2.6, which merely creates a new svc for a moment, and then that svc gets drained and the original replaces it, always.

How to reproduce it (as minimally and precisely as possible):

1. new ACS cluster, eastus.
2. helm init (for version 2.5.1)
3. draft init (version 0.6.rc2 or some such).
4. download and install helm 2.6.0. Helm init --upgrade
5. returns success. kubectl get po --all-namespaces will show the creation of a new tiller for a moment, only to be replaced by more tiller pods, which were created from 2.5.1.
6. Try whatever you want: k delete deployment tiller-deploy --namespace kube-system`, anything. It comes back within twenty seconds.

Anything else we need to know:
Draft cannot be upgraded if helm is also upgraded, meaning that ACS is a bit crippled for these tools when you create the cluster yourself. I should not have to recreate the cluster just to upgrade helm unless I don't own the cluster.

NOTE
None of this is true for clusters that were created for you; it's entirely possible that I should not be able to remove a tiller version for THOSE clusters, but I expect we don't have that implemented yet.

tiller installed here in acs-engine, if it helps: https://github.com/Azure/acs-engine/blob/master/parts/kubernetesmasteraddons-tiller-deployment.yaml

Quote from Jason: "The add-on manager enforces state from the manifests living on the master(s). Need to dig more to see what version of the manager we are using, may be able to use addonmanager.kubernetes.io/mode=EnsureExists rather than mode=Reconcile to allow users to change the Tiller version."

[squillace]

Didn't find this closed issue here: #55. Looks like we have the fix; I'll try it on my cluster. Can close this for that one if need be.

[bacongobbler]

closed via Azure/acs-engine#1319 for acs-engine. I assume this'll eventually make its way into ACS?