How can I upgrade tiller on ACS?

[bacongobbler]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Every time I try to bump the tiller version up to v2.6.0 in ACS, ACS keeps preventing that by re-deploying tiller v2.5.1, which is what was provided when bootstrapping the cluster. I'd like to upgrade tiller on my cluster, but I cannot do that without switching over to acs-engine to deploy a cluster.

[JackQuincy]

This appears to be because of how we are defining our tiller addon. We have it marked as a cluster service. Which means Kuberentes doesn't want to allow updates to it through the kubernetes api. Instead it wants the updates to happen on the file in the master. We could change to using addonmanager.kubernetes.io/mode=EnsureExists . @anhowe @seanknox @jackfrancis thoughts?

relevant Kubernetes documentation
https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/README.md

[seanknox]

addonmanager.kubernetes.io/mode=EnsureExists appears to be right call here. Agree @lachie83?

[JackQuincy]

Also what do we think about doing this for dashboard, heapster, and the storage classes addons? Do we want them static or do we want customers to be able to modify them through the api server

[seanknox]

Good call for dashboard and heapster, though my hunch without thinking through much is that we should err on the more prescriptive end with storage classes.

[JackQuincy]

I think I agree on not letting people shoot themselves in the foot with storage classes. Mostly I was wanting to look at all the extensions and decide if we want to update them while doing this.

[JackQuincy]

@bacongobbler if it wasn't clear there is a work around by updating the add-on file for tiller, /etc/kubernetes/addons/kube-tiller-deployment.yaml, to be the newer spec for tiller on each master.

[lachie83]

Confirming everything @JackQuincy @seanknox have said.

[bacongobbler]

I was able to work around this by deploying a hacked up acs-engine cluster with Tiller v2.6.0 but I'll keep that in mind for next time. Thanks @JackQuincy @seanknox @lachie83 for your help!

[berndverst]

Documenting my single command workaround based on comments above (make sure to replace the host with your ACS master node)

ssh [email protected] sudo sed -i s/'2.5.1'/'2.6.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade

[bacongobbler]

@berndverst I tried your example and it worked, however I would suggest restarting the kube-addon-manager pod by deleting it from the kube-system namespace, e.g.

$ kubectl -n kube-system delete po kube-addon-manager-k8s-master-a637ed6-0

This will force a new addon-manager pod to boot and reconcile all the resources managed by the addon manager... Including tiller. This will force a fresh tiller v2.6.0 instance to be deployed.

[seanknox]

JFYI we have the long-term fix for this slated to go into acs-engine soon: Azure/acs-engine#1319

[stuartleeks]

What is the timeline for this being fixed in ACS?

[JackQuincy]

It will be in the v2 regions in a week or two if it isn't already. In the v1 regions it will be next month.

[subho007]

Any updates on this?

[JackQuincy]

Definetly in ACSv2. V1 is still a ways out we have had a deployment having other issues that is delaying this some.

[lrills]

Any update on this? Cluster in Southeast Asia still can't upgrade with helm init --upgade now.

[berndverst]

@lrills I manually had to update the add on definition on the Kubernetes master again as a workaround. Like so:

ssh [email protected] sudo sed -i s/'2.6.0'/'2.7.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade

No other method has reliably worked for me.

[bacongobbler]

I’m going to reopen this for visibility. I’m still making weekly references to this ticket in #helm-users. :)