Add cloud-controller-manager support for Kubernetes cluster #1584
Conversation
karataliu
force-pushed
the
accm18n
branch
3 times, most recently
from
6d0d214 to
34651f8
Compare
| - "--cloud-provider=azure" | ||
| - "--cloud-config=/etc/kubernetes/azure.json" | ||
| - "--leader-elect=true" | ||
| # TODO: RBAC support |
There was a problem hiding this comment.
What's needed or outstanding for enabling RBAC?
There was a problem hiding this comment.
This is actually meant for enabling 'use-service-account-credentials', which does not work now, pending on kubernetes#53511
| sed -i "s|<kubernetesCcmImageSpec>|{{WrapAsVariable "kubernetesCcmImageSpec"}}|g; s|<masterFqdnPrefix>|{{WrapAsVariable "masterFqdnPrefix"}}|g; s|<allocateNodeCidrs>|{{WrapAsVariable "allocateNodeCidrs"}}|g; s|<kubeClusterCidr>|{{WrapAsVariable "kubeClusterCidr"}}|g; s|<kubernetesCtrlMgrRouteReconciliationPeriod>|{{WrapAsVariable "kubernetesCtrlMgrRouteReconciliationPeriod"}}|g" \ | ||
| /etc/kubernetes/manifests/cloud-controller-manager.yaml | ||
|
|
||
| sed -i "/--\(cloud-config\|cloud-provider\|route-reconciliation-period\)=/d" \ |
There was a problem hiding this comment.
Is this stripping off cloud-provider and cloud-config from apiserver? Doesn't that also need to be stripped off of controller-manager too?
There was a problem hiding this comment.
Yeah, both need to be updated. LIne323~324 is for kube-controller-manager, Line325~326 is for apiserver.
| CLOUD_PROVIDER=external | ||
| {{else}} | ||
| CLOUD_PROVIDER=azure | ||
| {{end}} | ||
|
|
||
| - path: "/etc/systemd/system/kubelet.service" |
There was a problem hiding this comment.
Is there no change needed for kubelet.service to include the --provider-id flag?
There was a problem hiding this comment.
|
What's the status of this @karataliu? What level of functionality does the cluster have when this is enabled? I'm guessing volumes don't work, but does networking + load balancing? I'm trying to understand how many of the items mentioned in the upstream issue need to happen before CCM + Azure is functional for most non-volume use-cases. Thanks. |
|
@colemickens Yes, volumes feature does not work. I've tried with e2e conformance tests and most cases would pass excepted for the volume ones. The upstream issue is more about migration work, Azure provider running with cloud-controller-manager could already work on a cluster. But since the whole feature is under beta stage (kubernetes/enhancements#88), it is not recommended for production use. |
|
Thanks for the info. Change seems good to me. I made a similar change in
Tectonic-Installer and everything seemed to work (at least,the sunset I
expected to work). (Granted the network setup in Tectonic-Installer is
Calico and not kubelet...)
…On Oct 17, 2017 10:39 PM, "Dong Liu" ***@***.***> wrote:
@colemickens <https://github.com/colemickens> Yes, volumes feature does
not work. I've tried with e2e conformance tests and most cases would pass
excepted for the volume ones.
The upstream issue is more about migration work, Azure provider running
with cloud-controller-manager could already work on a cluster. But since
the whole feature is under beta stage (kubernetes/enhancements#88
<kubernetes/enhancements#88>), it is not
recommended for production use.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1584 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAT9dG4RzGC0mOAm_QQbud8UANLZ20_Pks5stY8VgaJpZM4P2eEO>
.
|
|
Could someone please take a look? Thanks. |
|
(FWIW, this looks good to me. And in a few weeks hopefully, there should be alpha support for CCM under hyperkube as well, so there won't need to be a separate image/process... at least until CCM totally splits out of the main repo and is yanked back out of hyperkube, but that could be a while). |
|
@colemickens Thanks, I saw the ongoing work for hyperkube incorporating CCM. |
|
Thanks for your patience @karataliu, and apologies for the rebase needs. Not a few things have changed in the conflicting surface area, I'd be happy to spawn a new PR with this work if that's easier. |
|
@jackfrancis Updated, please take a look. Thanks. |
|
Thanks @karataliu ! Running E2E now. |
There was a problem hiding this comment.
This looks good. The only suggestion I'd make is to make the "enable/disable" property a *bool, so that we can more easily default to true at a later date. See this PR for an example:
https://github.com/Azure/acs-engine/pull/1763/files
If I had to do that PR again I'd make the return value here reference a boolean constant, and not a boolean literal, for easier future maintenance:
e.g.:
if <enableThing) == nil {
return DefaultThingEnabled
Let me know if that doesn't make sense! Thanks again!
|
In fact the default value should be dependent on k8s version, and as planned for 1.10+ release it should only have true value. Anyway it is good to switch to ternary here, I've updated accordingly. |
ghost
removed
the
What this PR does / why we need it:
Kubernetes v1.8 supports cloud-controller-manager which runs cloud provider related logic, and it is planned in v1.10 that using cloud-controller-manager becomes the only way to interact with clouds.
This PR adds an option 'UseCloudControllerManager' in apimodel for Kubernetes v1.8 version. If turned on, the cluster will be using cloud controller manager mode.
Currently it uses the common image 'cloud-controller-manager-amd64:v1.8.0' which includes all cloud provider implementation. Will later switch to azure dedicated cloud-controller-manager image once ready.
One TODO item is RBAC support, which pends on upstream Kubernetes#53511 .
Please find more details here: kubernetes/kubernetes#50752
Which issue this PR fixes
Special notes for your reviewer:
Release note: