Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Internal Load Balancer not automatically adding target network IP configurations #2151

Closed
michaelburch opened this issue Jan 25, 2018 · 24 comments
Closed

Internal Load Balancer not automatically adding target network IP configurations #2151

michaelburch opened this issue Jan 25, 2018 · 24 comments
Assignees
@jackfrancis

Comments

@michaelburch
Copy link

Is this a request for help?:
Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine?:
0.12.1

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.9.1

What happened:
Internal Load Balancer created and associated with availability set but no target network ip configurations
were added. Manually adding them in the portal is a functional workaround.

What you expected to happen:
Minimally, the target ip configuration of the load balancer endpoints should be added.

How to reproduce it (as minimally and precisely as possible):

  1. Deploy acs kubernetes cluster with single master, single agentpool.
    Use "availabilityProfile": "AvailabilitySet"

  2. Create Kubernetes service of type LoadBalancer, annotate with "service.beta.kubernetes.io/azure-load-balancer-internal=true"

  3. Observe no target network ip configurations were added to the backend pool. Test to confirm that traffic does not flow as expected.

Anything else we need to know:
Deployed to custom vnet
@rbankole
Copy link

i can confirm the same, the ELBS are not adding target network ip configurations. Also using custom vnet via acs-engine v.12

@amit-kumar-4
Copy link
Contributor

amit-kumar-4 commented Jan 30, 2018 •
edited
Loading

I've run into this issue as well and I can see kube-controller manager spitting out logs for any service created and looks like it could be related to this issue.

I0124 10:42:29.921411       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"aaeedbfb-fd36-11e7-9146-000d3ab7f4cc", APIVersion:"v1", ResourceVersion:"434", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0129 15:07:32.792452       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"4167b3f7-0503-11e8-a123-000d3ab2d356", APIVersion:"v1", ResourceVersion:"36", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found

Not sure yet why this it's happening in the first place though.

@amit-kumar-4
Copy link
Contributor

amit-kumar-4 commented Jan 30, 2018 •
edited
Loading

Alright, I get a segfault and restart of kube-controller manager when I try to deploy an ILB. Logs below:

I0130 13:58:44.790852       1 azure_loadbalancer.go:435] reconcileLoadBalancer(kube-system/nginx-ingress-nginx-ingress-controller) - wantLb(false): started
I0130 13:58:44.791111       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"nginx-ingress-nginx-ingress-controller", UID:"af970566-05c5-11e8-828a-000d3ab2d356", APIVersion:"v1", ResourceVersion:"131951", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0130 13:58:44.860271       1 azure_backoff.go:207] LoadBalancerClient.List(xxxxx-k8s191-clstr-rg) - backoff: success
I0130 13:58:44.860301       1 azure_loadbalancer.go:441] reconcileLoadBalancer(kube-system/nginx-ingress-nginx-ingress-controller): lb(k8s191-master) wantLb(false) resolved load balancer name
I0130 13:58:44.860358       1 azure_loadbalancer.go:784] ensure(kube-system/nginx-ingress-nginx-ingress-controller): lb(k8s191-master) finished
I0130 13:58:44.897926       1 azure_backoff.go:262] PublicIPAddressesClient.List(xxxxx-k8s191-clstr-rg) - backoff: success
I0130 13:58:44.897985       1 azure_loadbalancer.go:435] reconcileLoadBalancer(kube-system/nginx-ingress-nginx-ingress-controller) - wantLb(true): started
I0130 13:58:44.937159       1 azure_backoff.go:207] LoadBalancerClient.List(xxxxx-k8s191-clstr-rg) - backoff: success
I0130 13:58:44.937224       1 azure_loadbalancer.go:242] selectLoadBalancer: cluster(k8s191-master) service(kube-system/nginx-ingress-nginx-ingress-controller) isInternal(true) - availabilitysetsnames [agent-availabilitySet-37925753]
I0130 13:58:44.937269       1 azure_loadbalancer.go:441] reconcileLoadBalancer(kube-system/nginx-ingress-nginx-ingress-controller): lb(k8s191-master-internal) wantLb(true) resolved load balancer name
I0130 13:58:45.352504       1 azure_backoff.go:408] backoff: success, HTTP response=201
I0130 13:58:45.421068       1 azure_backoff.go:89] backoff: success
I0130 13:58:45.439062       1 azure_backoff.go:89] backoff: success
I0130 13:58:45.439979       1 azure_backoff.go:89] backoff: success
I0130 13:58:45.454895       1 azure_backoff.go:89] backoff: success
I0130 13:58:45.511866       1 azure_backoff.go:89] backoff: success
I0130 13:58:45.590804       1 azure_backoff.go:89] backoff: success
I0130 13:58:45.590849       1 azure_loadbalancer.go:784] ensure(kube-system/nginx-ingress-nginx-ingress-controller): lb(k8s191-master-internal) finished
E0130 13:58:45.591132       1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:367
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:326
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:113
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:306
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:249
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:771
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:217
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d]

goroutine 1461 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111
panic(0x40b2e00, 0xa603590)
	/usr/local/go/src/runtime/panic.go:491 +0x283
k8s.io/kubernetes/pkg/cloudprovider/providers/azure.(*Cloud).getServiceLoadBalancerStatus(0xc4209a7500, 0xc4221ba780, 0xc424106eb0, 0xc4221ba780, 0xc4240dfa40, 0x6)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:326 +0x30d
k8s.io/kubernetes/pkg/cloudprovider/providers/azure.(*Cloud).EnsureLoadBalancer(0xc4209a7500, 0x7ffd7cb966d5, 0xd, 0xc4221ba780, 0xc4240dfa40, 0x6, 0x8, 0xbe943c652f2276bc, 0x1d10a1ad11, 0xa9f0d40)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:113 +0x2ea
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).ensureLoadBalancer(0xc4213be0e0, 0xc4221ba780, 0xc4221ba780, 0x4961da9, 0x6)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374 +0xcc
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).createLoadBalancerIfNeeded(0xc4213be0e0, 0xc42478fcc0, 0x32, 0xc4221ba780, 0xc424705c40, 0xc424705c78, 0x1265492)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:306 +0x20e
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processServiceUpdate(0xc4213be0e0, 0xc424085d60, 0xc4221ba780, 0xc42478fcc0, 0x32, 0x0, 0x0, 0x0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:249 +0xeb
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).syncService(0xc4213be0e0, 0xc42478fcc0, 0x32, 0x0, 0x0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:771 +0x3aa
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).worker.func1(0xc4213be0e0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213 +0xd9
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).worker(0xc4213be0e0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:217 +0x2b
k8s.io/kubernetes/pkg/controller/service.(*ServiceController).(k8s.io/kubernetes/pkg/controller/service.worker)-fm()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195 +0x2a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc422ba4170)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc422ba4170, 0x3b9aca00, 0x0, 0x1, 0xc420067860)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc422ba4170, 0x3b9aca00, 0xc420067860)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by k8s.io/kubernetes/pkg/controller/service.(*ServiceController).Run
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195 +0x20c

This issue only happens when creating ILB; ELBs doesn't give any such errors and are provisioned properly (as far as I can see).

@feiskyer
Copy link
Member

@aku105 confirmed the bug in azure cloud provider. Will fix and backport to v1.9.

@gmagniez
Copy link

Hello,
I have the same issue but I have seen that Azure LB "Basic" is limited to 1 availability set in the backend pool. (https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#load-balancer)
I though that's why my nodes where not added to the LB pool.
As I have at least 2 nodes pool configured by acs-engine, each one with it's own availability set, will it work even with the bug fixed; or how can I tell Kubernetes to only publish nodes IPs from 1 pool/as ?

@feiskyer
Copy link
Member

will it work even with the bug fixed

Yep, it worked. acs-engine will set primaryAvailabilitySetName in /etc/kubernetes/azure.json, which determines which nodes to add.

@feiskyer feiskyer mentioned this issue Jan 31, 2018
Closed
@jackfrancis
Copy link
Member

@feiskyer Do you have any ideas if your fix is coming in v1.9.3? Thanks for that btw!

@feiskyer
Copy link
Member

feiskyer commented Feb 7, 2018 •
edited
Loading

@jackfrancis Cherry pick to v1.9 branch has been merged. It will be included in v1.9.3

@diwakar-s-maurya
Copy link
Contributor

diwakar-s-maurya commented Feb 10, 2018 •
edited
Loading

With kubernetes v1.9.3 also, I am still facing this problem.

At the time of writing, the latest acs-engine release version: v0.12.5 does not support creating kubernetes 1.9.3 cluster. I built the acs-engine from the master branch commit fe57150.

./acs-engine version

Version: canary
GitCommit: fe57150
GitTreeState: clean

When a kubernetes service was created with this service.beta.kubernetes.io/azure-load-balancer-internal: "true" annotation, a load balancer with private ip was created but no target network ip configurations were added to the backend pool.

@feiskyer
Copy link
Member

@diwakar-s-maurya Could you check logs of kube-controller-manger and check whether there is something wrong happened? e.g. by

PODNAME=$(kubectl -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system logs $PODNAME --tail 500

@diwakar-s-maurya
Copy link
Contributor

@feiskyer These are the log. Creating internal load balancer type service does not automatically add target network IP configurations.

Logs after almost 5 mins of fresh cluster v1.9.3 creation by acs-engine fe57150

I0211 05:50:10.395429       1 feature_gate.go:190] feature gates: map[ServiceNodeExclusion:true]
I0211 05:50:10.395516       1 controllermanager.go:108] Version: v1.9.3
I0211 05:50:10.403199       1 leaderelection.go:174] attempting to acquire leader lease...
E0211 05:50:10.403609       1 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: Get https://10.240.0.4:443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 10.240.0.4:443: getsockopt: connection refused
I0211 05:50:23.324910       1 leaderelection.go:184] successfully acquired lease kube-system/kube-controller-manager
I0211 05:50:23.333035       1 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"7387e061-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"17", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s-master-40117112-0 became leader
E0211 05:50:23.347425       1 controllermanager.go:386] Server isn't healthy yet.  Waiting a little while.
E0211 05:50:24.361405       1 controllermanager.go:386] Server isn't healthy yet.  Waiting a little while.
I0211 05:50:25.547239       1 azure.go:249] azure: using client_id+client_secret to retrieve access token
I0211 05:50:27.488072       1 controllermanager.go:483] Starting "replicationcontroller"
I0211 05:50:27.488131       1 controller_utils.go:1019] Waiting for caches to sync for tokens controller
I0211 05:50:27.588702       1 controller_utils.go:1026] Caches are synced for tokens controller
I0211 05:50:27.785005       1 controllermanager.go:493] Started "replicationcontroller"
I0211 05:50:27.785032       1 controllermanager.go:483] Starting "service"
I0211 05:50:27.785639       1 replica_set.go:183] Starting replicationcontroller controller
I0211 05:50:27.785782       1 controller_utils.go:1019] Waiting for caches to sync for ReplicationController controller
I0211 05:50:27.830834       1 controllermanager.go:493] Started "service"
I0211 05:50:27.830856       1 controllermanager.go:483] Starting "ttl"
I0211 05:50:27.830967       1 service_controller.go:187] Starting service controller
I0211 05:50:27.830983       1 controller_utils.go:1019] Waiting for caches to sync for service controller
I0211 05:50:27.872271       1 controllermanager.go:493] Started "ttl"
I0211 05:50:27.872339       1 controllermanager.go:483] Starting "route"
I0211 05:50:27.872362       1 core.go:132] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0211 05:50:27.872441       1 controllermanager.go:490] Skipping "route"
I0211 05:50:27.872477       1 controllermanager.go:483] Starting "pvc-protection"
W0211 05:50:27.872492       1 controllermanager.go:490] Skipping "pvc-protection"
I0211 05:50:27.872516       1 controllermanager.go:483] Starting "resourcequota"
I0211 05:50:27.872414       1 ttl_controller.go:116] Starting TTL controller
I0211 05:50:27.872644       1 controller_utils.go:1019] Waiting for caches to sync for TTL controller
I0211 05:50:27.983935       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions ingresses}
I0211 05:50:27.983986       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions replicasets}
I0211 05:50:27.984032       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { endpoints}
I0211 05:50:27.984131       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps controllerrevisions}
I0211 05:50:27.984174       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps daemonsets}
I0211 05:50:27.984240       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {networking.k8s.io networkpolicies}
W0211 05:50:27.984261       1 shared_informer.go:310] resyncPeriod 67853085526268 is smaller than resyncCheckPeriod 71908994299038 and the informer has already started. Changing it to 71908994299038
I0211 05:50:27.984341       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { serviceaccounts}
I0211 05:50:27.984449       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { limitranges}
I0211 05:50:27.984529       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps statefulsets}
I0211 05:50:27.984602       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {events.k8s.io events}
I0211 05:50:27.984641       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {batch cronjobs}
I0211 05:50:27.984785       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions daemonsets}
I0211 05:50:27.984840       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for { podtemplates}
I0211 05:50:27.984945       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps deployments}
I0211 05:50:27.985018       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {batch jobs}
I0211 05:50:27.985052       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {rbac.authorization.k8s.io roles}
I0211 05:50:27.985122       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {rbac.authorization.k8s.io rolebindings}
I0211 05:50:27.985152       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {apps replicasets}
I0211 05:50:27.985232       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {autoscaling horizontalpodautoscalers}
I0211 05:50:27.985268       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {policy poddisruptionbudgets}
W0211 05:50:27.985324       1 shared_informer.go:310] resyncPeriod 66575513736377 is smaller than resyncCheckPeriod 71908994299038 and the informer has already started. Changing it to 71908994299038
I0211 05:50:27.985438       1 resource_quota_monitor.go:216] QuotaMonitor created object count evaluator for {extensions deployments}
I0211 05:50:27.985531       1 controllermanager.go:493] Started "resourcequota"
I0211 05:50:27.985551       1 controllermanager.go:483] Starting "daemonset"
I0211 05:50:27.985846       1 resource_quota_controller.go:272] Starting resource quota controller
I0211 05:50:27.985862       1 controller_utils.go:1019] Waiting for caches to sync for resource quota controller
I0211 05:50:27.986172       1 resource_quota_monitor.go:289] QuotaMonitor running
I0211 05:50:28.092787       1 controllermanager.go:493] Started "daemonset"
I0211 05:50:28.092853       1 controllermanager.go:483] Starting "job"
I0211 05:50:28.092997       1 daemon_controller.go:232] Starting daemon sets controller
I0211 05:50:28.093029       1 controller_utils.go:1019] Waiting for caches to sync for daemon sets controller
I0211 05:50:28.135334       1 controllermanager.go:493] Started "job"
I0211 05:50:28.135475       1 controllermanager.go:483] Starting "csrcleaner"
I0211 05:50:28.135657       1 job_controller.go:138] Starting job controller
I0211 05:50:28.135693       1 controller_utils.go:1019] Waiting for caches to sync for job controller
I0211 05:50:28.180672       1 controllermanager.go:493] Started "csrcleaner"
I0211 05:50:28.180726       1 controllermanager.go:483] Starting "attachdetach"
I0211 05:50:28.181010       1 cleaner.go:81] Starting CSR cleaner controller
W0211 05:50:28.233620       1 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
I0211 05:50:28.234340       1 plugins.go:453] Loaded volume plugin "kubernetes.io/aws-ebs"
I0211 05:50:28.234421       1 plugins.go:453] Loaded volume plugin "kubernetes.io/gce-pd"
I0211 05:50:28.234444       1 plugins.go:453] Loaded volume plugin "kubernetes.io/cinder"
I0211 05:50:28.234460       1 plugins.go:453] Loaded volume plugin "kubernetes.io/portworx-volume"
I0211 05:50:28.234489       1 plugins.go:453] Loaded volume plugin "kubernetes.io/vsphere-volume"
I0211 05:50:28.234506       1 plugins.go:453] Loaded volume plugin "kubernetes.io/azure-disk"
I0211 05:50:28.234521       1 plugins.go:453] Loaded volume plugin "kubernetes.io/photon-pd"
I0211 05:50:28.234537       1 plugins.go:453] Loaded volume plugin "kubernetes.io/scaleio"
I0211 05:50:28.234564       1 plugins.go:453] Loaded volume plugin "kubernetes.io/storageos"
I0211 05:50:28.234583       1 plugins.go:453] Loaded volume plugin "kubernetes.io/fc"
I0211 05:50:28.234599       1 plugins.go:453] Loaded volume plugin "kubernetes.io/iscsi"
I0211 05:50:28.234613       1 plugins.go:453] Loaded volume plugin "kubernetes.io/rbd"
I0211 05:50:28.234700       1 controllermanager.go:493] Started "attachdetach"
I0211 05:50:28.234731       1 controllermanager.go:483] Starting "clusterrole-aggregation"
I0211 05:50:28.234835       1 attach_detach_controller.go:258] Starting attach detach controller
I0211 05:50:28.234982       1 controller_utils.go:1019] Waiting for caches to sync for attach detach controller
I0211 05:50:28.353434       1 controllermanager.go:493] Started "clusterrole-aggregation"
I0211 05:50:28.353528       1 controllermanager.go:483] Starting "podgc"
I0211 05:50:28.353611       1 clusterroleaggregation_controller.go:148] Starting ClusterRoleAggregator
I0211 05:50:28.353820       1 controller_utils.go:1019] Waiting for caches to sync for ClusterRoleAggregator controller
I0211 05:50:28.601581       1 controllermanager.go:493] Started "podgc"
I0211 05:50:28.601633       1 controllermanager.go:483] Starting "garbagecollector"
I0211 05:50:28.601664       1 gc_controller.go:76] Starting GC controller
I0211 05:50:28.601798       1 controller_utils.go:1019] Waiting for caches to sync for GC controller
I0211 05:50:29.025783       1 resource_quota_controller.go:434] syncing resource quota controller with updated resources from discovery: map[{events.k8s.io v1beta1 events}:{} {batch v1 jobs}:{} { v1 configmaps}:{} {extensions v1beta1 deployments}:{} { v1 serviceaccounts}:{} {apps v1 replicasets}:{} {apps v1 daemonsets}:{} {autoscaling v1 horizontalpodautoscalers}:{} {networking.k8s.io v1 networkpolicies}:{} {policy v1beta1 poddisruptionbudgets}:{} {rbac.authorization.k8s.io v1 rolebindings}:{} { v1 endpoints}:{} { v1 secrets}:{} {extensions v1beta1 replicasets}:{} {extensions v1beta1 networkpolicies}:{} {extensions v1beta1 ingresses}:{} {apps v1 statefulsets}:{} {rbac.authorization.k8s.io v1 roles}:{} { v1 limitranges}:{} { v1 resourcequotas}:{} {extensions v1beta1 daemonsets}:{} { v1 events}:{} { v1 replicationcontrollers}:{} { v1 podtemplates}:{} { v1 pods}:{} {apps v1 controllerrevisions}:{} {apps v1 deployments}:{} {batch v1beta1 cronjobs}:{} { v1 services}:{} { v1 persistentvolumeclaims}:{}]
I0211 05:50:29.993316       1 controllermanager.go:493] Started "garbagecollector"
W0211 05:50:29.993411       1 controllermanager.go:477] "bootstrapsigner" is disabled
W0211 05:50:29.993427       1 controllermanager.go:477] "tokencleaner" is disabled
I0211 05:50:29.993441       1 controllermanager.go:483] Starting "persistentvolume-binder"
I0211 05:50:29.993758       1 garbagecollector.go:135] Starting garbage collector controller
I0211 05:50:29.993820       1 controller_utils.go:1019] Waiting for caches to sync for garbage collector controller
I0211 05:50:29.993857       1 graph_builder.go:323] GraphBuilder running
I0211 05:50:30.060662       1 plugins.go:453] Loaded volume plugin "kubernetes.io/host-path"
I0211 05:50:30.060739       1 plugins.go:453] Loaded volume plugin "kubernetes.io/nfs"
I0211 05:50:30.060765       1 plugins.go:453] Loaded volume plugin "kubernetes.io/glusterfs"
I0211 05:50:30.060786       1 plugins.go:453] Loaded volume plugin "kubernetes.io/rbd"
I0211 05:50:30.060829       1 plugins.go:453] Loaded volume plugin "kubernetes.io/quobyte"
I0211 05:50:30.060894       1 plugins.go:453] Loaded volume plugin "kubernetes.io/azure-file"
I0211 05:50:30.060911       1 plugins.go:453] Loaded volume plugin "kubernetes.io/flocker"
I0211 05:50:30.060927       1 plugins.go:453] Loaded volume plugin "kubernetes.io/portworx-volume"
I0211 05:50:30.060944       1 plugins.go:453] Loaded volume plugin "kubernetes.io/scaleio"
I0211 05:50:30.060994       1 plugins.go:453] Loaded volume plugin "kubernetes.io/local-volume"
I0211 05:50:30.061024       1 plugins.go:453] Loaded volume plugin "kubernetes.io/storageos"
I0211 05:50:30.061058       1 plugins.go:453] Loaded volume plugin "kubernetes.io/aws-ebs"
I0211 05:50:30.061075       1 plugins.go:453] Loaded volume plugin "kubernetes.io/gce-pd"
I0211 05:50:30.061092       1 plugins.go:453] Loaded volume plugin "kubernetes.io/cinder"
I0211 05:50:30.061127       1 plugins.go:453] Loaded volume plugin "kubernetes.io/vsphere-volume"
I0211 05:50:30.061154       1 plugins.go:453] Loaded volume plugin "kubernetes.io/azure-disk"
I0211 05:50:30.061170       1 plugins.go:453] Loaded volume plugin "kubernetes.io/photon-pd"
I0211 05:50:30.061219       1 controllermanager.go:493] Started "persistentvolume-binder"
I0211 05:50:30.061267       1 pv_controller_base.go:264] Starting persistent volume controller
I0211 05:50:30.061293       1 controller_utils.go:1019] Waiting for caches to sync for persistent volume controller
I0211 05:50:30.061242       1 controllermanager.go:483] Starting "persistentvolume-expander"
W0211 05:50:30.061338       1 controllermanager.go:490] Skipping "persistentvolume-expander"
I0211 05:50:30.061351       1 controllermanager.go:483] Starting "namespace"
I0211 05:50:30.127502       1 controllermanager.go:493] Started "namespace"
I0211 05:50:30.127584       1 controllermanager.go:483] Starting "cronjob"
I0211 05:50:30.127649       1 namespace_controller.go:186] Starting namespace controller
I0211 05:50:30.127688       1 controller_utils.go:1019] Waiting for caches to sync for namespace controller
I0211 05:50:30.193908       1 controllermanager.go:493] Started "cronjob"
I0211 05:50:30.193955       1 controllermanager.go:483] Starting "csrapproving"
I0211 05:50:30.194026       1 cronjob_controller.go:103] Starting CronJob Manager
I0211 05:50:30.209340       1 controllermanager.go:493] Started "csrapproving"
I0211 05:50:30.209360       1 controllermanager.go:483] Starting "node"
I0211 05:50:30.209501       1 certificate_controller.go:113] Starting certificate controller
I0211 05:50:30.209538       1 controller_utils.go:1019] Waiting for caches to sync for certificate controller
I0211 05:50:30.261683       1 node_controller.go:246] Sending events to api server.
I0211 05:50:30.261842       1 taint_controller.go:158] Sending events to api server.
I0211 05:50:30.261999       1 controllermanager.go:493] Started "node"
I0211 05:50:30.262041       1 controllermanager.go:483] Starting "replicaset"
I0211 05:50:30.262241       1 node_controller.go:558] Starting node controller
I0211 05:50:30.262281       1 controller_utils.go:1019] Waiting for caches to sync for node controller
I0211 05:50:30.401807       1 controllermanager.go:493] Started "replicaset"
I0211 05:50:30.401915       1 controllermanager.go:483] Starting "horizontalpodautoscaling"
I0211 05:50:30.401983       1 replica_set.go:183] Starting replicaset controller
I0211 05:50:30.402016       1 controller_utils.go:1019] Waiting for caches to sync for ReplicaSet controller
I0211 05:50:31.115152       1 controllermanager.go:493] Started "horizontalpodautoscaling"
I0211 05:50:31.115190       1 controllermanager.go:483] Starting "disruption"
I0211 05:50:31.115282       1 horizontal.go:128] Starting HPA controller
I0211 05:50:31.115296       1 controller_utils.go:1019] Waiting for caches to sync for HPA controller
I0211 05:50:31.290236       1 garbagecollector.go:182] syncing garbage collector with updated resources from discovery: map[{extensions v1beta1 daemonsets}:{} {apps v1 replicasets}:{} {networking.k8s.io v1 networkpolicies}:{} { v1 configmaps}:{} { v1 persistentvolumes}:{} { v1 events}:{} {apps v1 daemonsets}:{} {apps v1 statefulsets}:{} { v1 endpoints}:{} {events.k8s.io v1beta1 events}:{} {batch v1beta1 cronjobs}:{} { v1 namespaces}:{} {extensions v1beta1 deployments}:{} {certificates.k8s.io v1beta1 certificatesigningrequests}:{} {rbac.authorization.k8s.io v1 rolebindings}:{} {storage.k8s.io v1 storageclasses}:{} {admissionregistration.k8s.io v1beta1 validatingwebhookconfigurations}:{} { v1 persistentvolumeclaims}:{} { v1 serviceaccounts}:{} {batch v1 jobs}:{} {rbac.authorization.k8s.io v1 clusterrolebindings}:{} {admissionregistration.k8s.io v1beta1 mutatingwebhookconfigurations}:{} {apiextensions.k8s.io v1beta1 customresourcedefinitions}:{} { v1 secrets}:{} { v1 pods}:{} { v1 podtemplates}:{} {extensions v1beta1 networkpolicies}:{} {apps v1 controllerrevisions}:{} {autoscaling v1 horizontalpodautoscalers}:{} {policy v1beta1 poddisruptionbudgets}:{} {rbac.authorization.k8s.io v1 roles}:{} { v1 limitranges}:{} {extensions v1beta1 podsecuritypolicies}:{} {extensions v1beta1 ingresses}:{} {apps v1 deployments}:{} {rbac.authorization.k8s.io v1 clusterroles}:{} { v1 resourcequotas}:{} {apiregistration.k8s.io v1beta1 apiservices}:{} {extensions v1beta1 replicasets}:{} { v1 services}:{} { v1 nodes}:{} { v1 replicationcontrollers}:{}]
I0211 05:50:31.367111       1 controllermanager.go:493] Started "disruption"
I0211 05:50:31.367133       1 controllermanager.go:483] Starting "statefulset"
I0211 05:50:31.367214       1 disruption.go:288] Starting disruption controller
I0211 05:50:31.370501       1 controller_utils.go:1019] Waiting for caches to sync for disruption controller
I0211 05:50:31.602543       1 controllermanager.go:493] Started "statefulset"
I0211 05:50:31.602566       1 controllermanager.go:483] Starting "csrsigning"
I0211 05:50:31.602604       1 stateful_set.go:150] Starting stateful set controller
I0211 05:50:31.602617       1 controller_utils.go:1019] Waiting for caches to sync for stateful set controller
I0211 05:50:31.755235       1 controllermanager.go:493] Started "csrsigning"
I0211 05:50:31.755286       1 controllermanager.go:483] Starting "endpoint"
I0211 05:50:31.755318       1 certificate_controller.go:113] Starting certificate controller
I0211 05:50:31.755357       1 controller_utils.go:1019] Waiting for caches to sync for certificate controller
I0211 05:50:32.002965       1 controllermanager.go:493] Started "endpoint"
I0211 05:50:32.003006       1 controllermanager.go:483] Starting "serviceaccount"
I0211 05:50:32.003041       1 endpoints_controller.go:153] Starting endpoint controller
I0211 05:50:32.003054       1 controller_utils.go:1019] Waiting for caches to sync for endpoint controller
I0211 05:50:32.251459       1 controllermanager.go:493] Started "serviceaccount"
I0211 05:50:32.251483       1 controllermanager.go:483] Starting "deployment"
I0211 05:50:32.251516       1 serviceaccounts_controller.go:115] Starting service account controller
I0211 05:50:32.251524       1 controller_utils.go:1019] Waiting for caches to sync for service account controller
I0211 05:50:32.501238       1 controllermanager.go:493] Started "deployment"
I0211 05:50:32.501790       1 deployment_controller.go:153] Starting deployment controller
I0211 05:50:32.501811       1 controller_utils.go:1019] Waiting for caches to sync for deployment controller
I0211 05:50:32.506632       1 controller_utils.go:1019] Waiting for caches to sync for resource quota controller
I0211 05:50:32.554465       1 controller_utils.go:1026] Caches are synced for ClusterRoleAggregator controller
I0211 05:50:32.554469       1 controller_utils.go:1026] Caches are synced for service account controller
I0211 05:50:32.558334       1 controller_utils.go:1026] Caches are synced for certificate controller
E0211 05:50:32.565143       1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="k8s-master-40117112-0"  does not exist
E0211 05:50:32.565201       1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="k8s-worker-40117112-0"  does not exist
I0211 05:50:32.582421       1 controller_utils.go:1026] Caches are synced for TTL controller
I0211 05:50:32.587448       1 controller_utils.go:1026] Caches are synced for ReplicationController controller
I0211 05:50:32.599266       1 controller_utils.go:1019] Waiting for caches to sync for garbage collector controller
I0211 05:50:32.601133       1 ttl_controller.go:271] Changed ttl annotation for node k8s-master-40117112-0 to 0 seconds
I0211 05:50:32.603411       1 controller_utils.go:1026] Caches are synced for endpoint controller
I0211 05:50:32.603479       1 controller_utils.go:1026] Caches are synced for deployment controller
I0211 05:50:32.603494       1 controller_utils.go:1026] Caches are synced for GC controller
I0211 05:50:32.603647       1 controller_utils.go:1026] Caches are synced for ReplicaSet controller
I0211 05:50:32.607359       1 ttl_controller.go:271] Changed ttl annotation for node k8s-worker-40117112-0 to 0 seconds
I0211 05:50:32.609699       1 controller_utils.go:1026] Caches are synced for certificate controller
I0211 05:50:32.629952       1 controller_utils.go:1026] Caches are synced for namespace controller
I0211 05:50:32.631223       1 controller_utils.go:1026] Caches are synced for HPA controller
I0211 05:50:32.643167       1 controller_utils.go:1026] Caches are synced for job controller
I0211 05:50:32.643223       1 controller_utils.go:1026] Caches are synced for service controller
I0211 05:50:32.643299       1 service_controller.go:654] Detected change in list of current cluster nodes. New node set: [k8s-master-40117112-0 k8s-worker-40117112-0]
I0211 05:50:32.643330       1 service_controller.go:662] Successfully updated 0 out of 0 load balancers to direct traffic to the updated set of nodes
I0211 05:50:32.671630       1 controller_utils.go:1026] Caches are synced for disruption controller
I0211 05:50:32.671723       1 disruption.go:296] Sending events to api server.
E0211 05:50:32.677658       1 clusterroleaggregation_controller.go:180] edit failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "edit": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:32.702997       1 controller_utils.go:1026] Caches are synced for stateful set controller
E0211 05:50:32.712337       1 clusterroleaggregation_controller.go:180] view failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "view": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:32.735146       1 controller_utils.go:1026] Caches are synced for attach detach controller
I0211 05:50:32.775163       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:32.775200       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:32.775491       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:32.775516       1 controller_utils.go:1026] Caches are synced for persistent volume controller
I0211 05:50:32.862427       1 controller_utils.go:1026] Caches are synced for node controller
I0211 05:50:32.862555       1 node_controller.go:633] Controller observed a new Node: "k8s-master-40117112-0"
I0211 05:50:32.862639       1 controller_utils.go:197] Recording Registered Node k8s-master-40117112-0 in Controller event message for node k8s-master-40117112-0
I0211 05:50:32.862717       1 node_controller.go:611] Initializing eviction metric for zone: southcentralus:�:0
I0211 05:50:32.862780       1 node_controller.go:633] Controller observed a new Node: "k8s-worker-40117112-0"
I0211 05:50:32.862805       1 controller_utils.go:197] Recording Registered Node k8s-worker-40117112-0 in Controller event message for node k8s-worker-40117112-0
W0211 05:50:32.862922       1 node_controller.go:964] Missing timestamp for Node k8s-master-40117112-0. Assuming now as a timestamp.
W0211 05:50:32.862964       1 node_controller.go:964] Missing timestamp for Node k8s-worker-40117112-0. Assuming now as a timestamp.
I0211 05:50:32.863026       1 node_controller.go:880] Controller detected that zone southcentralus:�:0 is now in state Normal.
I0211 05:50:32.863512       1 taint_controller.go:181] Starting NoExecuteTaintManager
I0211 05:50:32.863597       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"k8s-worker-40117112-0", UID:"73740a28-0eef-11e8-851a-000d3a752906", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node k8s-worker-40117112-0 event: Registered Node k8s-worker-40117112-0 in Controller
I0211 05:50:32.863648       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"k8s-master-40117112-0", UID:"7362eeec-0eef-11e8-851a-000d3a752906", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node k8s-master-40117112-0 event: Registered Node k8s-master-40117112-0 in Controller
I0211 05:50:32.885970       1 controller_utils.go:1026] Caches are synced for resource quota controller
I0211 05:50:32.893225       1 controller_utils.go:1026] Caches are synced for daemon sets controller
I0211 05:50:32.894095       1 controller_utils.go:1026] Caches are synced for garbage collector controller
I0211 05:50:32.894106       1 garbagecollector.go:144] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0211 05:50:32.900004       1 controller_utils.go:1026] Caches are synced for garbage collector controller
I0211 05:50:32.900015       1 garbagecollector.go:219] synced garbage collector
I0211 05:50:32.906762       1 controller_utils.go:1026] Caches are synced for resource quota controller
I0211 05:50:33.680144       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.681823       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"heapster", UID:"79a8c0b0-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"302", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set heapster-56b857ddcc to 1
I0211 05:50:33.718086       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
I0211 05:50:33.720047       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"303", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.720658       1 deployment_controller.go:485] Error syncing deployment kube-system/heapster: Operation cannot be fulfilled on deployments.extensions "heapster": the object has been modified; please apply your changes to the latest version and try again
E0211 05:50:33.723027       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.726213       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.743869       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.744003       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.744112       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.744558       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.754578       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.754628       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.754635       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.754667       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.760623       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.761024       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.760947       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.774945       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.784691       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.784928       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.784879       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.866815       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:33.883712       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:33.883860       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:33.883946       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.029379       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e7672b-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"323", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set tiller-deploy-589f6788d7 to 1
I0211 05:50:34.029878       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/tiller-deploy-589f6788d7, need 1, creating 1
I0211 05:50:34.039388       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:34.042697       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:34.042766       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:34.043505       1 deployment_controller.go:485] Error syncing deployment kube-system/tiller-deploy: Operation cannot be fulfilled on deployments.extensions "tiller-deploy": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:34.048950       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:34.053782       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"tiller-deploy-589f6788d7", UID:"79eb6d39-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"324", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: tiller-deploy-589f6788d7-x49tv
I0211 05:50:34.064608       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
I0211 05:50:34.065732       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
E0211 05:50:34.066504       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.390543       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:34.396889       1 replica_set.go:515] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet kube-system/heapster-56b857ddcc
E0211 05:50:34.396964       1 replica_set.go:451] Sync "kube-system/heapster-56b857ddcc" failed with pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.397157       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "heapster-56b857ddcc-" is forbidden: error looking up service account kube-system/heapster: serviceaccount "heapster" not found
I0211 05:50:34.535077       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:34.535237       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:34.535401       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:34.562823       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/kube-dns-v20-597689868c, need 2, creating 2
I0211 05:50:34.563868       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"kube-dns-v20", UID:"7a3a1ae2-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"349", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set kube-dns-v20-597689868c to 2
I0211 05:50:34.582518       1 deployment_controller.go:485] Error syncing deployment kube-system/kube-dns-v20: Operation cannot be fulfilled on deployments.extensions "kube-dns-v20": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:34.588342       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kube-dns-v20-597689868c", UID:"7a3c3d4f-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"351", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-dns-v20-597689868c-r4qsc
I0211 05:50:34.614439       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kube-dns-v20-597689868c", UID:"7a3c3d4f-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"351", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-dns-v20-597689868c-sjmxm
I0211 05:50:34.916450       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:34.916519       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:34.916875       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:34.991423       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"7a7a0ad9-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"375", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-85z7d
I0211 05:50:34.997729       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"7a7a0ad9-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"375", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-tpkc5
I0211 05:50:35.037121       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 1, creating 1
I0211 05:50:35.042374       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"306", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: heapster-56b857ddcc-66zf9
I0211 05:50:35.775600       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:35.775707       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:35.775770       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:37.024208       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/kubernetes-dashboard-5dfcbcfb5c, need 1, creating 1
I0211 05:50:37.031076       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af736c4-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"405", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set kubernetes-dashboard-5dfcbcfb5c to 1
I0211 05:50:37.051202       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kubernetes-dashboard-5dfcbcfb5c", UID:"7bb46428-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"408", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kubernetes-dashboard-5dfcbcfb5c-fftnc
I0211 05:50:37.069942       1 deployment_controller.go:485] Error syncing deployment kube-system/kubernetes-dashboard: Operation cannot be fulfilled on deployments.extensions "kubernetes-dashboard": the object has been modified; please apply your changes to the latest version and try again
I0211 05:50:37.827710       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:37.828134       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 10s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:37.829436       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:39.092016       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:39.092074       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 10s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:39.092486       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:39.594505       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:39.594560       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 10s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:39.595051       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:39.971516       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:39.971580       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 10s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:39.972019       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:40.816779       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:40.816836       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 10s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:40.817408       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:47.867094       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:47.867265       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 20s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:47.867371       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:50:49.170888       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:49.171522       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 20s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:49.171715       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:50:49.636306       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:49.636379       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 20s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:49.636844       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:50:50.010733       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:50.010798       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 20s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:50.011233       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:50:50.862988       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:50:50.863058       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 20s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:50:50.863196       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:08.023961       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:08.024127       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 40s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:08.024278       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:09.213926       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:09.214180       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 40s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:09.214246       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:09.677321       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:09.677373       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 40s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:09.677409       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:10.048547       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:10.048604       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 40s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:51:10.048684       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:51:10.903084       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:10.903130       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 40s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:10.903164       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:48.062707       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:48.062835       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 1m20s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:48.063719       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:51:49.259820       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:49.260726       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 1m20s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:49.260986       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:51:49.719469       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:49.719640       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 1m20s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:49.720424       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:51:50.092085       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 05:51:50.092796       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
E0211 05:51:50.092245       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 1m20s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:51:50.939508       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:51:50.939569       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 1m20s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:50.939604       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:51:53.488947       1 replica_set.go:478] Too few replicas for ReplicaSet kube-system/heapster-77776c69b, need 1, creating 1
I0211 05:51:53.490226       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"heapster", UID:"79a8c0b0-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"609", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set heapster-77776c69b to 1
I0211 05:51:53.510165       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-77776c69b", UID:"a9484491-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"611", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: heapster-77776c69b-2b5h8
I0211 05:51:53.524115       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"heapster", UID:"79a8c0b0-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"609", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set heapster-56b857ddcc to 0
I0211 05:51:53.526860       1 replica_set.go:526] Too many replicas for ReplicaSet kube-system/heapster-56b857ddcc, need 0, deleting 1
I0211 05:51:53.527648       1 controller_utils.go:590] Controller heapster-56b857ddcc deleting pod kube-system/heapster-56b857ddcc-66zf9
I0211 05:51:53.532783       1 deployment_controller.go:485] Error syncing deployment kube-system/heapster: Operation cannot be fulfilled on deployments.extensions "heapster": the object has been modified; please apply your changes to the latest version and try again
I0211 05:51:53.708372       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"heapster-56b857ddcc", UID:"79b67d1a-0eef-11e8-851a-000d3a752906", APIVersion:"extensions", ResourceVersion:"617", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: heapster-56b857ddcc-66zf9
I0211 05:53:08.108215       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:08.108437       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 2m40s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:53:08.108519       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:53:09.301454       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:09.301537       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 2m40s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:53:09.301715       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:53:09.792969       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:09.793033       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 2m40s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:53:09.793100       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:53:10.135479       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:10.135672       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 2m40s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:53:10.135780       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:53:11.016962       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:53:11.017009       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 2m40s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:53:11.017139       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:55:48.543202       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:48.543251       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:55:48.543425       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 05:55:49.396169       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:49.396447       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:55:49.396531       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 05:55:49.829409       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:49.829555       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:55:49.829709       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 05:55:50.180718       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:50.180811       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:55:50.181459       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 05:55:51.061053       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 05:55:51.061111       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 05:55:51.061205       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:00:48.857764       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:48.857835       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:00:48.857925       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:00:49.436299       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:49.436360       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:00:49.436562       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:00:49.884395       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:49.884452       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:00:49.884521       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:00:50.218974       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:50.219028       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:00:50.219098       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:00:51.110232       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:00:51.110498       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:00:51.110644       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:05:49.026334       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:49.026393       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:05:49.026606       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:05:49.534528       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:49.534617       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:05:49.534724       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:05:49.919998       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:49.920045       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:05:49.920236       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:05:50.257611       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:50.257673       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:05:50.257781       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:05:51.165267       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:05:51.165311       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:05:51.165582       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found

After creating a service of type internal load balancer, these new log lines got printed

I0211 06:07:44.089297       1 service_controller.go:301] Ensuring LB for service default/ingress-nginx
I0211 06:07:44.089637       1 azure_loadbalancer.go:435] reconcileLoadBalancer(default/ingress-nginx) - wantLb(false): started
I0211 06:07:44.090674       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"ingress-nginx", UID:"dfe22760-0ef1-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"1861", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0211 06:07:44.228196       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 06:07:44.228224       1 azure_loadbalancer.go:441] reconcileLoadBalancer(default/ingress-nginx): lb(kube-5a7fd7a8) wantLb(false) resolved load balancer name
I0211 06:07:44.228241       1 azure_loadbalancer.go:797] ensure(default/ingress-nginx): lb(kube-5a7fd7a8) finished
I0211 06:07:44.285503       1 azure_backoff.go:248] PublicIPAddressesClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 06:07:44.285556       1 azure_loadbalancer.go:435] reconcileLoadBalancer(default/ingress-nginx) - wantLb(true): started
I0211 06:07:44.322257       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
I0211 06:07:44.322287       1 azure_loadbalancer.go:242] selectLoadBalancer: cluster(kube-5a7fd7a8) service(default/ingress-nginx) isInternal(true) - availabilitysetsnames [worker-availabilitySet-40117112]
I0211 06:07:44.322351       1 azure_loadbalancer.go:441] reconcileLoadBalancer(default/ingress-nginx): lb(kube-5a7fd7a8-internal) wantLb(true) resolved load balancer name
I0211 06:07:44.949151       1 azure_backoff.go:394] backoff: success, HTTP response=201
I0211 06:07:45.329576       1 azure_backoff.go:54] backoff: success
I0211 06:07:45.521068       1 azure_backoff.go:54] backoff: success
I0211 06:07:45.521115       1 azure_loadbalancer.go:797] ensure(default/ingress-nginx): lb(kube-5a7fd7a8-internal) finished
I0211 06:07:45.651638       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"ingress-nginx", UID:"dfe22760-0ef1-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"1861", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0211 06:10:49.216519       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:49.216651       1 service_controller.go:776] Failed to process service default/kubernetes. Retrying in 5m0s: error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:10:49.216757       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"kubernetes", UID:"73841c31-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"12", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service default/kubernetes: Service(default/kubernetes) - Loadbalancer not found
I0211 06:10:49.585439       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:49.585508       1 service_controller.go:776] Failed to process service kube-system/tiller-deploy. Retrying in 5m0s: error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:10:49.585588       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"tiller-deploy", UID:"79e06974-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"321", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/tiller-deploy: Service(kube-system/tiller-deploy) - Loadbalancer not found
I0211 06:10:49.962871       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:49.963071       1 service_controller.go:776] Failed to process service kube-system/kube-dns. Retrying in 5m0s: error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:10:49.963140       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kube-dns", UID:"7a31f461-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"347", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found
I0211 06:10:50.307581       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:50.313925       1 service_controller.go:776] Failed to process service kube-system/heapster. Retrying in 5m0s: error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:10:50.314360       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"heapster", UID:"7a5b4ff8-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"372", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/heapster: Service(kube-system/heapster) - Loadbalancer not found
I0211 06:10:51.215423       1 azure_backoff.go:193] LoadBalancerClient.List(diwakar-test-kube-cluster) - backoff: success
E0211 06:10:51.215571       1 service_controller.go:776] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5m0s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found
I0211 06:10:51.215694       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"7af06ea9-0eef-11e8-851a-000d3a752906", APIVersion:"v1", ResourceVersion:"402", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found

kuberernetes server and cli version

kubectl version                                                                                    [11:41]
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

acs-engine version

./acs-engine version
Version: canary
GitCommit: fe57150
GitTreeState: clean

Internal load balancer service which i tested

cat << EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app: ingress-nginx
EOF

acs-engine template used to create kubernetes cluster

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.9",
      "orchestratorVersion": "1.9.3"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "kube",
      "vmSize": "Standard_B2s"
    },
    "agentPoolProfiles": [
      {
        "name": "worker",
        "count": 1,
        "vmSize": "Standard_B2s",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
      "secret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
  }
}

No target network IP configurations present for the internal load balancer even after 15 mins
image

If i create a public load balancer service, it's target network IP configurations are set automatically immediately
image

@kaluzaaa
Copy link

It seems that in 1.9.3 it's fixed - kubernetes/kubernetes#59083, https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#changelog-since-v192

@diwakar-s-maurya
Copy link
Contributor

diwakar-s-maurya commented Feb 11, 2018 •
edited
Loading

@kaluzaaa I also thought so, but it is still not working.
Can someone please confirm that they also hitting this problem too by creating the cluster v1.9.3 using acs-engine? Or point out at what step I might be doing wrong? I have provided my acs-engine template.json and service.yaml above.

@feiskyer feiskyer mentioned this issue Feb 12, 2018
Closed
@feiskyer
Copy link
Member

feiskyer commented Feb 12, 2018 •
edited
Loading

@diwakar-s-maurya Thanks of reporting the problem. There is another bug for internal load balancers. Will file a PR to fix it.

Update: PR created at kubernetes/kubernetes#59747

@feiskyer feiskyer mentioned this issue Feb 12, 2018
Merged
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Feb 12, 2018
Merge pull request #59747 from feiskyer/fix-59746
4cc993a 
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Map correct vmset name for internal load balancers

**What this PR does / why we need it**:

When creating an internal loadbalancer, e.g.

```sh
cat << EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app: ingress-nginx
EOF
```

Then wait a while, and no target backends present for the internal load balancer even after 15 mins.

![](https://user-images.githubusercontent.com/10303889/36070528-54aa9848-0f22-11e8-834b-7401a2fa7611.png)

Refer Azure/acs-engine#2151 (comment).

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #59746

**Special notes for your reviewer**:

Should cherry pick to v1.9, v1.8, and v1.7 (and requires resolving conflicts manually).

**Release note**:

```release-note
Map correct vmset name for internal load balancers
```
@imranq2
Copy link

imranq2 commented Feb 26, 2018 •
edited
Loading

I ended up writing a powershell function to fix the internal load balancer. Here it is in case someone is interested:
fixloadbalancer.txt

@mblaschke
Copy link

Can confirm the bug for v1.9.3
Public LB is getting updated, internal LB is not updated.

@feiskyer
Copy link
Member

@mblaschke The cherry-pick PR kubernetes/kubernetes#59754 has been merged. It should be included in v1.9.4.

@feiskyer feiskyer mentioned this issue Mar 7, 2018
Closed
@jackfrancis jackfrancis mentioned this issue Mar 7, 2018
Merged
@ghost ghost assigned jackfrancis Mar 7, 2018
@ghost ghost added the in progress label Mar 7, 2018
@youkoulayley youkoulayley mentioned this issue Mar 9, 2018
Closed
@ghost ghost removed the in progress label Mar 13, 2018
@JerryJoyce
Copy link

I just provisioned v 1.9.4 and I have this issue

~/r/j/cluster-manifests   master *…  kubectl describe svc kube-dns -n kube-system 1078ms  Thu Mar 15 12:09:17 2018
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=KubeDNS
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kube-dns","kubernetes.io/cluster-service":"true","kubernetes.io/n...
Selector: k8s-app=kube-dns
Type: ClusterIP
IP: 10.0.0.10
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 10.240.0.103:53,10.240.0.241:53
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 10.240.0.103:53,10.240.0.241:53
Session Affinity: None
Events:
Type Reason Age From Message

Warning CreatingLoadBalancerFailed 4m (x212 over 17h) service-controller Error creating load balancer (will retry): error getting LB for service kube-system/kube-dns: Service(kube-system/kube-dns) - Loadbalancer not found

@feiskyer
Copy link
Member

@JerryJoyce Since this is a ClusterIP service, the warning message could be omit safely. We will remove this warning in future releases.

@jackfrancis
Copy link
Member

@feiskyer is there an upstream issue/PR to address this behavior?

@jackfrancis
Copy link
Member

@feiskyer We are seeing additional symptoms that suggest that non-cloudprovider services are being processed via cloudprovider and being sent to Azure API. We're seeing Azure cloudprovider backoff entries in the logs which are generated from responses from Azure API.

The reason this would be bad is that add'l, unnecessary requests to Azure API count against per-subscription network quota. A large cluster that generated a significant number of these unnecessary request could send an entire subscription (and thus all of its clusters) into a network backoff state (no network requests allowed by Azure API).

Have you seen this? @lachie83

@feiskyer
Copy link
Member

PR kubernetes/kubernetes#61260 is for removing this warning for 1.9 release.

@feiskyer We are seeing additional symptoms that suggest that non-cloudprovider services are being processed via cloudprovider and being sent to Azure API. We're seeing Azure cloudprovider backoff entries in the logs which are generated from responses from Azure API.

The check is required because a service may be changed from LoadBalancer to ClusterIP. Cloud provider should ensure ALB cleaned up in this case. In v1.10, caches for LB/NSG/RouteTable/VMSS/VM have been added, which could help to reduce such API calls.

@jackfrancis
Copy link
Member

@feiskyer I don't think your PR addresses these issues:

kubernetes/kubernetes#61100
kubernetes/kubernetes#60528

@feiskyer
Copy link
Member

@jackfrancis This is working as expected. See @brendandburns's comments here.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jackfrancis jackfrancis

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests