Two alerts failing out of the box: K8SControllerManagerDown and K8SSchedulerDown

[dmcnaught]

Installed with KOPS 1.4.1, K8s 1.4.6 on AWS.

It looks to me like the query is set to alert when there is one kube-scheduler (or kube-contoller-manager), which I don't understand.

ALERT K8SSchedulerDown
  IF absent(up{job="kube-scheduler"}) or (count(up{job="kube-scheduler"} == 1) BY (cluster) == 0)

I’m pretty new to prometheus queries and I’m not really sure how the BY (cluster) == 0) relates.
Any pointers appreciated.
Thanks for the great project!
--Duncan

[brancz]

The BY (cluster) == 0 part is unlikely important in most setups it just allows this to be used to monitor/alert for more than one Kubernetes cluster (eg. when using Federation). I'm guessing your alert is rather triggering because of the absence of kube-scheduler/kube-controller-manager jobs. Could you make sure you can find it in the /targets page? If it is not there, what is likely happening is that there is no Endpoints object that lists the kube-scheduler and kube-controller-manager, which means you either don't have the Services created from manifests/k8s/. Or your kube-scheduler and kube-controller-manager are not discoverable via those, in which case the output of the following would help:

$ kubectl get pods --all-namespace

Or if you cannot disclose all of that information this should give us all the information applicable:

$ kubectl -n monitoring get pods
$ kubectl -n kube-system get pods

We typically test the content of this repository with clusters created with bootkube, but it would be great if we can get a section/guides for kops as it's pretty widely adopted as well.

[dmcnaught]

Thanks for the quick response.
I don't see them on the /targets page.
Here is the info requested:

NAMESPACE            NAME                                                     READY     STATUS             RESTARTS   AGE
athena-graphql       athena-graphql-cmd-3150689734-xqkkq                      1/1       Running            0          4d
deis                 deis-builder-2759337600-en6fr                            1/1       Running            1          17d
deis                 deis-controller-873470834-re8zj                          1/1       Running            0          17d
deis                 deis-database-1712966127-plp30                           1/1       Running            0          17d
deis                 deis-logger-9212198-xjpsj                                1/1       Running            3          17d
deis                 deis-logger-fluentd-0ppwl                                1/1       Running            0          17d
deis                 deis-logger-fluentd-a8zly                                1/1       Running            0          17d
deis                 deis-logger-redis-663064164-3wq80                        1/1       Running            0          17d
deis                 deis-monitor-grafana-432364990-joygg                     1/1       Running            0          17d
deis                 deis-monitor-influxdb-2729526471-7npow                   1/1       Running            0          17d
deis                 deis-monitor-telegraf-2poea                              0/1       CrashLoopBackOff   112        17d
deis                 deis-monitor-telegraf-cy218                              1/1       Running            0          17d
deis                 deis-nsqd-3264449345-c1t0c                               1/1       Running            0          17d
deis                 deis-registry-680832981-64b4s                            1/1       Running            0          17d
deis                 deis-registry-proxy-94w6s                                1/1       Running            0          17d
deis                 deis-registry-proxy-p46bn                                1/1       Running            0          17d
deis                 deis-router-2457652422-sx6c7                             1/1       Running            0          17d
deis                 deis-workflow-manager-2210821749-ggzm3                   1/1       Running            0          17d
hades-graphql        hades-graphql-cmd-1371319327-6sqb6                       1/1       Running            0          8m
kube-system          dns-controller-2613152787-l8bj4                          1/1       Running            0          17d
kube-system          etcd-server-events-ip-10-101-115-158.ec2.internal        1/1       Running            0          17d
kube-system          etcd-server-ip-10-101-115-158.ec2.internal               1/1       Running            0          17d
kube-system          kube-apiserver-ip-10-101-115-158.ec2.internal            1/1       Running            2          17d
kube-system          kube-controller-manager-ip-10-101-115-158.ec2.internal   1/1       Running            0          17d
kube-system          kube-dns-v20-3531996453-ban95                            3/3       Running            0          17d
kube-system          kube-dns-v20-3531996453-v66h9                            3/3       Running            0          17d
kube-system          kube-proxy-ip-10-101-115-158.ec2.internal                1/1       Running            0          17d
kube-system          kube-proxy-ip-10-101-175-18.ec2.internal                 1/1       Running            0          17d
kube-system          kube-scheduler-ip-10-101-115-158.ec2.internal            1/1       Running            0          17d
monitoring           alertmanager-main-0                                      2/2       Running            0          4d
monitoring           alertmanager-main-1                                      2/2       Running            0          4d
monitoring           alertmanager-main-2                                      2/2       Running            0          4d
monitoring           grafana-874468113-0atmz                                  2/2       Running            0          4d
monitoring           kube-state-metrics-3229993571-aqo7z                      1/1       Running            0          4d
monitoring           node-exporter-5xyxb                                      1/1       Running            0          4d
monitoring           node-exporter-xwgn6                                      1/1       Running            0          4d
monitoring           prometheus-k8s-0                                         3/3       Running            0          4d
monitoring           prometheus-operator-479044303-ris0n                      1/1       Running            0          4d
splunkspout          k8ssplunkspout-nonprod-2ykwk                             1/1       Running            0          17d
splunkspout          k8ssplunkspout-nonprod-xdp7d                             1/1       Running            0          17d
styleguide           styleguide-cmd-685725177-pwl1c                           1/1       Running            0          4d
styleguide-staging   styleguide-staging-cmd-2993321210-1cgmo                  1/1       Running            0          2h
wellbot              wellbot-web-3878855632-34s4e                             1/1       Running            0          15d
welltok-arch-k8s     welltok-arch-k8s-1857575956-ky06l                        1/1       Running            0          14d```

[brancz]

I believe I have seen this before, the problem I think is that kops doesn't label the kubernetes component pods correctly with k8s-app=<component-name>. To confirm that can you give me the output of kubectl -n kube-system get kube-controller-manager-ip-10-101-115-158.ec2.internal -oyaml and kubectl -n kube-system get kube-scheduler-ip-10-101-115-158.ec2.internal -oyaml (in case they got rescheduled the pods that now start with kube-scheduler or kube-controller-manager respectively)

If what I am guessing is correct then we should push on kops side to use upstream manifests like bootkube does.

[dmcnaught]

kube-controller-manager.txt
kube-scheduler.txt

[dmcnaught]

This was a similar issue - to add that label to kube-proxy: kubernetes/kops#617

[brancz]

I didn't see that one, thanks for pointing it out! I opened kubernetes/kops#1226 to start a discussion on it. Hopefully we will get those soon then. In the mean time I think you'll have to either ssh onto those servers, change the templates and restart them (which makes the objects be recreated from the templates IIRC; disclaimer: not super familiar with kops) or comment out/remove those alerts for now.

(also remember that changes to single machines will disappear when recreating machines from the ASG unless you make the changes to the ASG)

[dmcnaught]

I just noticed etcd is not appearing in my prometheus targets either.

[dmcnaught]

Oh, and kube-dns. Should we update kubernetes/kops#1226 ?

[brancz]

It seems like we won't have an answer before the holidays. So I'll keep pushing in the new year. But yes I will keep pushing for a consistent labelling strategy, we'll add the respective manifests for Prometheus to properly discover the components here once we have that consistent labelling. I don't mind maintaining a set of manifests for kops, bootkube, etc. as long as each of those labelling strategies make sense and exist.

So far so goot :) happy holidays!

[dmcnaught]

I added the labels on the master (/etc/kubernetes/manifests/kube-controller-manager,kube-scheduler), and then ran kubectl create -f manifests/k8s/self-hosted
I ran hack/cluster-monitoring/teardown and then hack/cluster-monitoring/deploy and that fixed the alerts for kube-scheduler and kube-controller-manager.
kube-dns now has four endpoints listed under /targets, and they are all getting error: getsockopt: connection refused
I'd also like to add etcd, but don't find any explicit instructions on that. It would be great to add a kops default setup in manifests/k8s/kops/

[brancz]

Yep that's the plan as soon as we have consistent labeling in upstream kops.

[dmcnaught]

Labelling added to kops: kubernetes/kops#1314

[brancz]

I don't have a v1.5.x kops cluster handy, but I'll create the manifests with a best effort and then it would be great if you could test them.

[dmcnaught]

With pleasure. Thanks

[brancz]

In fact I think the manifests from manifests/k8s/self-hosted are suitable for kops when using the latest master, am I mistaken? Except kube-dns as the manifest is slightly outdated from the upstream manifest, but there are no alerts for kube-dns metrics yet so that wouldn't be a problem for now. Can you confirm that?

[brancz]

Actually it seems that the kube-dns manifest for v1.5.0 has been updated so in that case it should appear as well.

[dmcnaught]

I'll create a 1.5.x K8s cluster with the latest KOPS soon to test, thanks.

Right now I updated the labels on my 1.4.6 master and it looks good except:

• kube-dns now has four failing endpoints listed under /targets:

kube-dns
Endpoint	State	Labels	Last Scrape	Error
http://100.96.1.2:10054/metrics
DOWN	instance="100.96.1.2:10054"	198ms ago	Get http://100.96.1.2:10054/metrics: dial tcp 100.96.1.2:10054: getsockopt: connection refused
http://100.96.1.2:10055/metrics
DOWN	instance="100.96.1.2:10055"	14.087s ago	Get http://100.96.1.2:10055/metrics: dial tcp 100.96.1.2:10055: getsockopt: connection refused
http://100.96.1.3:10054/metrics
DOWN	instance="100.96.1.3:10054"	9.708s ago	Get http://100.96.1.3:10054/metrics: dial tcp 100.96.1.3:10054: getsockopt: connection refused
http://100.96.1.3:10055/metrics
DOWN	instance="100.96.1.3:10055"	327ms ago	Get http://100.96.1.3:10055/metrics: dial tcp 100.96.1.3:10055: getsockopt: connection refused

• kubernetes target is also failing (and this is causing K8SApiserverDown alert):

kubernetes
Endpoint	State	Labels	Last Scrape	Error
https://10.101.115.158:443/metrics
DOWN	instance="10.101.115.158:443"	11.904s ago	Get https://10.101.115.158:443/metrics: x509: certificate is valid for 100.64.0.1, not 10.101.115.158

[brancz]

The kube-dns failure is likely due to an old kube-dns manifest .. I can see that the 1.4.x manifest has not been updated to expose metrics in the upstream kops repo.

The kubernetes target failure is a bit more tricky, best would be if the certificate were to be created with the requested IP in the additional names section. (inspect your cert with openssl x509 -text -in your.crt) The other option is to "manually" maintain an Endpoints object through a headless Service as done for etcd (see manifests/etcd), that way the correct IP will be used to perform the request.

[dmcnaught]

@brancz I don't know whether we want to continue this thread on kops/kube-prometheus work - let me know if there's a better place. Maybe we should open a new issue.
I used kops latest (master: Version git-8b21ace) and Kubernetes 1.5.1 to create a new cluster in AWS.
Running hack/cluster-monitoring/deploy

--- github/kube-prometheus ‹master› » ./hack/cluster-monitoring/deploy
namespace "monitoring" created
deployment "prometheus-operator" created
the server doesn't have a resource type "servicemonitor"
the server doesn't have a resource type "servicemonitor"
the server doesn't have a resource type "servicemonitor"
No resources found.
No resources found.
No resources found.
deployment "kube-state-metrics" created
service "kube-state-metrics" created
daemonset "node-exporter" created
service "node-exporter" created
configmap "grafana-dashboards" created
deployment "grafana" created
service "grafana" created
configmap "prometheus-k8s" created
configmap "prometheus-k8s-rules" created
service "prometheus-k8s" created
prometheus "prometheus-k8s" created
configmap "alertmanager-main" created
service "alertmanager-main" created
alertmanager "alertmanager-main" created

[brancz]

Has this been solved on upstream kops? @dmcnaught

[dmcnaught]

I'm going to start with the kops - kube-prometheus config when kops 1.5 has been released.

[brancz]

Great thanks for the update! Are you aware of an ETA?

[dmcnaught]

I've heard "soon" - it's currently in alpha4: https://github.com/kubernetes/kops/releases

[brancz]

Great! Looking forward to "soon" 🙂

[dmcnaught]

Me too. I thought it would be "sooner" 😉

[dmcnaught]

Getting close with kops 1.5.0-alpha2 and k8s 1.5.2 ^ Just the api cert issue to go. 😄

[rocketraman]

Looks like this is also the case with clusters created via acs-engine on Azure. The labels on the controller-manager pod are:

  labels:
    component: kube-controller-manager
    tier: control-plane

[yann-soubeyrand]

Same with a cluster created using Kubeadm.

[brancz]

@yann-soubeyrand for kubeadm clusters you need to enable the controller manager and scheduler to listen on all or at least the pod networking interface/ip.

[yann-soubeyrand]

@brancz Thanks for the tip on modifying listening addresses which saved me some time ;-) However, I was mentioning the fact that the labeling done by Kubeadm is like rocketraman wrote above and therefore kube-prometheus was not able to discover the controller manager neither the scheduler nor etcd.

[rawkode]

@brancz Can confirm what @yann-soubeyrand and @rocketraman have said, kubeadm and gke use component: kube-scheduler, not k8s-app

[KeithTt]

@yann-soubeyrand for kubeadm clusters you need to enable the controller manager and scheduler to listen on all or at least the pod networking interface/ip.

I changed the bind address of controller manager and scheduler to 0.0.0.0, but they are still not up on prometheus.

Also, there is no data in grafana..