Kube-Proxy endpoint connection refused #16476
Comments
|
kubectl logs kube-proxy-cv6rf -n kube-system 0626 12:56:07.323006 1 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: the server was unable to return a response in the time allotted, but may still be processing the request (get endpoints) |
|
@ctyjrsy have you set the kube-proxy argument for metric-bind-address? |
|
Fixed it with the hint from @truncj $ kubectl edit cm/kube-proxy -n kube-system
## Change from
metricsBindAddress: 127.0.0.1:10249 ### <--- Too secure
## Change to
metricsBindAddress: 0.0.0.0:10249
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system |
|
I do not see any existing metricsBindAddress in my kube-proxy config kubectl edit cm/kube-proxy -n kube-system apiVersion: v1 If i introduce the bind address manually in the CM, and after the kube-system pods are restarted, the metrics start getting collected from kube-proxy. Thank you @truncj and @teochenglim |
|
For the instance I was working on the config map was kube-proxy-config |
|
I don't even have |
|
@demisx if you can give me your settings with this 2 lines. I would able to help you kubectl get pod -n kube-system # this shall output your kube-proxy pod name..
kubectl get pod/kube-proxy-jpgsr -n kube-system -o yaml # pod name kube-proxy-jpgsr from last commandexample of mine.. i know why it is in config map.. $ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
.
.
kube-proxy-jpgsr 1/1 Running 0 16d # this is what i choose
kube-proxy-rprpg 1/1 Running 0 16d # you can use this too..
.
.
$ kubectl get pod/kube-proxy-jpgsr -n kube-system -o yaml
containers:
- command:
- kube-proxy
- --v=2
- --config=/var/lib/kube-proxy-config/config # oh... so startup config is here
.
.
- mountPath: /var/lib/kube-proxy-config/ # it is kinda of a config mounting? But where?
name: config
.
.
volumes:
.
.
- configMap:
defaultMode: 420
name: kube-proxy-config # so it is in configmap.. |
|
@teochenglim Thank you so much for your response. Here is the output that you've requested: $ kubectl get pod -n kube-system
-------
kube-proxy-ip-172-20-121-121.us-west-2.compute.internal 1/1 Running 0 6d13h
kube-proxy-ip-172-20-41-132.us-west-2.compute.internal 1/1 Running 0 6d13h
kube-proxy-ip-172-20-44-4.us-west-2.compute.internal 1/1 Running 0 6d13h
kube-proxy-ip-172-20-66-190.us-west-2.compute.internal 1/1 Running 0 6d13h
kube-proxy-ip-172-20-78-191.us-west-2.compute.internal 1/1 Running 0 6d13h
kube-proxy-ip-172-20-98-153.us-west-2.compute.internal 1/1 Running 0 6d13h$ kubectl -n kube-system get pod/kube-proxy-ip-172-20-121-121.us-west-2.compute.internal -o yaml
------
spec:
containers:
- command:
- /bin/sh
- -c
- mkfifo /tmp/pipe; (tee -a /var/log/kube-proxy.log < /tmp/pipe & ) ; exec /usr/local/bin/kube-proxy
--cluster-cidr=100.96.0.0/11 --conntrack-max-per-core=131072 --hostname-override=ip-172-20-121-121.us-west-2.compute.internal
--kubeconfig=/var/lib/kube-proxy/kubeconfig --master=https://api.internal.prod.dimaslist.org
--oom-score-adj=-998 --resource-container="" --v=2 > /tmp/pipe 2>&1
image: k8s.gcr.io/kube-proxy:v1.14.6
imagePullPolicy: IfNotPresent
name: kube-proxy
resources:
requests:
cpu: 100m
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kube-proxy/kubeconfig
name: kubeconfig
readOnly: true
- mountPath: /var/log/kube-proxy.log
name: logfile
- mountPath: /lib/modules
name: modules
readOnly: true
- mountPath: /etc/ssl/certs
name: ssl-certs-hosts
readOnly: true
- mountPath: /run/xtables.lock
name: iptableslock
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostNetwork: true
nodeName: ip-172-20-121-121.us-west-2.compute.internal
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
operator: Exists
volumes:
- hostPath:
path: /var/lib/kube-proxy/kubeconfig
type: ""
name: kubeconfig
- hostPath:
path: /var/log/kube-proxy.log
type: ""
name: logfile
- hostPath:
path: /lib/modules
type: ""
name: modules
- hostPath:
path: /usr/share/ca-certificates
type: ""
name: ssl-certs-hosts
- hostPath:
path: /run/xtables.lock
type: FileOrCreate
name: iptableslock
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2019-11-02T13:41:26Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2019-11-02T13:41:32Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2019-11-02T13:41:32Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2019-11-02T13:41:26Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://076e5703ed1e273cb27251f1b217ebed329184551360af47e940fbeb1b4daa4e
image: k8s.gcr.io/kube-proxy:v1.14.6
imageID: docker-pullable://k8s.gcr.io/kube-proxy@sha256:e5c364dc75d816132bebf2d84b35518f0661fdeae39c686d92f9e5f9a07e96b9
lastState: {}
name: kube-proxy
ready: true
restartCount: 0
state:
running:
startedAt: "2019-11-02T13:41:32Z"
hostIP: 172.20.121.121
phase: Running
podIP: 172.20.121.121
qosClass: Burstable
startTime: "2019-11-02T13:41:26Z" |
|
Hi @demisx , Sure, I did a quick investigate and the kube-proxy metrics default listening to 127.0.0.1:10249 !!!!! To update it will/may cause down time, please I shall bare no responsibility if this is your production system.!!!! Yes. From the result you gave, it is reside on the hostPath, however if you ssh to the kops ckuster "nodes".. it is not the information like eks cluster. volumes:
- hostPath:
path: /var/lib/kube-proxy/kubeconfig
type: ""To patch it, i use $ export NAME=[Your cluster name]
$ kops edit cluster $NAMEinsert this (How I know? kubernetes/kops#6472) and then run "kops update cluster $NAME --yes --state=s3://[your s3 store]" once Then this part is I don't know how to do it better. Basically i tried both "cluster update" and "cluster rolling-update".. both doesn't trigger node update. Since I am on AWS, I go to EC2-Auto-Scaling-Groups and down the node by set minimum and desired node to be 0. Wait EC2 node to be shutting down, trigger the node back to your desire pod count again. !!!!! If you want zero down time, also must depends on your application is able to survive from kubernetes node taint, please node taint it after grown. https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ You can do the reverse that grow the autoscaling group, taint it and then reduce it. Best to schedule down time to do this. Both doesn't work for me kops update cluster $NAME --yes --state=s3://[your s3 store]
kops rolling-update clusterAfter scale down and up. I ssh to the kops nodes again Cheers. |
|
@teochenglim Thank you for the detail explanation. This is pretty much what I did as well. I've also edited the cluster yaml file to add the lines below, so next time I create a cluster from scratch I won't have to edit it manually: spec:
kubeProxy:
metricsBindAddress: 0.0.0.0 |
|
Good that I know it works for you. cheers and have a nice day. |
Describe the bug
Kube-Proxy endpoints are not being scraped by Prometheus, as a result many default metrics are unavailable. This is kubernets cluster in Digital Ocean.
Version of Helm and Kubernetes:
helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Which chart:
stable/prometheus-operator
What happened:
What you expected to happen:
The same chart
How to reproduce it (as minimally and precisely as possible):
helm install --name monitoring -f prom-operator.yaml stable/prometheus-operator --set kubelet.serviceMonitor.https=true --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false --set kube-proxy.servicemonitor.https=true
Anything else we need to know:
The text was updated successfully, but these errors were encountered: