kube-proxy config configmap missing - where is it ? needed for Prometheus operator

[sd2302]

Describe the bug

Per https://github.com/helm/charts/tree/master/stable/prometheus-operator

for Prometheus to scrape kubelet, we need to change kublet config as below

kubectl -n kube-system edit cm kube-proxy

But in aks-engine this cm for kube-proxy is missing

So where is the config for kubelet ?

we need change the following metricsBindAddress for prometheus to work correctly

apiVersion: v1
data:
config.conf: |-
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
# ...
# metricsBindAddress: 127.0.0.1:10249
metricsBindAddress: 0.0.0.0:10249
# ...
kubeconfig.conf: |-

Steps To Reproduce

Expected behavior

kubectl -n kube-system edit cm kube-proxy should work ?

AKS Engine version

Version: v0.45.0

Kubernetes version

15.4

Additional context

Deploying prometheus operator for K8 monitoring for aks-engine

[jackfrancis]

Hi @sd2302, the kube-proxy spec we deliver is defined here:

• https://github.com/Azure/aks-engine/blob/master/parts/k8s/addons/kubernetesmasteraddons-kube-proxy-daemonset.yaml

(for k8s < 1.16)

1.16-1.18 have version-specific specs:

• https://github.com/Azure/aks-engine/blob/master/parts/k8s/addons/1.16/kubernetesmasteraddons-kube-proxy-daemonset.yaml
• https://github.com/Azure/aks-engine/blob/master/parts/k8s/addons/1.17/kubernetesmasteraddons-kube-proxy-daemonset.yaml
• https://github.com/Azure/aks-engine/blob/master/parts/k8s/addons/1.18/kubernetesmasteraddons-kube-proxy-daemonset.yaml

Feel free to submit a PR w/ the ConfigMap spec that will allow you to customize it after deployment. Would an empty ConfigMap be an iterative improvement over the current scenario?

[jackfrancis]

Actually the >= 1.16 kube-proxy specs have a kube-proxy-config ConfigMap defined.

On a 1.16 cluster I just built w/ aks-engine:

$ kubectl get nodes
NAME                                 STATUS   ROLES    AGE     VERSION
k8s-agentpool1-31548157-vmss000000   Ready    agent    2m59s   v1.16.6
k8s-agentpool1-31548157-vmss000001   Ready    agent    99s     v1.16.6
k8s-master-31548157-0                Ready    master   2m59s   v1.16.6

$ k get configmap kube-proxy-config -n kube-system -o yaml
apiVersion: v1
data:
  config.yaml: |
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    clientConnection:
      kubeconfig: /var/lib/kubelet/kubeconfig
    clusterCIDR: "10.240.0.0/12"
    mode: "iptables"
    featureGates:
      {}
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"config.yaml":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\nclientConnection:\n  kubeconfig: /var/lib/kubelet/kubeconfig\nclusterCIDR: \"10.240.0.0/12\"\nmode: \"iptables\"\nfeatureGates:\n  {}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","component":"kube-proxy","k8s-app":"kube-proxy","kubernetes.io/cluster-service":"true","tier":"node"},"name":"kube-proxy-config","namespace":"kube-system"}}
  creationTimestamp: "2020-01-30T23:10:47Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    component: kube-proxy
    k8s-app: kube-proxy
    kubernetes.io/cluster-service: "true"
    tier: node
  name: kube-proxy-config
  namespace: kube-system
  resourceVersion: "389"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kube-proxy-config
  uid: 18fb07d4-6a46-49f4-a414-261c54f98b6d

Note that ConfigMap is named kube-proxy-config. The prometheus docs suggest that's normal.

Depending on the cluster, the relevant part config.conf will be in ConfigMap kube-system/kube-proxy or kube-system/kube-proxy-config

Ref:

https://github.com/helm/charts/tree/master/stable/prometheus-operator#kubeproxy

[sd2302]

@jackfrancis
Thanks.

I am on 1.15.4

Where would I put the metricsBindAddress: 0.0.0.0:10249 in the daemonset.apps/kube-proxy ?

[jackfrancis]

You'll have to add the configmap yourself on 1.15.4, as aks-engine didn't create one on cluster creation.

[sd2302]

Thanks, even in 1.16, where exactly would i put the metricsBindAddress: 0.0.0.0:1024 ?

I dont see kind: KubeProxyConfiguration in the 1.16 configmaps

[jackfrancis]

Looks like it wants it in the config.conf data key. There isn't one in the aks-engine-provided ConfigMap in >= 1.16, so you'll have to add it (probably via vi) when you do kubectl -n kube-system edit cm kube-proxy-config.

[sd2302]

Thanks.

In 1.15.4, i tried to add --metrics-bind-address=0.0.0.0:10249 in the kube-proxy spec as below but the deamonset is not taking it

spec:
containers:
- command:
- /hyperkube
- kube-proxy
- --kubeconfig=/var/lib/kubelet/kubeconfig
- --cluster-cidr=10.240.0.0/12
- --feature-gates=ExperimentalCriticalPodAnnotation=true
- --proxy-mode=iptables
- --metrics-bind-address=0.0.0.0:10249

https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/

[jackfrancis]

Right, the above is adding an additional arg --metrics-bind-address to the kube-proxy runtime; I assume that the kube-proxy runtime doesn't expect that argument.

[sd2302]

Per this https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/

it support it but unsure why the aks-engine deamon-set not taking it ?

--metrics-bind-address 0.0.0.0     Default: 127.0.0.1:10249

  | The IP address for the metrics server to serve on (set to 0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces)

[jackfrancis]

Those are the 1.17 docs, perhaps it was introduced after 1.15?

FWIW, there's nothing special about the kube-proxy that aks-engine delivers, it's the vanilla, upstream package.

[sd2302]

i am going to deploy aks 1.17 tomorrow and test this out. i really need to deploy prometheus-operator helm chart and prometheus is not discovering the targets b/c of the --metrics-bind-address issue

[jackfrancis]

Thanks for sticking it out, let us know how that goes!

[sd2302]

I re-deployed using 16.4 and tried to update the kube-proxy-config configmap and it immediately removes it

Is that a aks-engine issue ?

i really need to have kube-proxy-config persist the value of metricsBindAddress: 0.0.0.0:10249

sysdev@imabvlsys-496:~/aks-engine/aksengine-testbluev2/_output/tdcmastertestbluev2-5e38750b$ kubectl describe cm/kube-proxy-config -n kube-system
Name: kube-proxy-config
Namespace: kube-system
Labels: addonmanager.kubernetes.io/mode=Reconcile
component=kube-proxy
k8s-app=kube-proxy
kubernetes.io/cluster-service=true
tier=node
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","data":{"config.yaml":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\nclientConnection:\n ...

Data

config.yaml:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clientConnection:
kubeconfig: /var/lib/kubelet/kubeconfig
clusterCIDR: "10.240.0.0/12"
metricsBindAddress: 0.0.0.0:10249
mode: "iptables"
featureGates:
{}

Events:
sysdev@imabvlsys-496:~/aks-engine/aksengine-testbluev2/_output/tdcmastertestbluev2-5e38750b$ kubectl describe cm/kube-proxy-config -n kube-system
Name: kube-proxy-config
Namespace: kube-system
Labels: addonmanager.kubernetes.io/mode=Reconcile
component=kube-proxy
k8s-app=kube-proxy
kubernetes.io/cluster-service=true
tier=node
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","data":{"config.yaml":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\nclientConnection:\n ...

Data

config.yaml:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clientConnection:
kubeconfig: /var/lib/kubelet/kubeconfig
clusterCIDR: "10.240.0.0/12"
mode: "iptables"
featureGates:
{}

Events:

[jackfrancis]

Ah, yes. That's because:

addonmanager.kubernetes.io/mode: Reconcile

in the ConfigMap spec.

in the kube-proxy spec. You could do this right now. On each master vm:

$ sudo vi /etc/kubernetes/addons/kube-proxy-daemonset.yaml

Change the above to this in the ConfigMap spec:

addonmanager.kubernetes.io/mode: EnsureExists

Then, kick the kube-addon-manager pod to pick up the new settings:

$ k delete pod kube-addon-manager-k8s-master-19456504-0 -n kube-system

(kube-addon-manager-k8s-master-19456504-0 was the running kube-addon-manager pod on my single master cluster, you'll want to find the pod(s) particular to your cluster)

You should then be able to modify the ConfigMap.

I've filed this issue to make this less difficult going forward:

#2672

[sd2302]

@jackfrancis

Thanks. that worked.

With 16.4 K8 version, we are getting the metrics from Prom/Grafana helm operators and stack but yet the kubelet shows down in Prom configs. I need to open a bug with the Prom operator team. and i will update here how that goes. Right now even after setting metricsBindAddress: 0.0.0.0:10249, it is prom is scraping to internal-node-ip:10249

[jackfrancis]

Thanks for keeping us up-to-date. Bon courage

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dbrennand]

Hey @jackfrancis

I just ran into this issue with my k8s cluster deployed on Azure Stack Hub.

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"d2ea02d7bcf37724adabbdb9d37ede0cf2c0d627", GitTreeState:"clean", BuildDate:"2020-04-24T22:59:19Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

I am deploying the bitnami/prometheus-operator and realised kube-proxy metrics were not being collected.
When I attempt to alter the configmap by adding the metricsBindAddress it also keeps being altered back.

The workaround provided isn't ideal and I see a enchancement was raised #2672 but never acted upon. Is there any way to make this process easier going forward?

Many Thanks.

[jackfrancis]

Hi @dbrennand, I've re-opened #2672 so we can track enabling the kube-addon-manager mode.

Just FYI, you'll want to ensure that your kubectl client is running the same version as the cluster you're managing (I see you're managing a 1.16.9 cluster w/ the 1.18.5 client). You may not run into any issues, but any API breaking changes between minor versions will affect you here.