Kube-proxy support for Openshift

[ChrsMark]

Describe the enhancement:
Right now proxy metricseat does not work out of the box on Openshift installations. It would be nice to investigate how this can happen.

Extra info

Logs collected while trying to deploy on Openshift:

2020-04-06T11:35:37.554Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.proxy: error getting processed metrics: error making http request: Get http://localhost:10249/metrics: dial tcp 127.0.0.1:10249: connect: connection refused
Notes:

kube-proxy pod runs on kube-proxy namespace and not kube-system. Not that it would be the issue since we are trying to access host's network.

What is actually running in this pod:

kubectl -n kube-proxy exec -it kube-proxy-hj5zb /bin/bash           
[root@minishift origin]# curl http://0.0.0.0:10249/metrics
curl: (7) Failed connect to 0.0.0.0:10249; Connection refused
[root@minishift origin]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.6 475736 28220 ?        Ssl  10:04   0:10 openshift start network --enable=proxy --listen=https://0.0.0.0:8444 --config=/etc/origin/node/node-config.yaml
root      5784  0.4  0.1  20244  7036 ?        Ss   11:55   0:00 /bin/bash
root      5831  0.0  0.0  55184  1844 ?        R+   11:55   0:00 ps aux
Info:

Container image: openshift/origin-control-plane:v3.11.0

Info: https://github.com/openshift/sdn

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[exekias]

This one is still valid, please don't close it 🙂

[Protopopys]

@exekias, i have some problem with my OpenShift cluster.
I have read the documentation and it looks like Namespace: openshift-sdn configmaps: sdn-config --> metricsBindAddress: 0.0.0.0:29101 is the solution to the problem. I am currently testing an elastic-agent configuration with the following settings:

          - data_stream:
              dataset: kubernetes.proxy
              type: metrics
            metricsets:
              - proxy
            hosts:
              # Kubernetes
              # - 'localhost:10249'
              # Openshift
              - 'localhost:29101'
            period: 10s

[ChrsMark]

Thanks for your feedback @Protopopys! So did you need to change kube-proxy's config if I understand correctly? Also could you provide more details about your Openshift installation, version bare-metal/cloud etc?

Fyi we are running some verification for Agent these days. Find more info at elastic/integrations#2065. cc: @tetianakravchenko

[Protopopys]

Hi @ChrsMark, for local testing I use "Code Ready Containers"

CodeReady Containers version: 1.38.0+659b2cbd
OpenShift version: 4.9.12 (embedded in executable)

We use a bare metal installation Openshift 4.9 in development and production.

You don't need to change the kube-proxy configMap, metricsBindAddress: 0.0.0.0:29101 it's default value.
In the crc, you can find the configuration map at https://console-openshift-console.apps-crc.testing/k8s/ns/openshift-sdn/configmaps/sdn-config

kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:29101

My Elastic-Agent configMap - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75

1 - kubernetes-cluster-metrics - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L191

• Changed from 'kube-state-metrics:8080' to 'https://kube-state-metrics.openshift-monitoring.svc:8443' - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L229
• Added Bear token - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L230
• Added CA - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L231

2 - kubernetes-node-metrics - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L390

• Changed condition for kubernetes.controllermanager - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L412
• Changed condition for kubernetes.scheduler - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L425
• Changed address for kubernetes.proxy - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L435
• Added CA, Bear for all connection to 'https://${env.NODE_NAME}:10250' - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L444 https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L445
  CA for kubelet you can get from namespace: "openshift-kube-apiserver" --> kind: ConfigMap --> name: "kubelet-serving-ca"

3 - container-log - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L502

• Changed path to logs - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L529

[ChrsMark]

Thank you a lot for these details @Protopopys ! We will take care of these issues and for proxy specifically while at elastic/integrations#2065.

[tetianakravchenko]

Hi @Protopopys, thank you for the detailed reply!

I have few questions regarding your setup:
in your config you have 'https://kubernetes.default.svc.cluster.local:443' - https://gist.github.com/Protopopys/024679185b7a7e24fd3b551b1e343c75#file-gistfile1-txt-L210, instead of 'https://${env.KUBERNETES_SERVICE_HOST}:${env.KUBERNETES_SERVICE_PORT}' as defined here https://github.com/elastic/beats/blob/master/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L46
did you have some issue with 'https://${env.KUBERNETES_SERVICE_HOST}:${env.KUBERNETES_SERVICE_PORT}' ?

did you need to add any changes to the ClusterRole/Role(s) - https://github.com/elastic/beats/blob/master/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L640-L717 to have elastic-agent running?

[Protopopys]

Hi @tetianakravchenko !
1 - We can use a DNS record or an IP address (KUBERNETES_SERVICE_HOST). There were no problems in either case.
2 - My ClusterRole/Role(s) are identical to the ones you linked.

PS. I'm still testing and if I find anything, I'll let you know.