Update controllermanager metrics

[MichaelKatsoulis]

What does this PR do?

This PR updates the metrics that Kubernetes controllermamager metricset collects.
Also the OOTB dashboard is updated.

Why is it important?

controllermamager metricset currently collects metrics from kubernetes controllermanager prometheus endpoint that have been deprecated in latest versions of kubernetes. Currently we support k8s versions starting from 1.21.x.
As a result values from those fields are always nil and some of the dashboard visualisations show no values.

Those deprecated prometheus fields are

http_request_duration_microseconds
http_request_size_bytes          
http_response_size_bytes                
http_requests_total 

http_request_duration_microseconds can instead be collected from rest_client_request_duration_seconds
For the rest of them, there are no more, related metrics.

Also metrics like
process_max_fds, workqueue_work_duration_seconds are interesting and were added.

As part of this PR the following elasticsearch fields where dropped (their values were always nil in k8s versions we support)

• kubernetes.controllermanager.http.request.duration.us.percentile.*
• kubernetes.controllermanager.http.request.duration.us.sum
• kubernetes.controllermanager.http.request.duration.us.count
• kubernetes.controllermanager.http.request.size.bytes.percentile.*
• kubernetes.controllermanager.http.request.size.bytes.sum
• kubernetes.controllermanager.http.request.size.bytes.count
• kubernetes.controllermanager.http.responsesize.bytes.percentile.*
• kubernetes.controllermanager.http.response.size.bytes.sum
• kubernetes.controllermanager.http.response.size.bytes.count
• kubernetes.controllermanager.http.request.count
• kubernetes.controllermanager.handler

and the following fields added:

• kubernetes.controllermanager.url
• kubernetes.controllermanager.verb
• kubernetes.controllermanager.fds.max.count
• kubernetes.controllermanager.client.request.duration.sec.sum
• kubernetes.controllermanager.client.request.duration.sec.count
• kubernetes.controllermanager.client.request.duration.sec.bucket.*
• kubernetes.controllermanager.workqueue.work.duration.sec.sum
• kubernetes.controllermanager.workqueue.work.duration.sec.count
• kubernetes.controllermanager.workqueue.work.duration.sec.bucket.*

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Deploy Metricbeat from this PR branch with controllermanager metricset enabled.

- module: kubernetes
   metricsets:
       - controllermanager
   hosts: ["https://0.0.0.0:10257"]
   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
   ssl.verification_mode: "none"
   period: 10s

Prefer Kind or Minikube in order for metricbeat to have access to controllermanager endpoint.
Check for the new fields and setup the new dashboards
./metricbeat setup --dashboards -E setup.dashboards.directory=./module/kubernetes/_meta/kibana/ -E setup.kibana.host=http://elastic:changeme@localhost:5601 -E ssl.verification_mode=none -e -d "*"

Related issues

• Closes Update kubernetes controller manager metrics and dashboard #32007

Use cases

Screenshots

New dashboard

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-28T07:11:16.554+0000

• Duration: 60 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3532
Skipped	873
Total	4405

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[gizas]

Only if you have the official link with the list of updated metrics that new kubernetes controlmanager supports add it to the description