Cherry-pick #16373 to 7.x: Make k8s meta processor initialisation asynchronous

[ChrsMark]

Cherry-pick of PR #16373 to 7.x branch. Original message:

What does this PR do?

This PR changes add_kubernetes_metadata processor so as to get initialised asynchronously and being able to handle node not ready situations.

Why is it important?

This will be solve the problem of having to restart Beat in case the processor was not initialised if the node was not ready yet when Beat started.

How to test this PR locally

With the use of Minikube:

1. minikube start
2. Have a minikube ssh in one tab so as to handle k8s health up and down
3. Build Filebeat on the host where Minikube runs
4. Use the kubeconfig of Minikube: export KUBECONFIG=~/.kube/config
5. Make sure that add_kubernetes_metadata processor is enabled:

processors:
  - add_kubernetes_metadata: ~

6. Start filebeat and expect to see 2020-02-18T12:25:12.825+0200 INFO add_kubernetes_metadata/kubernetes.go:77 add_kubernetes_metadata: kubernetes env detected, with version: v1.17.0 in your logs, meaning that processor was able to reach k8s API.
7. Stop filebeat
8. In the session you have inside Minikube vm run sudo systemctl stop docker to bring docker down and make k8s API unreachable
9. Start filebeat again and you should see (after ~30secs) sth like 2020-02-18T12:27:16.958+0200 INFO add_kubernetes_metadata/kubernetes.go:89 add_kubernetes_metadata: could not detect kubernetes env: Get https://192.168.64.6:8443/version?timeout=32s: dial tcp 192.168.64.6:8443: connect: connection refused
10. Stop filebeat and start it again.
11. Wait 3-4 seconds and go to Minkube vm and start docker daemon again: sudo systemctl start docker.
12. In Filebeat's logs you should expect to see that the processor reached k8s API like in step 6.

Related issues

• Closes [Filebeat] pods on new nodes sometimes don't detect the Kubernetes environment #16064