Make k8s meta processor initialisation asynchronous

[ChrsMark]

What does this PR do?

This PR changes add_kubernetes_metadata processor so as to get initialised asynchronously and being able to handle node not ready situations.

Why is it important?

This will be solve the problem of having to restart Beat in case the processor was not initialised if the node was not ready yet when Beat started.

How to test this PR locally

With the use of Minikube:

1. minikube start
2. Have a minikube ssh in one tab so as to handle k8s health up and down
3. Build Filebeat on the host where Minikube runs
4. Use the kubeconfig of Minikube: export KUBECONFIG=~/.kube/config
5. Make sure that add_kubernetes_metadata processor is enabled:

processors:
  - add_kubernetes_metadata: ~

6. Start filebeat and expect to see 2020-02-18T12:25:12.825+0200 INFO add_kubernetes_metadata/kubernetes.go:77 add_kubernetes_metadata: kubernetes env detected, with version: v1.17.0 in your logs, meaning that processor was able to reach k8s API.
7. Stop filebeat
8. In the session you have inside Minikube vm run sudo systemctl stop docker to bring docker down and make k8s API unreachable
9. Start filebeat again and you should see (after ~30secs) sth like 2020-02-18T12:27:16.958+0200 INFO add_kubernetes_metadata/kubernetes.go:89 add_kubernetes_metadata: could not detect kubernetes env: Get https://192.168.64.6:8443/version?timeout=32s: dial tcp 192.168.64.6:8443: connect: connection refused
10. Stop filebeat and start it again.
11. Wait 3-4 seconds and go to Minkube vm and start docker daemon again: sudo systemctl start docker.
12. In Filebeat's logs you should expect to see that the processor reached k8s API like in step 6.

Related issues

• Closes [Filebeat] pods on new nodes sometimes don't detect the Kubernetes environment #16064

[jsoriano]

Only some small details, and there seems to be some format error.

[ChrsMark]

Changed some stuff and added testing notes. Tested it with minikube with the process described in the PR's description.

[jsoriano]

Nit. Imports order, this libbeat import should be below with the other ones.

[ChrsMark]

While fixing the import order I noticed that one import("github.com/cloudflare/cfssl/log") was not used 🤔 so I removed it and also tried to unify the logging using the processor's logger along. Sorry for the back-and-forth 🙂 .

[jsoriano]

While fixing the import order I noticed that one import("github.com/cloudflare/cfssl/log") was not used thinking so I removed it and also tried to unify the logging using the processor's logger along. Sorry for the back-and-forth slightly_smiling_face .

Good catch!