CrashloopBackoff at Beat startup when trying to connect to Kibana

[sebgl]

Deploying [file|metric]beat with Kibana already responding leads to 3 restarts and a CrashloopBackOff of the Beats Pods.

That's because Beats cannot connect to Kibana yet. See this issue.

It's not too bad since the Pods finally come up about a minute later.
The experience when Kibana is created simultaneously is a bit worse (about one extra minute of startup time).
Overall I think it's confusing for the user and not as smooth as I would expect.

Some options:

1. Do nothing and consider it will be eventually fixed in Beats

2. Try to workaround this by waiting until Kibana is available and credentials are propagated before starting Beats. Up to a certain timeout in case something else is wrong, after which we should still start Beats and let the error surface.
   2a. In the operator, don't create the DaemonSet/Deployment until that's the case
   2b. Setup an init container that waits until Kibana is available before Beats starts

[pebrc]

I agree that the experience is not great. But my thinking is that any workaround introduces a non-trivial amount of complexity:

2a. In the operator, don't create the DaemonSet/Deployment until that's the case

afaik we do not make HTTP calls from the Beats controller to Kibana/Elasticsearch yet and I am bit wary of introducing them given the potential negative impact on the operator itself only to fix what appears to be a somewhat cosmetic issue at the moment.

2b. Setup an init container that waits until Kibana is available before Beats starts

In a way this approach is preferable because we have everything at hand what we need. Certificates and credentials are available as volumes in the pod. So if we really want to do something, this would be probably the marginally better approach.

[pebrc]

Won't fix.