Healthcheck timeouts

[cachedout]

What does this PR do?

This is an attempt to possibly improve the situation with the recent round of APM Integration Test failures.

I have tried as many ways as I can think of to reproduce the errors, which seem to be related to the health-check for the Kibana container failing on start. I've run it locally, run it on the same type of Jenkins worker instance, and at no point have I been able to reproduce the 5-minute timeouts that we've been seeing.

We've also reported this problem to Kibana, and they also cannot seem to replicate it.

The only theory I really have at this point is that perhaps there is something specific in our health-check that is causing the issue, like perhaps cURL is getting the socket half-open and the health-check is just hanging.

This PR introduces a few means of trying to prevent that case from happening:

• Default timeout of 5s for all health-check calls
• A 10s period at the top of the Kibana health-check to wait for Kibana to become operational

Why is it important?

Trying to get our ITs back online.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-08-17T11:57:44.395+0000

• Duration: 38 min 4 sec

• Commit: acdd4b1

Test stats 🧪

Test	Results
Failed	0
Passed	496
Skipped	1
Total	497

Trends 🧪