Flaky x-pack libbeat build on travis

[ruflin]

x-pack libbeat build

• Test Name: x-pack libbeat build on Travis
• Branch: master
• Notes: I initially assume it's a timeout and travis machines are not powerful enough to build it. But I increased the timeout from 300 to 600 seconds which should be enough. As almost the same builds run for non x-pack libbeat and work I assume it's either related to security enabled and it takes longer to boot (and is indeed a timeout) or that in same cases the health check is not 100% correct for some states.

Build is skipped here: #9689

Stack Trace

Successfully tagged libbeat667d4597047394139f4e96d47c5ca886faea66b5_beat:latest
TESTING_ENVIRONMENT=snapshot ES_BEATS=../.. docker-compose -p libbeat667d4597047394139f4e96d47c5ca886faea66b5  -f docker-compose.yml run beat make integration-tests RACE_DETECTOR= DOCKER_COMPOSE_PROJECT_NAME=libbeat667d4597047394139f4e96d47c5ca886faea66b5 
Creating network "libbeat667d4597047394139f4e96d47c5ca886faea66b5_default" with the default driver
Pulling elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:6.6.0-SNAPSHOT)...
6.6.0-SNAPSHOT: Pulling from elasticsearch/elasticsearch
Digest: sha256:762b4aace32e8e0bd4412f58bb2b044b30a97ff1d670e0b667869ef541e1fb08
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:6.6.0-SNAPSHOT
Pulling kibana (docker.elastic.co/kibana/kibana:6.6.0-SNAPSHOT)...
6.6.0-SNAPSHOT: Pulling from kibana/kibana
Digest: sha256:41060012cfd3a8d8e26ea840341a9ec22b636c067da1d6e48662a4e49f0b6771
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:6.6.0-SNAPSHOT
Pulling proxy_dep (busybox:latest)...
latest: Pulling from library/busybox
Digest: sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Status: Downloaded newer image for busybox:latest
Creating libbeat667d4597047394139f4e96d47c5ca886faea66b5_kibana_1
Creating libbeat667d4597047394139f4e96d47c5ca886faea66b5_elasticsearch_1
ERROR: for proxy_dep  Container "83be7db278f5" is unhealthy.
ERROR: Encountered errors while bringing up the project.
make[1]: *** [integration-tests-environment] Error 1
make[1]: Leaving directory `/home/travis/gopath/src/github.com/elastic/beats/x-pack/libbeat'
make: *** [testsuite] Error 2
make: Leaving directory `/home/travis/gopath/src/github.com/elastic/beats/x-pack/libbeat'
The command "make $TARGETS" exited with 2.

[ph]

FYI @urso

[ph]

@ruflin This problem is not easy to reproduce and is really dependent on the infrastructure stability at Travis and probably related to noisy neighbors you have on the execution machine.

I wonder if we could keep the same healthcheck in our docker-compose file and instead add a restart: on-failure:5 (docs) policy on the containers, this would force to try to restart at most 5 times the containers if the execution status is not 0. So we would either hit the limit or hit the job execution limit of Travis.

I've made the change in the following PR #9814, and I've run it 20 times the failing job but it sadly it never failed.

I do not know if we ever tried the above before, maybe you know.

Also I went into a rabbit hole of docker version 3 and version 2 syntax difference ;)

[ruflin]

Let's merge it and see if it solves the issue.

I'm glad you didn't branch out into the 2.0/2.1 differences too ;-) One day we should start using newer version but as you found out, not that easy with our current dependency setup ...

[ph]

I'm glad you didn't branch out into the 2.0/2.1 differences too ;-) One day we should start using newer version but as you found out, not that easy with our current dependency setup ...

Yes I found that out. It appear that the docker team do not care about that specific usecase.

[ruflin]

Reopening because of #10068

[ruflin]

@ph Can this be closed again?

[ph]

@ruflin I am renabling them only for 6.6 and 7.0, I do believe we don't have any other snapshot ready. :(

[ruflin]

@ph Was just curious, will leave it to you to close it when needed.

[ph]

For 7.0 #10649
For master https://beats-ci.elastic.co/job/elastic+beats+pull-request/6610/

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.