[CI] TestClustersPluginIT.testMultiNode failure

[henningandersen]

This test failed on my master PR build. I have not been able to reproduce it locally. Unsure if it relates to #37889.

This seems to be the primary problem:

[2019-04-16T09:36:36,076][WARN ][o.e.c.NodeConnectionsService] [multiNode-3] failed to connect to {multiNode-1}{76UnrzNcSQCGO2byg8l0-g}{pWe_gZtjS_22EyAtKnzMDA}{127.0.0.1}{127.0.0.1:33513}{testattr=test, ml.machine_memory=63323983872, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
  org.elasticsearch.transport.ConnectTransportException: [multiNode-1][127.0.0.1:33513] connect_exception
  	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:956) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

leading to:

`cluster{::multiNode}` failed to wait for cluster health yellow after 40 SECONDS

testMultiNode.zip

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

I tried to reproduce this both locally and on an identical worker but it won't reproduce.
There's not much to go by from the logs., I don't see why the nodes wouldn't be able to talk to each-other. Will try some more to reproduce with the entire suite.

[henningandersen]

This test failed again on my PR build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12570/testReport/org.elasticsearch.gradle.testclusters/TestClustersPluginIT/testMultiProject/

Looks like a similar but not identical error:

> `cluster{:bravo:myTestCluster}` failed to wait for cluster health yellow after 40 SECONDS

Will leave test unmuted for now and retry PR.

[henningandersen]

Once more, seems it fails frequently on my PRs:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12727/testReport/

though not on all PR builds:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12703/

[henningandersen]

Once again here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12965/testReport/

[henningandersen]

Turns out the last failures were for testMultiProject. testMultiNode is still muted.

Muted testMultiProject in 40216b4

[droberts195]

Both the affected tests are failing again:

• testMultiNode in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/13551/testReport/
• testMultiProject in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/13545/testReport/

In both cases the error is:

> `cluster{:alpha:myTestCluster}` failed to wait for cluster health yellow after 40 SECONDS

I guess some CI workers are slower than others and the startup time could be longer than expected due to a slow CI worker, so in case it's useful the 13551 failure was on elasticsearch-ci-immutable-debian-8-1557309753632904860 and the 13545 failure was on elasticsearch-ci-immutable-debian-9-1557304813981342182.

[alpar-t]

@droberts195 can you merge master in and try again please ? I added additional logging to see what the issue is.

[droberts195]

My PR is merged now, as I managed to get a green build after rerunning the tests - they got assigned to an Ubuntu worker on the run that succeeded. I see that both the workers that had problems for my PR have been replaced in Jenkins. So maybe there was something dodgy about those Debian workers.

[droberts195]

@atorok was fa98c5e the commit with the extra logging? If so a failure of TestClustersPluginIT.testMultiNode containing that is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/13576/testReport/

That build also ran on a Debian worker...

[alpar-t]

That's the right commit, but I'm not seeing the additional logs I was expecting

[alpar-t]

muted in 54f7113b2e8

[henningandersen]

Backported muting of testMultiProject to 7.1, failed here:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.1+multijob-unix-compatibility/os=centos-6/24/console