TooManyJobsIT#testMultipleNodes fails: Had to resort to force-closing job

[DaveCTurner]

I see a couple of builds failed like this:

• https://gradle-enterprise.elastic.co/s/jxb4obrmhi7vo
• https://gradle-enterprise.elastic.co/s/kum2wqwz2pkcc

org.elasticsearch.xpack.ml.integration.TooManyJobsIT > testMultipleNodes FAILED
    java.lang.RuntimeException: Had to resort to force-closing job, something went wrong?

        Caused by:
        java.util.concurrent.ExecutionException: org.elasticsearch.transport.RemoteTransportException: [node_t1][127.0.0.1:44709][cluster:admin/xpack/ml/job/close]

            Caused by:
            org.elasticsearch.transport.RemoteTransportException: [node_t1][127.0.0.1:44709][cluster:admin/xpack/ml/job/close]

                Caused by:
                java.lang.IllegalStateException: Timed out when waiting for persistent tasks after 30s
REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testMultipleNodes" -Dtests.seed=62CDCB396D54BF7A -Dtests.security.manager=true -Dtests.locale=is -Dtests.timezone=Europe/Sofia -Dcompiler.java=12 -Druntime.java=11

Possibly this is #30300 again, or maybe this is just the CI machine running slowly?

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

or maybe this is just the CI machine running slowly?

Yes. We allow 30 seconds to close all the jobs. They did actually all close successfully, but this took 79 seconds in the first build and 48 seconds in the second. The log from the first failure shows:

[2019-08-09T02:02:48,163][INFO ][o.e.x.m.i.TooManyJobsIT  ] [testMultipleNodes] Closing jobs using [_all]
...
[2019-08-09T02:04:07,320][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [node_t2] [max-number-of-jobs-limit-job-39] job closed

and the log from the second failure shows:

[2019-10-25T07:48:17,023][INFO ][o.e.x.m.i.TooManyJobsIT  ] [testMultipleNodes] Closing jobs using [_all]
...
[2019-10-25T07:49:05,701][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [node_t3] [max-number-of-jobs-limit-job-37] job closed

This is probably due to running multiple test tasks in parallel. If this test happens to run at the some time as some other resource intensive test suite it takes too long.

I will bump the timeout for ML cleanup in the internal cluster tests up to 90 seconds.