[ML] Fix flaky update_groups api test

[jgowdyelastic]

Related to #161324 and #160370
Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2649

I believe the problem lies with the function cleanMLSavedObjects only cleaning up saved objects in the default space and not in any other of the spaces which jobs or trained models may have been added to.
This causes an intermittent clash where a job's saved object already exists, but is in a different space. I don't know why this doesn't fail on every run.
The fix is to update cleanMLSavedObjects so it can take a list of additional space IDs to also clean. Any test which adds jobs or trained models to spaces other than default need to call this function and supply the list of space IDs it is using.
I've updated every test I could find in this PR.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[pheyos]

I'm wondering if this extra step is actually needed in all the places. A usual after block looks like this:

await spacesService.delete(idSpace1);
await spacesService.delete(idSpace2);
await ml.api.cleanMlIndices();
await ml.testResources.cleanMLSavedObjects();

Since we delete the created spaces first, the should be no saved objects left to clean in these spaces anymore (unless we do things behind the scenes there on space delete that I'm not aware of). The cleanMLSavedObjects method was introduced because we can't delete the default space to clean up.
However, it doesn't hurt in these places and is actually needed for situations where we run multiple tests in the same space setup and clean saved objects during afterEach and only remove spaces in the after block.
Having said that, I leave it to you whether or not to remove the optional spaces parameter from the cleanMLSavedObjects call where we've already cleaned up the spaces. I'm fine either way.

[pheyos]

The deleteSavedObjectById call in this method should pass through the space.

[jgowdyelastic]

I'm wondering if this extra step is actually needed in all the places. A usual after block looks like this:

This is a good point, and looking at it again I think the problem might actually be the tests in x-pack/test/api_integration/apis/ml/jobs/jobs.ts
This file creates AD jobs in spaces, but it doesn't create the spaces beforehand, so there is no space clean up at the end.

I'll see if creating and deleting the spaces in the before and after also fixes the flakiness and if so, revert a lot the calls to cleanMLSavedObjects

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: e4f441b

Metrics [docs]

✅ unchanged

History

• 💔 Build #142836 failed 1dbb273
• 💚 Build #142814 succeeded 47a0174
• 💚 Build #141058 succeeded f172f34
• 💚 Build #140926 succeeded dd5a0c3
• 💔 Build #140903 failed 0fc8da0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

[jgowdyelastic]

@pheyos I've made changes so now I'm only calling cleanMLSavedObjects with an extra list of spaces when the the jobs are created and cleaned up in beforeEach/afterEach functions.
I've also updated the tests in x-pack/test/api_integration/apis/ml/jobs/jobs.ts so the space is created and deleted, which should clean up the leftover saved objects. I still suspect this was the cause of the general flakiness.

[peteharverson]

Code LGTM