[ML] Add basic BWC tests for data frame analytics

[dimitris-athanasiou]

Adds basic BWC tests for data frame analytics jobs.

[elasticmachine]

Pinging @elastic/ml-core

[przemekwitek]

LGTM

One question though: Which fields of the "get" and "get stats" response should we verify?
Do you think it is worth adding more fields to increase coverage?

[dimitris-athanasiou]

@przemekwitek Sure, why not? I'll add some more match clauses.

[benwtrent]

Do we want to test starting/stopping the analytics as well? I suppose it would start/stop very quickly, so validating state afterwards might not work, but we should at least verify that neither API throws an error.

[benwtrent]

I think we should test putting analytics on the mixed cluster as well.

[dimitris-athanasiou]

@benwtrent I thought about testing start/stop. We can't know how far the job went to decide whether we can restart it without conditionals, so we can only do so in one of the cluster states (old, mixed or upgraded). Thus, I don't see much value doing so. I have talked to QA asking them to add a test scenario where they start a longer running job and then perform a rolling upgrade to cover this.

[droberts195]

We can't know how far the job went to decide whether we can restart it without conditionals, so we can only do so in one of the cluster states (old, mixed or upgraded)

I agree it's pointless to test start/stop in the old cluster as that's what the integration tests of that branch will be testing. But since we can test start/stop in one of the states I think mixed would be worthwhile, as that's unlikely to get tested in other places. It would be useful to find out if something we've changed prevents old analytics jobs from starting up in the mixed version cluster. That could be an early warning of something we've overlooked with the BWC. (And then I agree that since the analytics job has done something in the mixed cluster it cannot be started again in the fully upgraded one.)

I have talked to QA asking them to add a test scenario where they start a longer running job and then perform a rolling upgrade to cover this.

Yes, I think this is best to leave this more complex scenario to the QA tests. There will be fewer spurious failures caused by things like #36816 (comment).

[dimitris-athanasiou]

I have updated this PR to include tests to put a job in the mixed cluster as well as start/stop jobs during that state.

[benwtrent]