[ML] anomaly detection Job failed without reason audit message when model snapshot is too old

[wwang500]

Steps to reproduce (easier to do this from a cloud env)

• Deploy a 7.8.1 cluster and start a real-time anomaly detection job,
• Then create a 8.0 cluster by restoring that 7.8.1 cluster snapshot,
• after 8.0 cluster successfully was created, login to check the AD job status:

AD job failed on UI, and there is no failed reason being thrown, neither in job message. (At the time of 11:56 on the below screenshot)

After stop datafeed and force-close job (at the time of 16:01 on above screen), restarting job gave the root cause of job failure, which was caused by old model snapshot:

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "[response_code_rates] job snapshot [1636472812] has min version before [7.0.0], please revert to a newer model snapshot or reset the job"
      }
    ],
    "type": "exception",
    "reason": "[response_code_rates] job snapshot [1636472812] has min version before [7.0.0], please revert to a newer model snapshot or reset the job"
  },
  "status": 500
}

Expected behaviour:
Good failure message on UI and job messages

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[droberts195]

Some extra information about this:

1. The 7.8.1 snapshot contained job model states dating back to 6.x. So this is why the 8.0 job (correctly) refuses to load these.
2. The job goes into the failed state at 11:56, but the datafeed remains started. This highlights that [ML] Datafeed assignment should cancel datafeeds with failed jobs #48934 is still a problem, and the fact that that issue is still not fixed is the reason why we see support cases where datafeeds are started while the corresponding jobs are not opened.

So we should add an audit message that records what happens when an open job goes into the failed state when being assigned to a new node. And we should also fix #48934 (which would have benefits beyond upgrade use cases).

[droberts195]

This is actually one particular case of a broader problem. There are a number of reasons why a job can fail when it is assigned to a new node, for example:

1. Failure to update mappings
2. Failure to revert to current snapshot
3. Failure to search for associated datafeed

None of these were creating an audit message, so would have been invisible to the end user viewing the UI. #80665 should fix those 3 (which existed in 7.x) plus the one this issue was opened for (which is new in 8.0).