Separate multi-node integ tests to their own job

[dbwiddis]

Description

Multi-node integration tests were set to run on the same server as integration tests had just run on. On cluster startup, the nodes were reading the previous cluster state from the (dropped) single node and uses the same stored indices.

Additionally ML Commons recently made the default true for a feature that auto-deploys ML models after a node drop and recovery, so these recoveries were occurring.

While no guarantee separating this out will fix the flaky macOS tests, I've seen enough hints in the logs to suggest that leftover bits from previous tests are slowing startup and potentially contributing to test failures.

Issues Resolved

Might resolve flaky tests. At least will cut down on the debugging noise.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ad199e3) 71.88% compared to head (cd14863) 71.88%.

Additional details and impacted files

@@            Coverage Diff            @@
##               main     #458   +/-   ##
=========================================
  Coverage     71.88%   71.88%           
  Complexity      620      620           
=========================================
  Files            78       78           
  Lines          3126     3126           
  Branches        236      236           
=========================================
  Hits           2247     2247           
  Misses          772      772           
  Partials        107      107           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[owaiskazi19]

Thanks for digging into this @dbwiddis. Great RCA.

[joshpalis]

Additionally ML Commons recently made the default true for a feature that auto-deploys ML models after a node drop and recovery, so these recoveries were occurring.

Interesting, I had thought we we're preventing the auto-redeployment of ML Models via the cluster settings here during set up

[dbwiddis]

Interesting, I had thought we we're preventing the auto-redeployment of ML Models via the cluster settings here during set up

Maybe that was a recent change that I missed when spelunking old logs.

But recovering indices and cluster state was happening and I'm pretty sure it impacted the timing of node startup.