Add an integration test for several Kibana instances performing migration in parallel

[mshustov]

Migrations v2 have been designed to support resilient migrations in the event of several Kibana instances running migrations in parallel. Considering that it's one of the main features of the new algorithm, it must be covered by an integration test. Right now, the only way to test it is to use a script created by @Bamieh #84141 (comment).
It cannot be used for automated testing on CI.

It would be great if an automated test covers the case when different Kibana instances have different sets of enabled plugins to prove this edge case doesn't break migrations.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[joshdover]

Meeting notes:

• We may be able to leverage the https://github.com/elastic/elastic-stack-testing repo for implementing this test
• Does not necessarily need to run on CI at first, just something we could run locally would be ok.
• Likely a pretty good amount of effort to implement this
• Should be considered a blocker to closing Implement SavedObject Migrations v2 #75780

[afharo]

Likely a pretty good amount of effort to implement this

Should we consider this a Meta issue and break it down into smaller pieces?

[joshdover]

• Scope:
  • Add a test to try to catch race conditions between multiple nodes
  • Add a test to start nodes with different plugins enabled
  • Add a test to start two ES nodes in the same cluster, with a Kibana node configured to each one
  • Use the jest_integration suite to run multiple Roots against a single (real) ES cluster
  • Try to find a manual change to the migration algo that can be made to emulate a race condition to verify the test will catch race condition
  • Generate 1000-ish SOs and use a batchSize of 100 to emulate a slow enough migration process to try to create a race condition