Handle unreasonable amounts of Kibana index objects more gracefully during index migrating when upgrading

[geekpete]

Describe the feature:

With a large amount of objects in Kibana system indices, an upgrade migration might take an unreasonable amount of time to complete.

Describe a specific use case for the feature:

• First, at least warn that an unusual/large amount of objects have been detected in the Kibana system indices at migration time to give a heads up in the logs of what's going on.
  • Upgrade Assistant might also warn if it's relevant to actions the user might take ahead of time (eg cleaning out old cruft that hasn't been automatically cleaned up?)
• Then either process them faster or process the objects in an order suitable to priority of getting Kibana up and running again as quickly as possible even if still processing some of the remaining objects after Kibana is up?
  Or use different batch, batch delay or concurrency settings if object count is over a certain limit.
  • the default migrations.batchSize is small (100 documents)
  • Batches are sent serially, so we wait for each request to respond before sending the next batch, adding ~50ms delay between every batch
• For any objects that are unnecessary to be stored in newer versions, just delete any unneeded objects (at the end of the migration) or avoid migrating them by skipping them all together.
  • for example any changes to how Fleet works in newer versions that change the need for specific object types to be stored or needed even for migration in Kibana system indices, [Fleet] Migrate Agents from saved object to fleet server indices #87372

Even if we put better data lifecycle processes into newer versions that clean up excessive data, there might be migrations from older versions that don't have this and have large amounts of objects.
Any cleanup process we do have in newer versions might also be triggered before migration starts to reduce the workload on migration tasks, this could also be logged if occurring.

A recent customer case saw very large counts of objects that delayed the migration step by longer than some timeouts:

  "aggregations": {
    "saved_object_type": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 383,
      "buckets": [
        {
          "key": "fleet-agent-events",
          "doc_count": 822682
        },
        {
          "key": "action_task_params",
          "doc_count": 196979
        },

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[geekpete]

I think there has been many improvements for the object migration process since this issue was opened, is this issue still relevant?

[pgayvallet]

@rudolf @gsoldevila I think we're good closing this, right? Unless we have more related enhancements planned?

[rudolf]

Yes, we've significantly improved the scalability of migrations. Once users have upgraded to 8.8, subsequent upgrade migrations should be a lot faster and scalable. More details in #144035

[gsoldevila]

++ migration logic has been significantly enhanced from 8.8 onwards.
FWIW there's a low priority enhancement that we have in our TODO:
#160038