migrationsv2: Dynamically adjust batch size to prevent exceeding ES HTTP payload size

[rudolf]

As described in #107288 migrations can fail when they create batches of saved objects that are larger than the configured ES http.max_content_length (default 100Mb).

Since saved objects vary greatly in size from a few kb to several megabytes it's very difficult to choose an appropriate batch size for a given deployment. Documents like siem-detection-engine-rule-status store debug logs which come from external systems introducing variability even within the same type.

To ensure a smooth upgrade experience without reducing the default batch size, we should dynamically adjust the size of each indexed batch to ensure that the default payload size is never exceeded.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

Thoughts on implementation...

For simplicity we should continue to read batches of migrations.batchSize, theoretically reading a large batch could exceed Kibana's available heap, but this is currently a low risk.

So the dynamic batch size would only be relevant when indexing documents (the bulkOverwriteTransformedDocuments action). I see two alternatives:

1. Use a library like json-size to calculate the size of each document as we add it to the current batch (elasticsearch uses the uncompressed payload size to enforce it's limit). Once the batch is "full" or has migrations.batchSize documents, index it and build the next batch.
2. Index using migrations.batchSize until we hit a 413 error and then halve the payload size before trying again.

Unless (1) adds significant CPU overhead and slows down migrations, it feels like the preferred approach.

[rudolf]

Duplicate of #107641