Document and use migrations.batchSize in v2 migrations

[kobelb]

The saved-object migrations v1 used the undocumented migrations.batchSize setting to control how many documents are migrated per batch. We found some utility in allowing users to increase this setting to deal with a large number of saved-objects that should be migrated.

However, the saved-object migrations v2 don't use this setting and they're currently hard-coded to migrate 1000 saved-objects at a time. Per @rudolf's suggestion, we should document this setting and consume it for v2 migrations.

Outdated old description

The default for `migrations.batchSize` was chosen a long time ago. We've continued to see issues with migrations run against Kibana system-indices that have a very large number of documents. One of the workarounds that have been employed is manually increasing the `migrations.batchSize` setting from `100` to `1000`. Should we consider increasing the default from `100` to `1000`?

Conceptually, this will cause Kibana to consume 10x more heap during migrations. Do we have any idea how large these saved-objects are to determine the real impact this change would have?

Are there any other side-effects we should take into consideration?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

Some documents like canvas templates can be 10 megabytes, but this is definitely the exception. We won't really be able to come up with a good estimate for how much heap it could consume because the document size could be so different.

When I tested with sample data and 100k saved objects I couldn't see a difference in Kibana's or Elasticsearch's heap consumption between batches of 100 or 1000 documents. So v2 migrations are currently hardcoded to 1000 document batches which is also Elasticsearch's default for the reindex operation.

Given that we can't really choose the perfect value here because of the uncertainty it makes sense that this should also be configurable for v2 migrations and documented.

[kobelb]

@rudolf I've updated the description to reflect the suggestions that you made and I've removed the discuss label.