Migrations: dynamically adjust batchSize when reading

[rudolf]

Release notes

Very large saved objects can cause migrations to fail when reading batches of saved objects from Elasticsearch that exceeds the NodeJS max string length. This forces users to use a much smaller migrations.batchSize value which in turn slows down migration performance. With this fix, Kibana will dynamically reduce the batch size when it encounters a batch that's too big to process improving migration performance.

Summary

Migrations read 1000 documents by default which works well for most deployments. But if any batch happens to be > ~512MB we hit NodeJS' max string length limit and cannot process that batch. This forces users to reduce the batch size to a smaller number which could severely slow down migrations.

This PR reduces the impact of large batches by catching elasticsearch-js' RequestAbortedError and reducing the batch size in half. When subsequent batches are successful the batchSize increases by 20%. This means we'll have a sequence like:

1. Read 1000 docs ✅ (small batch)
2. Read 1000 docs 🔴 (too large batch)
3. Read 500 docs ✅
4. Read 600 docs ✅
5. Read 720 docs ✅
6. Read 864 docs ✅
7. Read 1000 docs ✅ (small batch)

This assumes that most clusters just have a few large batches exceeding the limit. If all batches exceed the limit we'd have 1 failure for every 4 successful reads so we pay a 20% throughput penalty. In such a case it would be better to configure a lower migrations.batchSize.

Tested this manually:

1. Start ES with more heap than the default, otherwise reading large batches will cause it to run out of memory
   ES_JAVA_OPTS=' -Xms6g -Xmx6g' yarn es snapshot --data-archive=/Users/rudolf/dev/kibana/src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
2. Ingest lots of large documents of ~5mb

      curl -XPUT "elastic:changeme@localhost:9200/_security/role/grant_kibana_system_indices" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
      {                           
        "indices": [                                            
          {
            "names": [
              ".kibana*"
            ],
            "privileges": [
              "all"
            ],
            "allow_restricted_indices": true
          }
        ]
      }'
   
      curl -XPOST "elastic:changeme@localhost:9200/_security/user/superuser" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'    
      {
        "password" : "changeme",  
        "roles" : [ "superuser", "grant_kibana_system_indices" ]
      }'
   
      curl -XPUT "superuser:changeme@localhost:9200/.kibana_8.4.0_001/_mappings" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
      {
            "dynamic": false,                                                                       
            "properties": {
   
            }
   
      }'
   
      set -B                  # enable brace expansion
      for i in {1..400}; do
        curl -k --data-binary "@/Users/rudolf/dev/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/body.json" -X PUT "http://superuser:changeme@localhost:9200/.kibana_8.4.0_001/_doc/cases-comments:"{$i}"?&pretty=true" -H "Content-Type: application/json"
      done
   

3. Start Kibana with a modest batchSize otherwise we could OOM ES node scripts/kibana --dev --migrations.batchSize=120

Example logs. Note the "Processed x documents" only logs when the next batch is successfull read, so the order seems wrong. To improve it we'd need to log progress after a batch is successfully written instead 🤷

``` [.kibana] Processed 120 documents out of 542. [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3667ms. [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1740ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1376ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1402ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1311ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1388ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 900ms. [.kibana] Read a batch that exceeded the NodeJS maximum string length, retrying by reducing the batch size in half to 60. [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_READ. took: 1538ms. [.kibana] Processed 240 documents out of 542. [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 2054ms. [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1042ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1310ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1388ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 1130ms. [.kibana] Processed 300 documents out of 542. [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 2610ms. [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1262ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1299ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1363ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1341ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 572ms. [.kibana] Processed 372 documents out of 542. [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3330ms. [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1488ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1349ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1312ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1380ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1310ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 139ms. [.kibana] Processed 458 documents out of 542. [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3278ms. [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1460ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1370ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1303ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1384ms. [.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 1298ms. [.kibana] Processed 542 documents out of 542. [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 4ms. ```

### Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

Risks

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

CI was failing a few times on this test due to timeout

[rudolf]

This test configures a really small maxReadBatchSizeBytes so that we hit the es response size too large error. I tried by actually creating really large documents but this consumes a lot of memory on the ES side which both when reading batches, but also when the update_by_query runs.

[rudolf]

@elasticmachine merge upstream

[TinaHeiligers]

note to any reviewers: Replace the full path to the data archive with the full path to the archive on your machine when running the branch.
i.e.

ES_JAVA_OPTS=' -Xms6g -Xmx6g' yarn es snapshot --data-archive=<path-to-repo>/kibana/src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip

[TinaHeiligers]

I had to reduce the batch size all the way down to < 100 to get around
Error: Unable to complete saved object migrations for the [.kibana] index. RequestAbortedError: The content length (536936024) is bigger than the maximum allowed string (536870888)
Migrations ran fine after that! Being able to specify the migrations batch size is going to be a huge win for getting around the "too-many-saved-objects" issue.

LGTM on CI green.

[gsoldevila]

NIT you can directly use the startElasticsearch() method of the kibana_migrator_test_kit.ts

[gsoldevila]

TIL elastic-transport-js is handling this limit:
https://github.com/elastic/elastic-transport-js/blob/main/src/connection/HttpConnection.ts#L146,L159

[jloleysens]

Left a drive-by review, non blocker comments only

[jloleysens]

Left a drive-by review, non blocker comments only

[pgayvallet]

NIT: probably unnecessary, but we may want to have an escape hatch to avoid potentially entering an infinite loop here? Should we check if stateP.batchSize is higher than 1/2 or something here?

[rudolf]

I added better handling for when the response size > maxReadBatchSizeBytes even if the batchSize is 1. Also addresses a similar comment from @jloleysens. Even though we can't create documents that exceeds MAX_STRING_LENGTH users could configure maxReadBatchSizeBytes to a low value to e.g. avoid an OOM so it felt worth explicitly handling for this scenario.

[pgayvallet]

Do we want to also plug this logic into the zdt algorithm?

[rudolf]

yeah, would definitely be worth adding there. I'll do that in a follow-up PR 👍

[afharo]

nit: .test might be more performant because it doesn't need to extract the groups. I don't have numbers that back this assumption though.

Suggested change

	e.message.match(/The content length \(\d+\) is bigger than the maximum/) != null
	/The content length \(\d+\) is bigger than the maximum/.test(e.message)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: eb35b1b

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/core-saved-objects-migration-server-internal	86	89	+3

Unknown metric groups

API count

id	before	after	diff
@kbn/core-saved-objects-migration-server-internal	120	123	+3

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	401	405	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	481	485	+4
total			+6

History

• 💚 Build #130944 succeeded 0efb67a
• 💔 Build #130320 failed b81deb1
• 💔 Build #130045 failed c3c272f
• 💔 Build #130039 failed 98826d4
• 💔 Build #128513 failed 532a41d
• 💔 Build #128272 failed 5202b28

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @rudolf

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.8	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 157494

Questions ?

Please refer to the Backport tool documentation

[gsoldevila]

💚 All backports created successfully

Status	Branch	Result
✅	8.8

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation