[PoC] Attaining 10x alerting throughput (32,000 rules per minute)

[mikecote]

In this PoC, I made the improvements listed below to move the alerting scalability ceiling (rules per minute) by at least 10x. The scenario used is creating ES Query rules that run every minute on sample indices that do not detect alerts.

List of improvements

• Set .kibana_task_manager and .kibana_alerting_cases index configurations to 3 shards
• Set task manager's max workers to 50 from 10
• Set task manager's poll interval to 500ms from 3s
• Assign tasks to one of 360 partitions
• Assign Kibana instances to some of the 360 partitions in a round-robin manner
• Set xpack.alerting.maxScheduledPerMinute to 1000000 to increase the upper bound limit
• Use mget as the task-claiming strategy
• Skip loading dataViews and searchSourceClient alerting rule executor services when not necessary (Lazy load dataViews and wrappedSearchSourceClient services when running alerting rules #184322)
• Cache flapping settings (Optimize access to flapping settings in the task runner #149884)
• Cache query delay settings (Cache query delay settings in the alerting task runner #184321)
• Only load maintenance windows when there are alerts (Cache and load maintenance windows only when they are needed #184324)
• Remove event log document validation
• Removed the claiming phase of tasks (Make the mget task claimer skip the claiming phase and update the task document directly to running #184739)
• Perform a partial update when claiming tasks instead of using the saved-object repository (which does a get + index within Kibana)
• Perform a partial update when rescheduling a task instead of using the saved-object repository (which does a get + index within Kibana)
• Perform a partial update when updating the alerting rule instead of using the saved-object repository (which does a get + index within Kibana)
• Make the task claiming queries only return the necessary fields, then do an mget with the claimed tasks to get all the fields

Test scenario

• Elasticsearch Query rule
• 1m interval
• Queries either one of the sample indices or the event log index
• Rule is written to not detect alerts after running the query

Notes

• When the search queries encounter an HTTP 429 or 504 error, we need to set more replicas on the affected index
• When encountering slow or failing _has_privileges API calls to Elasticsearch, we need to set the .security index settings to have auto_expand_replicas: 0-all so not only one node is capable of performing the requests
• The API key cache needs to be bumped to a number higher than the number of alerting rules (xpack.security.authc.api_key.cache.max_keys: 50000)
• If 429 errors are observed when updating a task or alerting rule saved-object, we need to observe the Elasticsearch node in context and ensure it is not hosting other primary shards of task or rule saved-objects.
• Rendezvous hashing did not seem to distribute the partitions evenly across Kibana, using the round-robin method may be sufficient for now.
• At times when pushing the boundaries, Elasticsearch queries at times timeout while returning the majority of the queries within seconds. Moving Elasticsearch to multi zone seems to solve the issue.

Conclusion

These optimizations have shown that we can attain a 10x scale with the alerting system. However, during further testing, I was able to push the limits even further, attaining much more than 10x in various ES and Kibana configurations, confirming that this approach will break the horizontal scalability ceiling that we previously had.

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: 5dfa570
• Interpreting CI Failures
• Cloud Deployment

Failed CI Steps

History

• 💔 Build #218244 failed 5dfa570
• 💔 Build #217962 failed 9c6c627
• 💔 Build #217938 failed 674f433
• 💔 Build #217757 failed 14abf9d
• 💔 Build #217676 failed 7973743
• 💔 Build #217534 failed 7973743

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[mikecote]

/ci

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: a047b38
• Cloud Deployment

Failed CI Steps

History

• 💔 Build #221476 failed 899cdd3
• 💔 Build #221114 failed 5dfa570
• 💔 Build #219612 failed 5dfa570
• 💔 Build #219500 failed 5dfa570
• 💔 Build #219337 failed 5dfa570