Implement client-side updates w/ mget to prune stale docs

[kobelb]

Feature Description

Change the task manager task claiming algorithm to use a _search to retrieve candidate tasks, a _mget to prune the docs whose version number doesn't match, and then a _bulk to claim the tasks. This will increase the background task capacity in Serverless.

Business Value

Increased background task capacity, reducing the COGS for running alerting rules and actions, and providing a lower MTTD/MTTR.

Definition of Done

• Changes to the algorithm
• Performance tests against Stateful Elasticsearch to ensure performance did not degrade significantly
• Performance tests against Serverless Elasticsearch to measure performance
• Unit tests

Phases

1. [Task Manager] allow multiple task claiming strategies #171677
2. Create alternative task claimer that performs client-side updates w/ mget to prune stale docs #181325
3. Performance test the alternative task claimer that uses client-side updates w/ mget to prune stale docs #181326
4. Rollout client-side updates w/ mget as the default task claimer #181327

Implementation: multiple PRs:

• [Task Manager] allow multiple task claiming strategies #171677 - re-org code to allow multiple claim getting algorithms
• TBD - implement algorithm described here as an option to be used in serverless to start with

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[kobelb]

/cc @pmuellr

[pmuellr]

@kobelb looks like a POC of this is here: main...kobelb:kibana:task_clientside_update

Is that right?

[kobelb]

@pmuellr main...kobelb:kibana:task_clientside_update was a super early attempt at this that I wouldn't recommend treating as a proof-of-concept for this implementation.

#150769 minus the parts that do "task partitioning" are closer to what we want here. Happy to discuss further.

[mikecote]

@pmuellr here's a PR (#157156) to what I did for ON week, it contains some client-side update code that you can pick from as well (minus the cost parts). I removed a little bit of RxJS as well.

If ever we wanted to explore having the search -> update logic in a worker thread, I've POC'ed it here: mikecote#5.

I've also POC'ed how to skip the claiming phase if ever we wanted to save an update -> mikecote#6. But it may be useful to not do this if ever we decide to use worker threads..

[mikecote]

Here's a rollout plan after discussing with @kobelb:

• We'll need the task delay monitoring (https://github.com/elastic/response-ops-team/issues/138) before we start enabling the new polling mechanism on projects
• A configuration change will be required to opt-in to the new polling mechanism (feature flag)
• We'll slowly enable it on select projects and expand as we see the poller working fine
• We'll test as much as we can in unit tests and integration tests
• Code reviews

Following this plan should allow us to implement the new polling mechanism while mitigating risk.

cc @pmuellr