Test impact of using refresh: false for task manager internals

[dgieselaar]

In #99160, we are considering setting refresh to false for Task Manager internal operations, like creating/updating/deleting tasks. By default, the Saved Objects client will use wait_for, which means that it will keep a connection open to Elasticsearch until a shard gets refreshed. By default, this is 1 second. This means that workers are not freed up as quickly as they could be, and can have a negative impact on the rate of tasks that can be executed.

We should test not just the functionality, but also the performance impact, by running a small (local) load test.

We should also do a larger-scale load test, but given the complexity, we will address that separately.

cc @pmuellr @gmmorris

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

Thank Dario, added to out project for triage 👍

I would separate the additional instrumentation from this change - these are two distinct deliverables, and tying them together will make it harder to tell what the source of an issue is if we deliver them together.
Both changes have potential downstream impact in terms of perf and lifecycle eccentricities, so I'd like to keep them apart if possible.

[mikecote]

For this issue, we should help finalize this PR #99919 and do some load testing before the end of the release cycle (#95194).

[YulNaumenko]

This issue seems to be a part of the current one or should be implemented as the next step.

[YulNaumenko]

Based on the discussion with Dario, here is some details about setup a local environment:

1. Configure your local Kibana kibana.dev.yml with the next options:

elastic.apm.transactionSampleRate: 1
elastic.apm.breakdownMetrics: true
elastic.apm.active: true
elastic.apm.environment: yuliia
elastic.apm.disableInstrumentations:
  - http
  - https

2. yarn es snapshot --ssl
3. yarn start --ssl
4. Create Rule with actions. I'm using es-apm-sys-sim -r 40 15 es-apm-sys-sim https://elastic:changeme@localhost:9200 for the data
5. Makes sure it run successfully
6. Open https://ela.st/kibana-ops-ci-stats
7. Select your local environment name from the list of the environments:

8. Navigate to APM Transactions and select transaction type `taskManager run`:

9. Select some of the longest running the transactions could be improved:

[mikecote]

This issue didn't make it part of the 7.15 planning. @YulNaumenko, how much effort is left before resolving this? We are planning to move this to the backlog for now.

[YulNaumenko]

This issue didn't make it part of the 7.15 planning. @YulNaumenko, how much effort is left before resolving this? We are planning to move this to the backlog for now.

The most effort currently is on the functional and performance testing. It looks like loe:week

[ymao1]

Closing as done

[sorenlouv]

@ymao1 Did this get implemented or what was the outcome of the perf test?

[mikecote]

@ymao1 Did this get implemented or what was the outcome of the perf test?

@sqren I did some tests during my April ON-week and couldn't find places that were missing refresh: false. So we decided to close it during last week's grooming session. Everything should be 🔥 fast 😎 .

[dgieselaar]

@mikecote just to check: the default from the SO client is 'wait_for', no? Do you mean that the Alerting/Task Manager sets it to refresh: false where possible? It still would have been nice to see the impact :)

[mikecote]

@mikecote just to check: the default from the SO client is 'wait_for', no? Do you mean that the Alerting/Task Manager sets it to refresh: false where possible?

That's correct, so we had to change a bunch of places to refresh: false explicitly where we didn't rely on/need the data to be searchable right away. We've done those changes a long time ago (7.10 - 7.11 / Alerting GA) and have been holding onto this issue to see if there are places we've missed. Based on some testing earlier this year, it's all good! So unfortunately we don't have anything to compare as Task Manager has been running with these optimizations since the beginning of alerting.

It still would have been nice to see the impact :)

I think it was bad enough when using refresh: wait_for that we couldn't run many tasks per minute nor GA anything 🙈

[dgieselaar]

The linked PR in this ticket is tagged as 7.14. How do you reconcile that with:

So unfortunately we don't have anything to compare as Task Manager has been running with these optimizations since the beginning of alerting.

Do you define "beginning of alerting" as GA? Because I cannot see how it is true in any other case. When did alerting go GA?

[mikecote]

There might be a few refresh missing or that have been added over time but there isn't a specific release where we would see significant changes to alerting/task manager performance. Alerting went GA in 7.11.

Could be cool to compare 7.11 to now though if someone had spare cycles.

[dgieselaar]

@mikecote have you looked at APM traces? Or more generally, how have you verified your assumptions are correct? E.g., the PR I put up as a result of me looking into this has 4 changes - out of those 4, only one place actually seems to have added a refresh: false. Those changes were made based on looking at an actual trace and identifying bottlenecks. Maybe that one change is all we can do but this feels hand-wavy to me.