[BUG] Rollups - SearchRequest missing cancelAfterTimeInterval parameter

[petardz]

What is the bug?
In RollupRunner we don't set cancelAfterTimeInterval parameter on SearchRequest object. Consequence of this is that global cluster setting search.cancel_after_time_interval will override it, which if set too low, could impact rollup run success rate.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Set cluster setting search.cancel_after_time_interval to something very low, like 1s
2. Create rollup job on index with some number of documents/shards
3. See Cancellation timeout error (TaskCancelledExceptioncancelled exception)

What is the expected behavior?
Rollup job should have full control of this timeout.

[rishabhmaurya]

Are you proposing its resolution to be individual plugin's responsibility to set the search.cancel_after_time_interval with max(cluster setting search.cancel_after_time_interval, plugin's default minimum value say 5s) ?

[petardz]

@rishabhmaurya If I understood correctly, cluster setting search.cancel_after_time_interval will be used only if same param in SearchRequest is null?

If that's the case, then plugin can override this setting by setting a cancelAfterTimeInterval param in every SearchRequest before calling _search. For rollups and transforms I'd say to give it at least 10m.

[rishabhmaurya]

relates to - opensearch-project/alerting#827

[Joshua152]

Can I be assigned to this issue?