[Security Solution] Cannot enable more than 1000 detection rules at once

[banderror]

Summary

User reports:

I just tried to enable all of the rules in a test space and this error popped up saying that I can't enable more than 1000 rules at once. We are now up to 1233 built in detection rules in Elastic.

This restriction was added to the Alerting Framework about 5 months ago in #179778 and went live in v8.15.0. Confirmed to be wrong by @cnasikas, the restriction was always there on the Framework side, but started to be applied to detection rules only after we migrated to the Framework's bulk enable/disable methods a few months ago.

We need to figure out how we could improve the UX:

• Can we consider increasing or removing this limit from the Framework?
  • @cnasikas: "Removing the limit should not be an option. We can analyze the performance and the impact increasing the limit has on the route."
• Can we enable rules in batches of 1000 on the Security side, or that would cause issues on the Framework and Task Manager side for large numbers of rules?
  • @cnasikas: "I would avoid it as it can stress the framework and the TM as you said. Maybe a POC of this approach could reveal some insights."
• Should we handle it in our UI so the user understands that it's not possible to enable more than 1000 rules at once?
  • @cnasikas: "A 1K limit is pretty big and letting users do it in batches themselves seems like a reasonable approach and my preference from the other options."
  • This option looks like the best way forward.

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

Hi @elastic/response-ops-ram @JiaweiWu @cnasikas, do you know why in #179778 we introduced this limit of max 1000 rules to enable at once?

[cnasikas]

Hi @banderror! It was always like this from the beginning. The PR did not introduce the limit but versioned the route. The limit is there for performance reasons, to protect the stability and resilience of Kibana.

Can we consider increasing or removing this limit from the Framework?

Removing the limit should not be an option. We can analyze the performance and the impact increasing the limit has on the route.

Can we enable rules in batches of 1000 on the Security side, or that would cause issues on the Framework and Task Manager side for large numbers of rules?

I would avoid it as it can stress the framework and the TM as you said. Maybe a POC of this approach could reveal some insights.

Should we handle it in our UI so the user understands that it's not possible to enable more than 1000 rules at once?

A 1K limit is pretty big and letting users do it in batches themselves seems like a reasonable approach and my preference from the other options.

[banderror]

@cnasikas Thank you for the feedback, that all sounds reasonable 👍

We'll explore this path:

A 1K limit is pretty big and letting users do it in batches themselves seems like a reasonable approach and my preference from the other options.

[banderror]

@approksiu @ARWNightingale I assigned you so you could track this issue it on your side. It's low impact and low priority because there's a simple workaround.

In the future, it would be great to explore ways to change the Rule Management table's UI in a way that would prevent the user from trying to enable more than 1000 rules at once.