[Security Solution] Architecture design for prebuilt rules customization

[banderror]

Epic: https://github.com/elastic/security-team/issues/1974 (internal)

Summary

Come up with an architecture design for the workflows of:

• customizing prebuilt Elastic rules
• upgrading prebuilt Elastic rules to new versions
• resolving conflicts during the upgrade
• rolling back customized prebuilt rules to their original versions

We need to find answers to some technical questions, including:

• Where do we store original documents of prebuilt rules? What is the data model?
• Where do we store user customizations to prebuilt rules? What is the data model?
• How do we apply customizations to a prebuilt rule?
• How do we roll back a customized rule to its original version?
• How do we compare changes between an original version of a prebuilt rule, its new version from EPR, and the version customized by the user? How do we find conflicts between the user customizations and the new changes from Elastic? How do we allow the user to resolve these conflicts? Something like a git three-way merge?
• Is the current rule versioning sufficient or needs to be reworked?
  • Is it enough to have 1 version which is a simple number?
  • How do we plan to distinguish between content versions and versions of the modifications applied by the user? Do we need a separate version (could call it "revision") for the user modifications?
  • Could the current content versioning system in https://github.com/elastic/detection-rules interfere in a bad way with the prebuilt rules customization and upgrade workflows?
  • [Security Solution] Side effects of no-content version bumps in prebuilt rules #130576
• Do we need to modify any of our existing APIs (contracts, implementation)?
• Do we need to introduce any new APIs?
• Can we split the implementation into stages to iterate?
• Can we hide the implementation under a feature switch to avoid long-living branches?
• Do we need any support for these workflows from Alerting Framework or Kibana Core, now or in the future?
• How would the design support rules sharing between spaces in the future?

We'd need a proposal (e.g. an architecture design document) providing answers to those questions and clarifying how it's all going to work. We should start opening tickets and/or RFCs where applicable for individual parts of the big problem.

Architecture Design

Architecture Design Document, status: done.

Follow-up work

Here are some tickets that you can track to follow the progress on implementing prebuilt rules customization:

Design:

• [Security Solution] PoC of Prebuilt Detection Rules package with historical versions #137420
• [Security Solution] PoC of the rule upgrade workflow #137446
• [Security Solution] Discuss rule fields in the context of the prebuilt rule upgrade workflow #147239

Removing filesystem rules:

• [Security Solution] Remove filesystem detection rules in favor of a bundled Fleet package #139926
• [Security Solution] Support testing of prerelease detection rules with Kibana #147466

Rule versions and revisions:

• [Security Solution] Separate rule versions and revisions #136213
• [ResponseOps] Add revision field support to Alerting framework rules #137164
• [Security Solution] Remove logic that bumps rule version on every change #137168
• [Security Solution] Remove version column from the Rules table #148196
• [Security Solution] Support extended semver versions in prebuilt rules #146316

Fixes for Fleet:

• [Fleet] Packages with a large number of saved objects in them cause Kibana to crash #147695
• [Fleet] Remove the mapping for the nested istalled_kibana field #148174
• [Fleet] Error when importing more than 10000 saved objects in a Fleet package #148175

Fleet package with historical versions of prebuilt rules (support in the current application):

• [Security Solution] Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules #148176
• [Security Solution] Add support for Fleet package with historical versions of prebuilt rules #148179

API for prebuilt rule upgrade and installation workflows:

• [Security Solution] Introduce an endpoint to use for the detection rules table filters/data  #137428
• [Security Solution] Add an experimental feature flag for the new prebuilt rule upgrade and installation workflows #148181
• [Security Solution] Implement prebuilt rule status API endpoint #148182
• [Security Solution] Implement prebuilt rule upgrade/_review API endpoint #148183
• [Security Solution] Implement prebuilt rule upgrade/_perform API endpoint #148184
• [Security Solution] Implement prebuilt rule installation/_review API endpoint #148185
• [Security Solution] Implement prebuilt rule installation/_perform API endpoint #148186
• [Security Solution] Implement overall rule diff algorithm #148189
• [Security Solution] Assign proper diff algorithms to all rule fields #148191

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[mbudge]

We have been waiting for this for a long time, as the team find it difficult to identify new pre-built rules to enabled then keep their clones in sync with regular updates pushed out by elastic.

Some of the customisation's we make are

• Change rule name to add to custom rule ID prefix
• Change index patterns so the rules work with cross-cluster search / remote data clusters
• Add tags
• Alter schedule and look-back time
• Apply custom filtering
• Set timestamp override to use event.ingested
• Set Actions for when an alert is raised

[banderror]

@mbudge Thank you so much for your feedback and for listing the customizations you usually make.

Could you give an example of this one: Change rule name to add to custom rule ID prefix?

[twanva]

Hi,

I'm also interested in this feature, any updates?

[banderror]

Hey @twanva, our team is working on it. At this point, you can look for updates in the related GH issues which you can find in the "Proposal" section of this issue's description. Do you have any specific questions or thoughts?

[banderror]

The architecture is done, for everyone curious: please read the Architecture Design Document and the updated description of this issue. We are working on the implementation and hoping to ship this feature in one of the future releases! 🚢