[Security Solution] Handle missing base versions during rule upgrade

[xcrzx]

Resolves: #152836
Resolves: #148183
Resolves partially: #148189

Summary

Updates the rule status (prebuilt_rules/status) and upgrade review (prebuilt_rules/upgrade/_review) APIs to continue working when the base rule version is unavailable, for example, when a detection rules package is released without historical versions.

The logic applied when calculating rule diff is as follows:

  if (baseVersion === MissingVersion) {
    /**
     * We couldn't find the base version of the rule in the package, so further
     * version comparison is not possible. We assume that the rule is not
     * customized, and the value can be updated if there's an update.
     */
    return isEqual(currentVersion, targetVersion)
      ? ThreeWayDiffOutcome.StockValueNoUpdate
      : ThreeWayDiffOutcome.StockValueCanUpdate;
  }

The above logic has considerable implications. When a historical version of a rule is unavailable, we assume that the rule had no modifications. Therefore, in cases where it has been modified, the upgrade review endpoint would still return "no conflicts detected", and users risk losing all their edits.

{
  name: {
    current_version: "My Modified Version Test 123",
    target_version: "Attempt to Modify an Okta Policy Rule",
    merged_version: "Attempt to Modify an Okta Policy Rule",
    diff_outcome: "BASE=A, CURRENT=A, TARGET=B",
    merge_outcome: "TARGET",
    has_update: true,
    has_conflict: false,
  },
}

If we had the base versions, the result would be different:

{
  name: {
    current_version: "My Modified Version Test 123",
    target_version: "Attempt to Modify an Okta Policy Rule",
    merged_version: "My Modified Version Test 123",
    diff_outcome: "BASE=A, CURRENT=B, TARGET=A",
    merge_outcome: "CURRENT",
    has_update: false,
    has_conflict: false,
  },
}

Minor changes

This PR adds the RuleUpgradeStatsForReview.tags implementation.

Additionally, this PR removes some unused API fields:

• PrebuiltRulesStatusStats.num_prebuilt_rules_total
• RuleUpgradeStatsForReview.num_rules_to_upgrade_not_customized
• RuleUpgradeStatsForReview.num_rules_to_upgrade_customized
• RuleUpgradeStatsForReview.fields

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[spong]

nit: maybe extract these invariants to a common function.

[xcrzx]

I tried to, but Typescript started complaining about the rule type mismatch. Unfortunately, I haven't found a clean way to fix that, so I left the checks as is.

[spong]

WRT

The above logic has considerable implications. When a historical version of a rule is unavailable, we assume that the rule had no modifications. Therefore, in cases where it has been modified, the upgrade review endpoint would still return "no conflicts detected", and users risk losing all their edits.

In these instances, I'm curious if we should have some UI/UX to help mitigate the issue in the event it actually happens. Whether that's encouraging the user to do a quick export of existing rules, a system-triggered temporary auto-backup of the rules in question, or something along those lines? Depending on how the storage of historical rules goes, perhaps this is uncommon enough to not even be necessary?

[xcrzx]

Given that users could lose all their modifications without the ability to restore them, I think it makes sense to implement some safeguards on the client side. It's not a big concern, though, as long as we do not allow them to customize prebuilt rules. So, maybe we should wait and see how the adoption of historical rules goes. If we see a substantial number of packages without historical versions or not all versions available for every rule, then we'll need to implement some workarounds in the UI.

[spong]

Code changes LGTM! 👍 Thanks @xcrzx 🙂

Left a couple nits and a discussion around losing user data when missing historical versions, but no changes required.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 8d9a34f

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	432	434	+2

Total ESLint disabled count

id	before	after	diff
securitySolution	512	514	+2

History

• 💛 Build #120701 was flaky b42bc1a320dbd40fe9cdf4291e8156c8262cc2b4
• 💛 Build #120366 was flaky 409217e76ac49f51737de70f683aef7f6e0513d4
• 💛 Build #120264 was flaky 263fe0d3e48a8bbb06a460b01cc89b72c07d7199
• 💚 Build #118375 succeeded 062ba3cb0a67cce90b047e8e4628b6328d109d94

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx