[RAC][Rule Registry] Write an RFC for index bootstrapping and backwards compatibility

[banderror]

Parent ticket: #101016
Related to: #109293

Summary

Document the use case and desired behavior around index bootstrapping enhancements and solution for backwards compatibility in an RFC. Topics to cover:

• Index upgrade logic. When and how should we update existing concrete indices VS rolling over / creating new ones. Tradeoff between oversharding due to aggressive rollover vs the risk of updating mappings in place. It should not play against the ideas for backwards compatibility.
• Solution for backwards compatibility. In which cases and how would we apply field aliases and runtime fields to the old concrete indices? At which stage would this be happening? What would be the algorithm and how would the API look like? Would it be enough for compatibility-on-read or we'd need to transform the alerts to the current schema in the code in some cases? Do we need compatibility-on-write (e.g. transformations in the code to old schemas)?
• How versioning fits into these solutions?
• How should we implement update logic for existing alerts (potentially old documents with an out-of-date schema)?

Coordinate with the Elasticsearch team. We'd like the ES team's input on possible new ES features to support our use case and thoughts on the tradeoffs.

Background

The background for this is our discussions with @kobelb (see #109276 (comment) and above comments) on the "compatibility" of the current index upgrade logic with the ideas for backwards compatibility (#109293).

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

Hey everyone, I removed this ticket from the backlog of the Detection Rules area.

We (@elastic/security-detections-response-rules) are not the owners anymore (however feel free to still ping us if you have any tech questions about the ticket). Ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area (Team:Detection Alerts label). Please ping @peluja1012 and @marshallmain if you have any questions.