[RAC][Rule Registry] Updating old alerts by "reindexing" them into the current write index

[banderror]

Parent ticket: #101016

Summary

Background: #109276 (comment)

We should consider a "hybrid" approach in which an "update" means we write a document to the current write index (which we can assume to have the most recent mappings) and we delete the document from the index it was previously in. That would be almost like a reindex-on-write approach.

Benefits:

• In Observability old alerts can be "recovered" in a rule executor, which leads to their timestamps and many other properties being updated. In this case, it probably doesn't make a lot of sense to continue keeping them in the old indices.
• A recovered alert would move "closer" in the index (e.g. potentially move from the warm to the hot tier).
• If we update all types of alerts this way, it will allow us to only maintain compatibility of old indices with read operations, but not write/update operations. Which would simplify our lives when dealing with [RAC][Rule Registry] Write an RFC for index bootstrapping and backwards compatibility #110800 and [RAC][Rule Registry] Applying aliases/runtime fields to old indices for backwards compatibility #110808.

Risks/cons:

• We don't have a transaction to make this operation atomic, so we might have intermittently missing or duplicate alert docs.
• In Security we don't update @timestamp and other fields except the status, and the benefits are not that obvious for Security.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

cc @weltenwort

[banderror]

Hey everyone, I removed this ticket from the backlog of the Detection Rules area.

We (@elastic/security-detections-response-rules) are not the owners anymore (however feel free to still ping us if you have any tech questions about the ticket). Ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area (Team:Detection Alerts label). Please ping @peluja1012 and @marshallmain if you have any questions.