[Security Solution][Detections] Migrate to using Saved Object References array to reference alert saved objects from security saved objects

[ymao1]

Currently, there are several security saved object types that directly store the associated rule (alert) SO ID instead of utilizing the references array. With alerting planning to make alerts share-capable in 8.0.0, the SO IDs for these rules will be re-generated, which will break the association in these security saved objects.

Detection Engine Rule Status SO

Stores the associated alert SO ID in the alertId field

Example `siem-detection-engine-rule-status` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "siem-detection-engine-rule-status:ab8f4fc0-e8b6-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"siem-detection-engine-rule-status": {
			"alertId": "a9f99490-e8b6-11eb-bc58-13b6ae6d011a",
			"statusDate": "2021-07-19T17:34:38.755Z",
			"status": "partial failure",
			"lastFailureAt": null,
			"lastSuccessAt": "2021-07-19T17:34:38.755Z",
			"lastFailureMessage": null,
			"lastSuccessMessage": """The following indices are missing the timestamp field "@timestamp ": [".kibana_8 .0 .0_001 "]""",
			"gap": null,
			"bulkCreateTimeDurations": [],
			"searchAfterTimeDurations": [],
			"lastLookBackDate": null
		},
		"type": "siem-detection-engine-rule-status",
		"references": [],
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:34:38.757Z"
	}
}

When the alert SO is made share-capable, no statuses will be returned for the rules in the detection UI. However, maybe this is ok? There doesn't seem to be any errors in the UI and since I think only the last few statuses are shown, user would just have to wait for the rule to run again with the new ID to start seeing statuses.

Detection Engine Rule Action SO

Stores the associated alert SO ID in the ruleAlertId field

Example `siem-detection-engine-rule-action` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "siem-detection-engine-rule-actions:ab321490-e8b6-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"siem-detection-engine-rule-actions": {
			"ruleAlertId": "a9f99490-e8b6-11eb-bc58-13b6ae6d011a",
			"actions": [{
				"action_type_id": ".email",
				"id": "8fd8e910-e8a4-11eb-bc58-13b6ae6d011a",
				"params": {
					"subject": "hi",
					"to": [
						"[email protected]"
					],
					"message": "Rule {{context.rule.name}} generated {{state.signals_count}} alerts"
				},
				"group": "default"
			}],
			"ruleThrottle": "rule",
			"alertThrottle": null
		},
		"type": "siem-detection-engine-rule-actions",
		"references": [],
		"migrationVersion": {
			"siem-detection-engine-rule-actions": "7.11.2"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:37:28.027Z"
	}
}

When the alert SO is made share-capable, the association between siem-detection-engine-rule-actions SO and its detection rule is lost, so it looks like no notification actions would be scheduled anymore for detection rules. When retrieving the rule through the security UI or API, the actions array always returns as empty.

Recommended approach

The recommended approach is to update the siem-detection-engine-rule-status and siem-detection-engine-rule-actions saved objects to use the references array. This requires a code update to extracted the associated alert SO ID on create/update from the document into the references array and to inject the ID on find/get. This would also require a migration to update existing SOs of this type to use the references array. Once this is done, the saved object service will takes care of updating the references when alert SOs are made share-capable.

Alternative approach

Alternatively, the security team could explore using the .resolve() functionality provided by the saved object service in order to resolve the outdated alert SO ID to the re-generated ID. This would require tracking down all the places where the siem-detection-engine-rule-status and siem-detection-engine-rule-actions saved objects are used to retrieve the associated alert SO and injecting the resolve layer to ensure the correct SO is loaded. This seems like a more brittle approach.

When does this need to be done?

These changes should be completed by 7.16 FF as the saved object type conversions will be done in 8.0

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

Detection Engine Rule Status SO
When the alert SO is made share-capable, no statuses will be returned for the rules in the detection UI. However, maybe this is ok? There doesn't seem to be any errors in the UI and since I think only the last few statuses are shown, user would just have to wait for the rule to run again with the new ID to start seeing statuses.

Yes, this shouldn't be a big issue. These custom SOs will be replaced by documents in event_log (see #101013), and migrations/fallbacks to the SOs are not required (#101013 (comment)).

However, I think this can still be a problem for execution events in event_log. They will still need to reference rule SOs via their ids. Or it's gonna be handled transparently by event_log? What do you think @ymao1 ?

Anyway, the issue with Detection Engine Rule Status SOs will be tracked as part of #101013

Thank you for writing up this ticket! It's very-very helpful.

[ymao1]

@banderror Yes! We are doing the research for the event log association in this issue: #100666

[banderror]

siem-detection-engine-rule-status migration is addressed in these 3 PRs:

• [Security Solution] Migrates siem-detection-engine-rule-status alertId to saved object references array #114585
• [Security Solution][Detections] Fix a bug in siem-detection-engine-rule-status Saved Object migration to SO references #115355
• [Security Solution][Detections] Enable writing rule execution events to Event Log by default #115394

[yctercero]

Addressed in 7.16: https://github.com/elastic/security-team/issues/1656