Research approach for pre-defined rule IDs when making rules share-capable #100667
Comments
mikecote
added
Feature:Alerting
Team:ResponseOps
|
Pinging @elastic/kibana-alerting-services (Team:Alerting Services) |
|
cc @spong |
gmmorris
added
the
Feature:Alerting/RulesFramework
gmmorris
added
the
loe:needs-research
|
Marking as blocked until the research in #100067 confirms this is still necessary and with what requirements. |
|
Detection rules assign a Example security solution rule SO:This is also stored as a tag with the format Currently, it looks like There is an open security issue for investigating updating We need to investigate the implications of making alerting rules share capable with and without #99741. Should we even be doing #99741 if making rules share-capable will re-generate all the ids in 8.0? |
|
Rules with pre-defined IDs can be created one of two ways:
Reviewing the internal usages of both, there are currently no plugins that are creating rules with a pre-determined rule id. This leave potential users that are using the API to create rules with a pre-determined id. Testing with the POC for making rules share-capable shows that no additional changes will be required for these rules on top of the changes already needed to get alerting and actions working with share-capable SOs Steps to Test
Also tested this when you have a rule with predefined id in two different spaces. The rules are each migrated to a new rule with a re-generated ID but using the previously predefined id resolves correctly. |
|
With #99741, security solutions is investigating the possibility of using a migration to use the static rule id as the rule saved object id. Assuming they are successful with this migration, what would be the implications of changing the Current, security rules assign a separate rule_id field.
Example original pre-packaged security rule as a saved object (relevant fields only) If #99741 is successfully implemented, the This would be the point where we could include security rules in SO rule export and security could remove their custom import/export But then, when |
|
In the previous comment, I assumed that the proposed security solutions migration and the migration to share-capable I then tried combining the migrations, so that they would be done all as part of the 8.0.0 migration. This means converting the The output of this was that the |
I also tried loading pre-built rules into 2 different non-default spaces and migrating as described and post-migration, only one set of rules remained. However, only one of the name spaces was preserved with the rule, meaning where previously I had rule with ids |
@ymao1 the migration in your comment attempts to change the ID of the alert itself. Doing that in consumer-defined migration functions is not supported, and it will break any inbound references and otherwise result in unwanted collisions like the one you described. Instead have you considered changing the "static rule ID" to match the object ID? E.g., function updateIdForSecurityRules(
doc: SavedObjectUnsanitizedDoc<RawAlert>
): SavedObjectUnsanitizedDoc<RawAlert> {
const { id, attributes } = doc;
const { tags = [], params } = attributes;
const ruleStaticIdIndex = tags.findIndex((tag: string) => tag.startsWith('__internal_rule_id:'));
const newTags = [...tags];
const newParams = { ...params };
if (ruleStaticIdIndex > -1) {
// In case this object's ID has changed, synchronize the static rule ID fields with the object ID
newTags[ruleStaticIdIndex] = `__internal_rule_id:${id}`;
newParams.ruleId = id;
}
return {
...doc,
attributes: {
...doc.attributes,
tags: newTags,
params: newParams,
},
};
}Edit: note that this would mean all objects that existed before the migration will also exist after the migration. So if you have 2 rules before the migration, you will have 2 rules after the migration. |
|
Thanks for weighing in @jportner! Good to know that this would not be officially supported. For security rules, the static id for prebuilt detection rules are the same across all Kibana instances anywhere. Migrating so that the static ID becomes the object ID would break this contract, so I don't think this is an option. It sounds like they should wait until 8.x (after |
|
Ah, thanks for clarifying. I read the linked issue now. Yes, they also mentioned preventing exports for those rules instead, which may be a better option. |
|
Okay after some offline discussions, this seems to be the situation:
The only guaranteed way of creating "at most one copy of an object" in a space, both pre- and post-8.0, is to use the We also refer to this as "pseudo-copy mode", as you aren't creating a new copy of an object each time you import, you are effectively ensuring that exactly one copy exists in the destination space. Under the hood, this relies on a special Perhaps instead of relying on However, this doesn't change the fact that, at the moment, rules' object IDs are completely divorced from their type-level "internal rule IDs". function updateOriginIdForSecurityRules(
doc: SavedObjectUnsanitizedDoc<RawAlert>
): SavedObjectUnsanitizedDoc<RawAlert> {
return {
...doc,
originId: doc.attributes.params.ruleId ?? doc.originId
};
}Just to reiterate: the |
That's correct. For further context; the rule create API allows users to provide a custom ID to support capabilities like singleton rules (per space). An approach like this is preferred over lookup / create for situations we encounter race conditions (ex: two Kibanas creating a rule of ID “1” on startup and only wanting one to be created). This brings a few problems when making rule SOs share-capable:
We are leaning on maybe deprecating this capability and removing it altogether in 8.0. The use cases for pre-defined IDs sound more like needing a programmatical way to configure rules (ex: via kibana.yml) and pre-defined IDs doesn't seem to have much adoption as of today. |
|
Thanks @mikecote and @jportner. It looks like we would be able to remove the ability to specify pre-defined rule ID on create while still allowing a path forward for security rules to align with SO import/export using If security were able to add a migration to set @jportner I looked at using |
That's correct, it's not really intended to be exposed for other consumers, but the Spaces plugin actually uses it under the hood for copy-to-space functionality. The Spaces plugin simply creates a readable stream from the object array, then passes the stream to kibana/x-pack/plugins/spaces/server/lib/copy_to_spaces/copy_to_spaces.ts Lines 68 to 100 in b2d36b8 |
|
After some research, we have decided to deprecate support for pre-defined rule ids via the HTTP API in 8.0. Previously, with Issues:
Finally, we've created an issue to discuss the best way to continue supporting pre-defined IDs in the rules client: |
|
Closing this research issue in favor of the followup issues linked in the previous comment. |
gmmorris
added
the
estimate:needs-research
gmmorris
removed
the
loe:needs-research
Research issue for #100067 regarding pre-defined rule IDs.
In 8.0, we plan to run a migration that removes the namespace from the saved-object serialized ID. To do so, new rule and connector IDs will be generated. With this issue, we should research the impacts of rules that rely on pre-defined IDs, how the migration will work when there are conflicts, and how de-duplication will be handled with pre-packaged rules. If necessary, work with the @elastic/kibana-security team if we need anything further from them to support this.
The text was updated successfully, but these errors were encountered: