[Alerting] Add more rule execution context

[dgieselaar]

Closes #113506.

The following changes were made:

• When rule execution starts, update the transaction name: Executing alerting rule
• When rule execution starts, update the transaction labels with the rule id
• When the rule is loaded, update the transaction name by appending the rule name
• When the rule is loaded, update the transaction labels with the consumer, rule type id, rule name, rule params and rule tags
• When the rule has executed, set the appropriate outcome
• When the rule has executed, update the transaction labels with the number of active, recovered and new instances

[cyrille-leclerc]

The added label and the new transaction name will be very helpful!

[cyrille-leclerc]

@dgieselaar Can you please remind us how we identify as spans the different rule actions: email, index, server-log, ServiceNow-itsm, webhook, pagerduty...

For example,

• For remote systems, it will be helpful to track the destination/URL to get the span identified as a remote call (visualized on the service map, surface the destination as an uninstrumented backend...)
• For the write operation in Elasticsearch, it would be interesting to have these actions visualized as generic Elasticsearch access spans
  See example of span for CI steps in Jenkins: https://github.com/jenkinsci/opentelemetry-plugin/blob/4706034b0e8a2c7de88ecfe03326a6f2d9c6fef7/src/main/java/io/jenkins/plugins/opentelemetry/job/AbstractGitStepHandler.java#L60

[dgieselaar]

@cyrille-leclerc unfortunately the trace waterfall is broken, @cauemarcondes is working on a fix, I'll update the PR with a screenshot for actions if that's fixed before this PR lands. I'm not sure if I get your second point though, can you elaborate?

[cyrille-leclerc]

Thanks @dgieselaar

I'm not sure if I get your second point though, can you elaborate?

For the rule actions (email, index, server-log, servicenow-itsm, webhook, pagerduty...), it would be great to capture span labels that characterize the execution like the URL invoked, the authentication username ... in order to slice and dice traces in any dimension and also enable visualization of the destination on the service map and as an uninstrumented backend.

Here is an example that has a lot of commonalities with the labels we collect on CI/CD pipelines steps like the pipeline checkout step:

Labels
labels.ci_pipeline_run_user	SYSTEM
labels.git_branch	master
labels.git_clone_depth	0
labels.git_clone_shallow	FALSE
labels.git_repository	cyrille-leclerc/my-war
labels.git_username	cyrille-leclerc
labels.host_ip	192.168.1.46
labels.host_name	cyrillerclaptop.localdomain
labels.jenkins_computer_name	#controller#
labels.jenkins_pipeline_step_id	7
labels.jenkins_pipeline_step_name	Check out from version control
labels.jenkins_pipeline_step_plugin_name	workflow-scm-step
labels.jenkins_pipeline_step_plugin_version	2.13
labels.jenkins_pipeline_step_type	checkout
labels.jenkins_url	http://localhost:9600/
labels.service_namespace	jenkins
Trace
trace.id	dd7cc4c4220d72dc85f2a6de6b0c0d69
Span
span.id	a6c33c39e2e3141a
Service
service.name	jenkins

[dgieselaar]

For the rule actions (email, index, server-log, servicenow-itsm, webhook, pagerduty...), it would be great to capture span labels that characterize the execution like the URL invoked, the authentication username ... in order to slice and dice traces in any dimension and also enable visualization of the destination on the service map and as an uninstrumented backend.

Hmm, I don't want to inadvertently leak sensitive data, I'm not sure how to prevent that if we for instance stringify params or config. Any thoughts here @elastic/kibana-alerting-services?

[cyrille-leclerc]

I don't want to inadvertently leak sensitive data,

Good catch @dgieselaar , we sanitized a bunch of attributes in the Jenkins Otel integration, typically parsing URLs and reconstructing them to ensure they don't leak credentials.

Here is an example:
https://github.com/jenkinsci/opentelemetry-plugin/blob/opentelemetry-0.21/src/main/java/io/jenkins/plugins/opentelemetry/job/AbstractGitStepHandler.java#L133-L153

[dgieselaar]

@elasticmachine merge upstream

[xcrzx]

@dgieselaar Is it possible to view latency distribution for all rules of a specific rule type with these changes? Let's say I'm investigating performance issues with siem.queryRule. I used to do that in the following way:

1. Select all transactions of the siem.queryRule rule type
2. On the latency distribution diagram, drag mouse to select slowest 10%
3. Examine their spans to identify performance bottlenecks

But with this PR, I cannot select all query rules in a single view, as transactions now split but rule name. So of I have 100+ activated rules, it becomes tedious to examine them one by one. Or I'm missing something?

[dgieselaar]

@dgieselaar Is it possible to view latency distribution for all rules of a specific rule type with these changes?

Not in the APM app. You can use e.g. Lens to gather that data, but it won't allow you to inspect a trace without manually copying trace ids.

Usually we separate transaction groups if they have different performance characteristics, which I would expect to be the case here from rule instance to rule instance.

[dgieselaar]

@elasticmachine merge upstream

[cyrille-leclerc]

Usually we separate transaction groups if they have different performance characteristics, which I would expect to be the case here from rule instance to rule instance.

From my observations, performance characteristics are homogeneous for Security rules of the same rule type. That's why I think grouping by rule type would be helpful.

Would it make sense to group rules by rule type by default as before and leave the ability to filter by rule name (it is already possible, labels.alerting_rule_name : "Rule Name") for those who need that level of granularity?

That's a very good point: for guided rules, the execution path and the performance characteristics are likely to be very homogeneous and thus it could make sense to have the same transaction group.
This would be different for generic rules that can have completely different performance characteristics and probably different execution path.
For this reason of homogeneity of "guided rules", it could make sense to use the rule type name as the name of the transaction. I'm not sure.

[dgieselaar]

For this reason of homogeneity of "guided rules", it could make sense to use the rule type name as the name of the transaction. I'm not sure.

I'm not sure how to make that distinction (between guided and generic rules) from the alerting framework's perspective. It is something that the security rule types can set themselves in the rule executor. Maybe that's a good compromise?

[cyrille-leclerc]

I'm not sure how to make that distinction (between guided and generic rules) from the alerting framework's perspective. It is something that the security rule types can set themselves in the rule executor. Maybe that's a good compromise?

That could be an interesting starting point

[xcrzx]

I'm not sure how to make that distinction (between guided and generic rules) from the alerting framework's perspective. It is something that the security rule types can set themselves in the rule executor. Maybe that's a good compromise?

Yeah, it looks like we can use apm.setTransactionName to overwrite transaction names in Security to whatever value we want. Well, that sounds ok with me then.

[dgieselaar]

@elasticmachine merge upstream

[dgieselaar]

@elasticmachine merge upstream

[ymao1]

LGTM! Left some nits on naming but looks great otherwise. Thanks!

[ymao1]

Following our new terminology, I think this should be actions_connector_id and actions_connector_type_id

[dgieselaar]

Thanks, updated the labels w/ new terminology!

[ymao1]

Following our updated terminology, I believe this should be alerting_new_alerts, alerting_active_alerts, alerting_recovered_alerts

addressed

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: edf303d
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #8258 succeeded cb46022
• 💚 Build #8003 succeeded d9d0700
• 💚 Build #6926 succeeded 04a0f6d
• 💚 Build #6716 succeeded 9c51b78
• 💚 Build #6099 succeeded 51901da
• 💚 Build #4956 succeeded b99d0c2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	8.0
✅	7.16

The backport PRs will be merged automatically after passing CI.