Show Stack management rules as part of the Observability rules and alerts tables

[katrin-freihofner]

Describe the feature:
The Observability rules and alerts tables should display Stack management rules and alerts produced by these rules. This also means that users should be able to create these rules from the Observability rules view.

Rules we will need to add (in order of priority)

• Elasticsearch query rule done (Epic)
• Anomaly detection alert done (PR)
• Anomaly detection jobs health -> use case should be reevaluated
• Transform health -> use case should be reevaluated
• Tracking containment -> use case should be reevaluated
• Index threshold -> will be replaced by new, combined rule.

Open questions

• Do these rules fit in one of the existing groups in the list? (vs. it being in it's own group)

Describe a specific use case for the feature:
The Elasticsearch query rule can be used for Observability use cases, for example, logs.

[elasticmachine]

Pinging @elastic/actionable-observability (Team: Actionable Observability)

[fkanout]

@katrin-freihofner one thing is left, please comment here .

[katrin-freihofner]

Thanks @fkanout I added it to the open questions.

[chrisdistasio]

@emma-raffenne @vinaychandrasekhar who can help assess whether there is something here that needs to be addressed?

[vinaychandrasekhar]

@jasonrhodes

[jasonrhodes]

We will look into this.

[jasonrhodes]

@maryam-saeidi can you and @vinaychandrasekhar speak with @shanisagiv1 to decide if any of the rules in this description are still in need of any changes? If so, we can create relevant issues, then either way we can close this one.

[maryam-saeidi]

Actions:

• Creating a list with all the rules. (Columns: visible in the stateful new design?, visible in serverless?)
  • Adding this information to this document. @maryam-saeidi Added here.
• Index threshold:
  • keeping it in the stack management and not moving it to observability. (Custom threshold rule can be used instead.)
  • Creating a banner + documentation to suggest using the custom threshold rule instead. (For observability use cases) We need to create a ticket for this + asking designer's input. @shanisagiv1
• Eventually, with the new stateful design, we will remove the observability rules from the stack management rule list. (tracked by @shanisagiv1 )
• Transform health, Anomaly detection jobs health
  • New stateful design: Keep under stack rules. (No need to move it to observability)
  • Serverless: We don't show it in serverless for now.
• Tracking containment
  • Is this related to the security use case? @shanisagiv1

cc @jasonrhodes @heespi @kobelb

[sophiec20]

Please share why Transform Health and Anomaly Detection health rules are being excluded from Serverless -- both AD jobs and transforms exist in serverless, and can be created in Serverless during job creation. Why are their associated health rules being excluded?

[vinaychandrasekhar]

@sophiec20 - are all the inputs/options in both those rules applicable to Serverless users? Or are some of those options more applicable to Stateful deployments (and/or applicable to our SREs managing Serverless deployments on customers' behalf)? Thanks

[peteharverson]

Note that you can still create Anomaly Detection health rules and Transform health rules in an Observability Serverless project, by going to the Rules page from Stack Management via a search in the top search bar. The version of the rules flyout you get from this page gives you the health rules too:

In contrast to the list you see if you go via Alerts on the Observability project left side nav:

[sophiec20]

Inputs/options in both those health rules are applicable to Serverless users and ESS/on-prem users.

@vinaychandrasekhar I appreciate this needs a deeper discussion, so let's meet to ensure we are making the right engineering choices. There is a lot of alignment here with the One Discover vision, so it is timely.

[vinaychandrasekhar]

Discussed with @peteharverson and @darnautov.

• Both AD Health and Transforms Health rules, and all their inputs/options, are applicable to and actionable in both Serverless and Stateful.
• Both rules, and their corresponding alerts, will (continue to) be surfaced in the Stack area/nav and not within the Observability solution area. This applies to both Serverless and Stateful.
• In Serverless, we will need the Stack Rules and Alerts views pages. I will check with @shanisagiv1 and @mwtyang on this.

cc @sophiec20

[jasonrhodes]

In Serverless, we will need the Stack Rules and Alerts views pages. I will check with @shanisagiv1 and @mwtyang on this.

@vinaychandrasekhar any update on this?

[jasonrhodes]

From @vinaychandrasekhar on Tues, 30 April: "Hi Jason, no updates. I'm discussing with Shani. And might have to chat with Sophie / Platform leadership on plans. Will keep you posted"