Saved-objects as data

[kobelb]

Starting in 8.0 with the formal implementation of system-indices, it will no longer be possible to use applications like Discover, Visualize, Canvas, Maps, Machine Learning, etc. to analyze saved-objects. While it's technically possible to create visualizations, etc. against any of Kibana's system indices prior to 8.0, it is not supported. Kibana's system indices, and the documents that are stored in these indices, are an implementation detail of Kibana. As such, these indices and the documents that are stored in them are subject to breaking changes in minors. Additionally, Kibana enforces its own entity model and role-based access control. Accessing Kibana's system indices using Elasticsearch APIs side-steps all of these protections.

However, this is a rather unfortunate situation because applications like Discover, Visualize, Lens etc. offer incredibly powerful tools for free-form analysis. This has become a recurring sticking point when discussing how to architect certain features because this free-form analysis is so compelling.

While we've found some interesting short-term solutions, like the one that we're using for Alerts as data, it doesn't fulfill all of our needs. An ideal solution would allow us to perform free-form analysis using tools like Discover, Visualize, Lens, etc. against saved-objects while respecting Kibana's entity model and role-based access control.

[mikecote]

The @elastic/kibana-alerting-services will eventually have to provide a platform solution while respecting our RBAC model and based itself on the alerts as data story (#93728). If we were to use saved objects for this approach, below are some mentions I've seen brought up when discussing the schema (#93728).

• Ability to have an ECS schema ([RAC] Alerts as Data Schema Definition #93728 (comment))
  • If this remains true, we would need to ensure the public contract of this saved object can work with such a structure. (ex: building visualizations on event.kind instead of attributes.event.kind)
• Support for ILM (thread: [RAC] Alerts as Data Schema Definition #93728 (comment))
  • There is an expectation of using ILM to delete old alert documents
• Index per rule type
  • I've highly recommended the alert indices are created per rule type (index threshold, Geo threshold, etc.). Each has different parameter mappings that users may want to filter, sort and free-text search on ([RAC] Alerts as Data Schema Definition #93728 (comment)).
  • If saved objects support an index per SO type, we would also be able to revisit making rule parameters searchable (Make alert params searchable #50213 (comment)).
• Migrations (thread: [RAC] Alerts as Data Schema Definition #93728 (comment))
  • We'll need to create a new index for each release version that includes the latest mappings. The current assumption is additive mappings. The discussions around storing workflow fields make me re-think this.
  • I'm not sure what the saved object migrations should do with existing documents. The existing documents are left in older indices and queried via index patterns across all these indices in the current implementation. It is assumed these indices will have a large (potentially huge) number of documents.

If we do create an index per saved object type, one thought would be to swap attributes for saved_object and allow custom fields at the root while storing saved object specifics within a nested object.

Example:

Before	After
type	saved_object.type
namespcae	saved_object.namespace
rerefences	saved_object.references
attributes.ruleType	ruleType
attributes.params	params

mappings before

{
  "type": { "type": "keyword" },
  "namespace": { "type": "keyword" },
  "references": {
    "type": "nested",
    "properties": {
      "type": { "type": "keyword" },
      "id": { "type": "keyword" },
      "name": { "type": "keyword" }
    }
  },
  "attributes": {
    "properties": {
      "ruleType": { "type": "keyword" },
      "params": {
        "properties": {
          "foo": { "type": "keyword" }
        }
      }
    }
  }
}

mappings after

{
  "saved_object": {
    "type": { "type": "keyword" },
    "namespace": { "type": "keyword" },
    "references": {
      "type": "nested",
      "properties": {
        "type": { "type": "keyword" },
        "id": { "type": "keyword" },
        "name": { "type": "keyword" }
      }
    },
  },
  "ruleType": { "type": "keyword" },
  "params": {
    "properties": {
      "foo": { "type": "keyword" }
    }
  }
}

This allows saved object properties to be stored at the root (ex: ECS fields) while still having a place to store saved object specific fields.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[oatkiller]

Heya @kobelb. I've got a question/suggestion that's unlikely to be practical, but your answer will help me to get some extra context on the issue. You said

As such, these indices and the documents that are stored in them are subject to breaking changes in minors. Additionally, Kibana enforces its own entity model and role-based access control.

What if we totally broke those expectations and instead did this:

• SavedObjects must maintain backwards compatibility during minors
• All role-based access control must be modeled in Elasticsearch

Would that help to solve the issue? What would be the costs/downsides of such an approach? Thanks for humoring me.

[pgayvallet]

SavedObjects must maintain backwards compatibility during minors

After 8.0, there will no longer be minor/major distinctions, at least regarding breaking changes, meaning that if we do want to declare that the SO model/schema (and other Kibana system indices) is no longer an implement detail, we'll be forced to wait an undetermined amount of time before being able to effectively change it. Not saying this is not necessarily an option, but we need to be aware of that.

All role-based access control must be modeled in Elasticsearch

@elastic/kibana-security would be the best to answer that, but from my understanding, if this is an option we definitely want to dig, we're pretty far from it right now. Also, RBAC is not the only 'filtering' performed for SO queries, we also got spaces for instance, and we're also performing some additional validation (on aggregations for example) due to some ES api/options being too permissive.

Overall, I think the best approach currently to answer 'SO as data' would be the Kibana Persistence ToolKit we presented during GAH: #102177

[pgayvallet]

@kobelb is this still something, or can we potentially close this?

[kobelb]

@kobelb is this still something, or can we potentially close this?

I think we should leave this one open, solely because I've heard it mentioned recently regarding alerting rules in regard to RAG.