Entity definition use case

[smith]

Currently the Elastic Entity Model has hard-coded entity definitions (https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability_solution/entity_manager/docs/entity_definitions.md)

We would like to have packages include entity definitions (for instance, a kubernetes package could contain entity definitions for k8s.pod, k8s.node, etc.)

While everything related to entities currently happens in Kibana/ES, it would also be possible to do entity discovery in the otel collector, in which case we would want to distribute the entity definition assets to the collectors.

[jsoriano]

How do entity definitions look like? Is it enough with a JSON like the one linked below, or JS code is also needed?
https://github.com/elastic/kibana/blob/a8aa215db5b26b1dcdb049231dad2ae787167048/x-pack/plugins/observability_solution/entity_manager/server/lib/entities/built_in/services.ts#L22

[smith]

@jsoriano that's the right place. It's just JSON, and a REST API.

There's an open PR with the OpenAPI docs: elastic/kibana#190203 that's probably the best place to look now. You can clone that branch and look at the docs locally or just read the JSDoc comments in the TS server route definitions.

So the UI for defining entities at this point is creating json files and posting them to a Kibana API.

I'm not sure where docs for the entities schema are, but you can probably figure it out from https://github.com/elastic/kibana/blob/main/x-pack/packages/kbn-entities-schema/src/schema/entity_definition.ts.

[klacabane]

As far as implementation goes, we should be able to copy the SLO implementation with a few minor tweaks since they share the same use case. This means supporting creation of a <package>/kibana/entity_definition folder with json files

[klacabane]

@jsoriano since entity definitions are tech preview atm and we expect a few (breaking) changes to be introduced fairly quickly, is there a way to specify that in the package-spec (ie anything under kibana/entity_definition folder is not production ready) ? Something similar to the release property that one can add to a package root configuration, or a data stream ?

Also if I look at the SLO package-spec addition, SLOs are added as saved object format - is that a requirement or just a convenient way to handle the blob in fleet (since everything else under kibana/.. is a saved object) ? Is there anything preventing us from dumping a custom json schema as long as we handle it correctly in fleet ?

[jsoriano]

is there a way to specify that in the package-spec (ie anything under kibana/entity_definition folder is not production ready) ? Something similar to the release property that one can add to a package root configuration, or a data stream ?

Yes, there are two possible approaches, one is to use release: beta in the spec of the directory, as we do with content packages in

package-spec/spec/content/spec.yml

Line 14 in d9888bd

release: beta

The other one is to add the feature to a future version, as the features we have reserved for 3.3 and 3.4:

package-spec/spec/changelog.yml

Line 5 in d9888bd

- version: 3.4.0-next

These are controlled with JSON patches like these ones:

package-spec/spec/integration/kibana/spec.yml

Line 140 in d9888bd

- before: 3.3.0

These declarations prevent the publication of GA packages that use these features, but don't prevent the features to be used in prerelease packages.

Also if I look at the SLO package-spec addition, SLOs are added as saved object format - is that a requirement or just a convenient way to handle the blob in fleet (since everything else under kibana/.. is a saved object) ? Is there anything preventing us from dumping a custom json schema as long as we handle it correctly in fleet ?

This was convenient for the SLO case but is not a requirement, there is nothing preventing other approaches. In any case changes in Kibana might be needed.