[EBT] Add level to event base telemetry

[Dosant]

Describe the feature:

Since we plan to also use new event based telemetry for tracking performance via a new single user journey framework, I would like to propose and discuss introducing a notion of "levels" to ebt events. Similar like we have levels in logs (info warning debug etc) we could have a set of levels for ebt events.

Describe a specific use case for the feature:

With the help of levels in EBT we could capture more events and metadata that we want to use for synthetic performance tests, without worrying about these excessive events getting into a telemetry cluster with real usage data.
For example, in AppService we would like to capture every search and every expression execution to create metrics on top of those in single user journey performance tests, but we doubt all those events would be useful in the real telemetry data.

These levels might also be useful for support purposes, where if needed users could increase telemetry details level.

Alternatives

We could probably work around this in a single user journey test by adding a custom plugin in a test and reporting some specific performance-only events from a test plugin. This way these events will be generated only in synthetic performance runs, but not in real clusters.

Related #121992

cc @afharo @lizozom @spalger

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[afharo]

I think this is a great idea! However, while I'm excited that EBT is used to report performance metrics, I think we should provide a new API to help developers in Kibana with tracking the performance metrics:

IMO, this API would provide performance-tracking-specific features like ensuring a standard format, providing common timing APIs, or filtering the level of verbosity depending on the deployment. It would then report the tracked transactions/spans via EBT/APM/others as we see fit.

What do you think?

[lizozom]

In the call where this came up, and like @Dosant mentioned, we could use a custom plugin for testing and reporting any metrics an easy workaround in case we want to report metrics (doesn't have to be performance related) that we don't want in production.

Since we're in the early stages of using EBT, could we defer this discussion to a later stage?

[pgayvallet]

@afharo is this something we may still want / need?

[afharo]

I'll leave that call to @Dosant. My comment here recommended against implementing it at the client level.