[EBT] Add internal shipper that users can enable/disable on demand to audit the telemetry we collect.

[afharo]

At Elastic we are open about the telemetry we collect, and we want to keep it this way for Event-based telemetry. For that reason, we need to implement a way of letting users see the data that we report through this new collection mechanism.

We can't simply generate a snapshot and print it to the user because in EBT we don't persist the data locally. We need a new approach to this.

Send all or selected events?

We can offer the user the possibility to audit all the events or a subset of them if they are only interested in some of them.

Where to ship the events?

IMO, we can approach this in 2 ways:

1. Logging events

When the user enables this feature, they'll see all the selected events in the logs. Users can use the logging configuration to log to a separate file if needed, and even use filebeat to ship it to their cluster or an external one.

2. Write the events to Elasticsearch

When the user enables this feature, Kibana will ship all the selected events to an index of choice (how do we handle authentication?).

Additional considerations

In either implementation, browser-generated events will need to be sent to the server. In the Logging solution it may result in unsorted events as server events may make it faster to the logs than the ones coming from the browser through HTTP requests.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

From grooming:

1. Logging is the preferred approach
2. telemetry.audit.log_events: ['*'] <- Do not throw if it's not an actual event. Just validate it's a list of strings.
3. ☝️ with the config above, register 2 shippers (server and browser) that will listen to all events (change the client to avoid the exclusive* config in this audit case)
   1. Browser will batch and send to the server via a new HTTP route (that logs the received events).
   2. Server will log straight away.
4. Where to log? Expose telemetry.audit.logger config so users can select the destination (console/custom file). Use https://github.com/elastic/kibana/pull/95960/files#diff-367973a209b434cf45640c1824fb18c4521eba4cfc7120ed0b63bb419a66afe8R175-R196 as inspiration

Split this task into 2:

1. Shippers implementation (M)
2. Logging configuration (S)

[afharo]

We can achieve a similar behaviour by leveraging the log entries coming from

kibana/packages/analytics/client/src/analytics_client/analytics_client.ts

Lines 133 to 137 in dd43c2d

	if (this.initContext.isDev) {
	this.initContext.logger.debug<EventDebugLogMeta>(`Report event "${eventType}"`, {
	ebt_event: event,
	});
	}

We would only need to:

1. Remove the if statement in that block
2. Add more context to the logger for easier testing (i.e.: analytics.events instead of the current analytics), so only events are reported that way.

Then, we can document that adding the following to kibana.yml would log them.

logging:
  appenders:
    json-layout:
      type: console
      layout:
        type: json
  loggers:
    - name: analytics.events
      level: all
      appenders: [json-layout]

IMO doing this will reduce a lot of our LOE and still achieve the same results. What do you think?

[pgayvallet]

We can achieve a similar behaviour by leveraging the log entries

What about client-side events? The logs approach will only work for server-side usages of EBT I believe?

[afharo]

@pgayvallet right! That is a good point! Posibly, if we ever tackle the browser-side core logging service and any sort of redirection to the server, we'd get this same expected behaviour. But, for now, that's not the case.