[POC] Event-based Telemetry #95960

afharo · 2021-03-31T16:36:12Z

Summary

This PR implements a POC for generally available Event-based telemetry APIs.

The reasoning for adding this new set of APIs will be further explained in a subsequent RFC. The TL;DR reason: there are some use cases where we need more granular data as opposed to the current snapshot-aggregated telemetry.

The goal of this PR is to showcase a Proof-of-Concept of how this API will work while maintaining the main constraints we want to enforce:

Low impact on our users: we should use as minimum resources as possible so we don’t affect the performance of the product nor the cost of maintenance.
Trust: Kibana is connected to Elasticsearch that holds customer’s data. We want our users to be confident that we are not collecting any of that information or any other PII.
Transparency: building on trust, we want to share with our users the data that we collect about them if they request it.

To ensure those constraints this PR implements:

A set of Byte-Sized in-memory queues that limit the amount of data each plugin can send [Low impact on resources]:
- Plugins are assigned an allowance of 1MB each to be split across all the channels they register.
- Whenever a channel enqueues a new event, if its queue is already full, the eldest document will be removed to accommodate the new one.
- When registering the channels, plugins can optionally enforce a quota percentage to a specific channel. This is to help developers to fine-tune the drop-rate they're willing to accept in specific channels in favour of others.
- When registering the channels, plugins must provide a schema that will be audited to ensure no PII-related data is collected [Trust]. The schema is used to validate the incoming events, so we'll discard anything that doesn't match.
Extends core's APIs to provide the new PluginScopedAPIs. This prepends the caller's plugin name as an argument to the callee API so plugins cannot impersonate others to gain access to a larger allowance.
A Leaky Bucket to send the events at a maximum constant rate:
- If enough data is enqueued, it defaults to 10kb payloads sent every 10s.
- Otherwise, it sends any remainder that has been in the queues for longer than 1h.
- It round-robins through all the plugins, and then through the channels to make sure that one plugin doesn't take all the bandwidth. i.e.: Plugin A-Channel 1 of 2, Plugin B-Channel 1 of 1, Plugin C-Channel 1 of 3, Plugin A-Channel 2 of 2, Plugin B-Channel 1 of 1, Plugin C-Channel 2 of 3, ...
Sets the telemetry logger to silent by default in production ([Telemetry] Custom logger appender? #89839), and allows users to explicitly enable on demand by setting the telemetry.logging config according to the logging settings. This will log the events for users to be aware of what we collect about them [Transparency]
Adds 2 example plugins to demonstrate all these capabilities.

TODOs

Document the new Core API
Update telemetry README file
Update telemetry settings asciidoc to explain about the new logging features

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

afharo

Self-review

afharo · 2021-04-01T11:30:08Z

src/plugins/telemetry/server/event_based_telemetry/byte_sized_queue.ts

+   * @private
+   */
+  private ensureQueueSize() {
+    while (this.queue.length > 1 && this.size > this.maxByteSize) {


Since this is the only place that we check this.maxByteSize, we might be able to dynamically update the value once we implement the custom disabling of queues. i.e.: If a plugin registers 3 channels but we disable 2 of them, the remainder channel can get all the plugin's allowance for itself instead of the initial 33%.

afharo · 2021-04-01T11:31:53Z

src/plugins/telemetry/server/event_based_telemetry/byte_sized_queue.ts

+    this.queue.push(buffer);
+    this.ensureQueueSize();
+  }
+


NIT: a public clear method would be useful for:

Clearing queues when opted-out

Clearing queues when a channel is disabled

afharo · 2021-04-01T16:57:07Z

src/plugins/telemetry/common/schema/schema_to_io_ts.ts

+ * Side Public License, v 1.
+ */
+
+import * as t from 'io-ts';


Using io-ts as the validation tool because it's orders of magnitude more performant than joi/@kbn/config-schema

afharo · 2021-04-01T16:58:22Z

src/plugins/telemetry/common/schema/types.ts

+ */
+export type TelemetrySchemaValue =
+  | {
+      type: AllowedSchemaTypes | 'pass_through';


Initially intentionally allowing pass_through for specific use cases where we want a bit of a weaker validation.

afharo · 2021-04-01T16:58:38Z

src/plugins/telemetry/common/schema/types.ts

+  | {
+      type: AllowedSchemaTypes | 'pass_through';
+      _meta: {
+        description: string; // Intentionally enforcing the descriptions here


Description will be mandatory on the leaves

src/plugins/telemetry/server/event_based_telemetry/event_based_telemetry_service.ts

src/plugins/telemetry/server/event_based_telemetry/leaky_bucket.ts

afharo · 2021-04-01T17:10:54Z

src/plugins/telemetry/server/event_based_telemetry/leaky_bucket.ts

+
+  private isFullQueue(): boolean {
+    const queueSize = this.queue.reduce((acc, buffer) => acc + buffer.length, 0);
+    return queueSize >= this.config.threshold.getValueInBytes();


NIT: Possible, performance improvement: precalculate this.config.threshold.getValueInBytes() into a private variable to avoid calling it on every loop.

afharo · 2021-04-01T17:12:02Z

src/plugins/telemetry/server/event_based_telemetry/leaky_bucket.ts

+  private hasMaxWaitExpired(): boolean {
+    const diff = moment().diff(this.lastSend, 'milliseconds');
+    return diff >= this.maxWaitInMs;
+  }


NIT: Maybe Date.now() is more performant? We don't call it that often though to be a problem.

src/plugins/telemetry/server/event_based_telemetry/leaky_bucket.ts

kibanamachine · 2021-04-06T14:12:41Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 1d99487
Storybooks Preview
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

Kibana Pipeline / general / Jest Integration Tests.src/core/server/logging/integration_tests.RollingFileAppender `size-limit` policy with `numeric` strategy rolls the log file in the correct order

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toEqual(expected) // deep equality

- Expected  - 1
+ Received  + 0

  Array [
    "kibana.1.log",
-   "kibana.2.log",
    "kibana.log",
  ]
    at Object.<anonymous> (/dev/shm/workspace/parallel/4/kibana/src/core/server/logging/integration_tests/rolling_file_appender.test.ts:103:28)

Kibana Pipeline / general / Jest Integration Tests.src/core/server/logging/integration_tests.RollingFileAppender `size-limit` policy with `numeric` strategy only keep the correct number of files

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toEqual(expected) // deep equality

- Expected  - 1
+ Received  + 0

  Array [
    "kibana-1.log",
-   "kibana-2.log",
    "kibana.log",
  ]
    at Object.<anonymous> (/dev/shm/workspace/parallel/4/kibana/src/core/server/logging/integration_tests/rolling_file_appender.test.ts:150:28)

Metrics [docs]

✅ unchanged

History

💔 Build #116877 failed b90fdec
💔 Build #116590 failed c38d4962d09cfbb66b9ab48cc4601bc82e410cf6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

afharo · 2022-04-20T15:17:34Z

Closing as this POC is no longer relevant.

afharo added the release_note:skip Skip the PR/issue when compiling release notes label Mar 31, 2021

afharo commented Apr 1, 2021

View reviewed changes

afharo added 5 commits April 1, 2021 19:25

Plugin-Scoped APIs

a564667

Event-based telemetry

c5522e0

Use the APIs in the Security Solutions plugin

e6b57c0

Alphabetically sorted plugins in pie

4685f63

Fix some self-review notes

b90fdec

afharo force-pushed the event-based-telemetry branch from 43e34fd to b90fdec Compare April 1, 2021 17:29

afharo added 2 commits April 6, 2021 15:35

Fix NITs for demo

2fe6c0c

Examples README

1d99487

afharo mentioned this pull request Apr 22, 2021

[Usage Collection/schema] Add _meta.isOptional #98000

Closed

This was referenced Jan 18, 2022

[Telemetry] Custom logger appender? #89839

Open

[Osquery] Add telemetry for packs and saved queries #122501

Merged

afharo mentioned this pull request Feb 15, 2022

[EBT][Server] V3 in-house shipper #125693

Closed

afharo closed this Apr 20, 2022

afharo deleted the event-based-telemetry branch October 1, 2024 08:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[POC] Event-based Telemetry #95960

[POC] Event-based Telemetry #95960

afharo commented Mar 31, 2021 •

edited

Loading

afharo left a comment

afharo Apr 1, 2021

afharo Apr 1, 2021

afharo Apr 1, 2021

afharo Apr 1, 2021

afharo Apr 1, 2021

afharo Apr 1, 2021

afharo Apr 1, 2021

kibanamachine commented Apr 6, 2021 •

edited

Loading

Standard Out

Stack Trace

Standard Out

Stack Trace

afharo commented Apr 20, 2022

[POC] Event-based Telemetry #95960

[POC] Event-based Telemetry #95960

Conversation

afharo commented Mar 31, 2021 • edited Loading

Summary

TODOs

Checklist

For maintainers

afharo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Apr 6, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Standard Out

Stack Trace

Standard Out

Stack Trace

Metrics [docs]

History

afharo commented Apr 20, 2022

afharo commented Mar 31, 2021 •

edited

Loading

kibanamachine commented Apr 6, 2021 •

edited

Loading