Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Usage Collection] Supports field descriptions as meta data #92701

Merged
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
094b2d4
WIP debugging not writing meta description fields
TinaHeiligers Feb 21, 2021
754fc00
Merge branch 'master' into usage-collection/supports-descriptions
TinaHeiligers Feb 22, 2021
8ae620e
Adds telemetry_tools manage_schema test to ensure descriptions are wr…
TinaHeiligers Feb 22, 2021
dda1dc8
Adds collector with a description field fixture, updates telemetry_to…
TinaHeiligers Feb 22, 2021
3e75092
Merge branch 'master' into usage-collection/supports-descriptions
TinaHeiligers Feb 23, 2021
b1c8b9b
Merge branch 'master' into usage-collection/supports-descriptions
TinaHeiligers Feb 24, 2021
1d95958
Checks that a collector support _meta.description
TinaHeiligers Feb 24, 2021
6c16a7f
Updates schema for localization collector
TinaHeiligers Feb 24, 2021
e22bd14
Updates test
TinaHeiligers Feb 24, 2021
25757a7
Adds description support to README
TinaHeiligers Feb 24, 2021
d400be6
Merge branch 'master' into usage-collection/supports-descriptions
TinaHeiligers Feb 25, 2021
c40e790
adds descriptions to core_usage_collector savedObjects apiCalls
TinaHeiligers Feb 25, 2021
aecce17
Adds descriptions to the remaining core_usage_collector fields
TinaHeiligers Feb 25, 2021
4a0e224
Adds descriptions to the csp collector fields
TinaHeiligers Feb 25, 2021
ab01797
Adds schema and descriptions for ops_stats usage collector
TinaHeiligers Feb 25, 2021
a826e9b
Deletes unused interface
TinaHeiligers Feb 25, 2021
bfd3cc0
Adds descriptions to ui_counters fields
TinaHeiligers Feb 26, 2021
0a25a4d
Updates telemetry schemas
TinaHeiligers Feb 26, 2021
dc62bcd
Merge branch 'master' into usage-collection/supports-descriptions
TinaHeiligers Feb 26, 2021
c48bb50
debugging collector integrity
TinaHeiligers Feb 26, 2021
093f6cc
fixes incorrect type
TinaHeiligers Feb 26, 2021
2f93064
Dirty support for _meta in stackManagementSchema type
TinaHeiligers Feb 26, 2021
4dfb2a9
Moves schema to main collector registration file, deletes schema file…
TinaHeiligers Feb 26, 2021
7ccfb91
Adds descriptions to kibana saved objects count collector
TinaHeiligers Feb 26, 2021
46ffb45
Adds descriptions to application_usage fields
TinaHeiligers Feb 26, 2021
3772d8c
Adds uiSettings schema back to stack_management_usage folder
TinaHeiligers Feb 26, 2021
d0e4a69
Adds descriptions to ui_metric collector
TinaHeiligers Feb 26, 2021
482c674
Updates oss_plugins telemetry schema file
TinaHeiligers Feb 26, 2021
8c67ebd
Updates management collector README
TinaHeiligers Feb 27, 2021
e6ba5b3
Merge branch 'master' into usage-collection/supports-descriptions
kibanamachine Mar 1, 2021
0d6b1e5
Changes descriptions for application_usage collector fields
TinaHeiligers Mar 1, 2021
dbafa7c
Addresses minor text changes
TinaHeiligers Mar 1, 2021
8a09021
Reverts changes to ops_stats_collector
TinaHeiligers Mar 1, 2021
0fd72ab
application_usage README fix
TinaHeiligers Mar 1, 2021
15c0631
Moves descriptions to telemetry_management_collector schema
TinaHeiligers Mar 1, 2021
f9e6566
Reverts changes to README
TinaHeiligers Mar 1, 2021
82cf340
Merge branch 'master' into usage-collection/supports-descriptions
TinaHeiligers Mar 1, 2021
6a0e7d1
Merge branch 'master' into usage-collection/supports-descriptions
TinaHeiligers Mar 2, 2021
14b1bbb
Resolves conflict
TinaHeiligers Mar 2, 2021
0eb72c3
Update src/plugins/kibana_usage_collection/server/collectors/manageme…
TinaHeiligers Mar 2, 2021
aee674a
Update src/plugins/kibana_usage_collection/server/collectors/manageme…
TinaHeiligers Mar 2, 2021
3b23636
Update README.mdx
TinaHeiligers Mar 2, 2021
9c1437c
Updates telemetry schema
TinaHeiligers Mar 2, 2021
827cbfc
Updates descriptions in management collector schema
TinaHeiligers Mar 2, 2021
97def09
Updates oss_plugin collectors schemas
TinaHeiligers Mar 2, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import { parsedIndexedInterfaceWithNoMatchingSchema } from './parsed_indexed_int
import { parsedNestedCollector } from './parsed_nested_collector';
import { parsedSchemaDefinedWithSpreadsCollector } from './parsed_schema_defined_with_spreads_collector';
import { parsedWorkingCollector } from './parsed_working_collector';
import { parsedCollectorWithDescription } from './parsed_working_collector_with_description';
import { ParsedUsageCollection } from '../ts_parser';

export const allExtractedCollectors: ParsedUsageCollection[] = [
Expand All @@ -22,5 +23,6 @@ export const allExtractedCollectors: ParsedUsageCollection[] = [
parsedIndexedInterfaceWithNoMatchingSchema,
parsedNestedCollector,
parsedSchemaDefinedWithSpreadsCollector,
parsedCollectorWithDescription,
parsedWorkingCollector,
];
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"properties": {
"my_working_collector_with_description": {
"properties": {
"flat": {
"type": "keyword",
"_meta": {
"description": "A flat keyword string"
}
},
"my_index_signature_prop": {
"properties": {
"avg": {
"type": "float"
},
"count": {
"type": "long"
},
"max": {
"type": "long"
},
"min": {
"type": "long"
}
}
},
"my_str": {
"type": "text"
},
"my_objects": {
"properties": {
"total": {
"type": "long"
},
"type": {
"type": "boolean"
}
}
},
"my_array": {
"type": "array",
"items": {
"properties": {
"total": {
"type": "long"
},
"type": {
"type": "boolean"
}
}
}
},
"my_str_array": { "type": "array", "items": { "type": "keyword" } }
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { SyntaxKind } from 'typescript';
import { ParsedUsageCollection } from '../ts_parser';

export const parsedCollectorWithDescription: ParsedUsageCollection = [
'src/fixtures/telemetry_collectors/working_collector_with_description.ts',
{
collectorName: 'my_working_collector_with_description',
schema: {
value: {
flat: {
type: 'keyword',
_meta: {
description: 'A flat keyword string',
},
},
my_str: {
type: 'text',
},
my_index_signature_prop: {
avg: {
type: 'float',
},
count: {
type: 'long',
},
max: {
type: 'long',
},
min: {
type: 'long',
},
},
my_objects: {
total: {
type: 'long',
},
type: {
type: 'boolean',
},
},
my_array: {
type: 'array',
items: {
total: {
type: 'long',
},
type: { type: 'boolean' },
},
},
my_str_array: { type: 'array', items: { type: 'keyword' } },
},
},
fetch: {
typeName: 'Usage',
typeDescriptor: {
flat: {
kind: SyntaxKind.StringKeyword,
type: 'StringKeyword',
},
my_str: {
kind: SyntaxKind.StringKeyword,
type: 'StringKeyword',
},
my_index_signature_prop: {
'@@INDEX@@': {
kind: SyntaxKind.NumberKeyword,
type: 'NumberKeyword',
},
},
my_objects: {
total: {
kind: SyntaxKind.NumberKeyword,
type: 'NumberKeyword',
},
type: {
kind: SyntaxKind.BooleanKeyword,
type: 'BooleanKeyword',
},
},
my_array: {
items: {
total: {
kind: SyntaxKind.NumberKeyword,
type: 'NumberKeyword',
},
type: {
kind: SyntaxKind.BooleanKeyword,
type: 'BooleanKeyword',
},
},
},
my_str_array: {
items: {
kind: SyntaxKind.StringKeyword,
type: 'StringKeyword',
},
},
},
},
},
];
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ describe('extractCollectors', () => {
const programPaths = await getProgramPaths(configs[0]);

const results = [...extractCollectors(programPaths, tsConfig)];
expect(results).toHaveLength(8);
expect(results).toHaveLength(9);
expect(results).toStrictEqual(allExtractedCollectors);
});
});
6 changes: 6 additions & 0 deletions packages/kbn-telemetry-tools/src/tools/manage_schema.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import { generateMapping } from './manage_schema';
import { parsedWorkingCollector } from './__fixture__/parsed_working_collector';
import { parsedCollectorWithDescription } from './__fixture__/parsed_working_collector_with_description';
import * as path from 'path';
import { readFile } from 'fs';
import { promisify } from 'util';
Expand All @@ -25,4 +26,9 @@ describe('generateMapping', () => {
const result = generateMapping([parsedWorkingCollector]);
expect(result).toEqual(mockSchema);
});
it('generates a mapping file that includes _meta.description fields', async () => {
const mockSchema = await parseJsonFile('mock_schema_with_descriptions.json');
const result = generateMapping([parsedCollectorWithDescription]);
expect(result).toEqual(mockSchema);
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { CollectorSet } from '../../plugins/usage_collection/server/collector';
import { loggerMock } from '../../core/server/logging/logger.mock';

const { makeUsageCollector } = new CollectorSet({
logger: loggerMock.create(),
maximumWaitTimeForAllCollectorsInS: 0,
});

interface MyObject {
total: number;
type: boolean;
}

interface Usage {
flat?: string;
my_str?: string;
my_objects: MyObject;
my_array?: MyObject[];
my_str_array?: string[];
my_index_signature_prop?: {
[key: string]: number;
};
}

const SOME_NUMBER: number = 123;

export const myCollector = makeUsageCollector<Usage>({
type: 'my_working_collector_with_description',
isReady: () => true,
fetch() {
const testString = '123';
// query ES and get some data

// summarize the data into a model
// return the modeled object that includes whatever you want to track
try {
return {
flat: 'hello',
my_str: testString,
my_objects: {
total: SOME_NUMBER,
type: true,
},
my_array: [
{
total: SOME_NUMBER,
type: true,
},
],
my_str_array: ['hello', 'world'],
};
} catch (err) {
return {
my_objects: {
total: 0,
type: true,
},
};
}
},
schema: {
flat: {
type: 'keyword',
_meta: {
description: 'A flat keyword string',
},
},
my_str: {
type: 'text',
},
my_objects: {
total: {
type: 'long',
},
type: { type: 'boolean' },
},
my_array: {
type: 'array',
items: {
total: {
type: 'long',
},
type: { type: 'boolean' },
},
},
my_str_array: { type: 'array', items: { type: 'keyword' } },
my_index_signature_prop: {
count: { type: 'long' },
avg: { type: 'float' },
max: { type: 'long' },
min: { type: 'long' },
},
},
});
17 changes: 12 additions & 5 deletions src/plugins/kibana_usage_collection/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,16 @@
This plugin registers the basic usage collectors from Kibana:

- [Application Usage](./server/collectors/application_usage/README.md)
- UI Metrics
- Ops stats
- Number of Saved Objects per type
- [User-changed UI Settings](./server/collectors/management/README.md)
- CSP configuration
- Core Metrics
- CSP configuration
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the order of the collectors to match the order with have in the plugin code.

- Kibana: Number of Saved Objects per type
- Localization data
- [User-changed UI Settings](./server/collectors/management/README.md)
- Ops stats
- UI Counts
- UI Metrics





Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ To track a sub view inside your application (ie a flyout, a tab, form step, etc)

#### For a React Component

For tracking an application view rendered using react the simplest way is to wrap your component with the `TrackApplicationView` Higher order component:
For tracking an application view rendered using react the simplest way is to wrap your component with the `TrackApplicationView` Higher order component exposed from the `usageCollection` plugin. You will need to add the plugin to plugin's `kibana.json` file as an item in the `optionalPlugins` and `requiredBundles` declarations:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the phrases/sentences to describe what we're showing in the code snippet.


kibana.json
```json
Expand All @@ -26,6 +26,7 @@ kibana.json

At the application level, the application must be wrapped by the `ApplicationUsageTrackingProvider` provided in the `usageCollection`'s setup contract.

For example:
```typescript jsx
class MyPlugin implements Plugin {
...
Expand Down Expand Up @@ -69,7 +70,6 @@ const MyTrackedComponent = () => {
Application Usage will automatically track the active minutes on screen and clicks for both the application and the `MyComponent` component whenever it is mounted on the screen. Application Usage pauses counting screen minutes whenever the user is tabbed to another browser window.

The prop `viewId` is used as a unique identifier for your plugin. The Application Id is automatically attached to the tracked usage, based on the ID used when registering your app via `core.application.register`.

## Application Usage Telemetry Data

This collector reports the number of general clicks and minutes on screen for each registered application in Kibana.
Expand Down Expand Up @@ -126,4 +126,4 @@ In order to keep the count of the events, this collector uses 3 Saved Objects:

All the types use the shared fields `appId: 'keyword'`, `viewId: 'keyword'`, `numberOfClicks: 'long'` and `minutesOnScreen: 'float'`, but they are currently not added in the mappings because we don't use them for search purposes, and we need to be thoughtful with the number of mapped fields in the SavedObjects index ([#43673](https://github.com/elastic/kibana/issues/43673)). `application_usage_transactional` and `application_usage_daily` also store `timestamp: { type: 'date' }`.

Rollups uses `appId` in the savedObject id for the default view. For other views `viewId` is concatenated. This keeps backwards compatiblity with previously stored documents on the clusters without requiring any form of migration.
Rollups uses `appId` in the savedObject id for the default view. For other views `viewId` is concatenated. This keeps backwards compatiblity with previously stored documents on the clusters without requiring any form of migration.
Loading