[Telemetry] Server side fetcher #50015

Bamieh · 2019-11-04T16:08:06Z

Details:

This PR mainly adds server side fetcher. A few refactoring + features where needed to fully accommodate this change.

Adds server fetcher feature (Closes [Telemetry] server side fetcher #49711)
Allows telemetry to be fetched from the server instead of the browser if this kibana.yml config is set:
```
 telemetry.usageFetcher: server
```
Currently this config accepts server and browser; in the future we might be adding a beat option.

This telemetry config, like optIn can be overriden via saved objects. The reason behind this is to allow configuring this without restaring kibana.

Adds telemetry for telemetry (Closes Add telemetry for telemetry #50013) This currently reports:

 export interface TelemetryUsageStats {
   opt_in_status?: boolean | null;
   usage_fetcher?: 'browser' | 'server';
   last_reported?: number;
 }

Added user-agent tracking under ui metrics:

 "kibana-user_agent": [
   {
     "key": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36",
     "value": 1
   }
 ]

tests for all telemetry configs
ui metric use kfetch instead of $http to hide the loading bar when sending ui metrics usage to kibana server and to solve https://github.com/elastic/kibana/pull/49671 (CC @justinkambic @epixa ).
refactor configs overriding in the telemetry saved object by adding a repository file to get/update saved objetcts. (Helps resolving *yml* based setting to ship telemetry (and disable the UI to opt out) #49751 (comment))

Notes

Added server fetcher feature to send telemetry from kibana server instead of browser under the flag telemetry.usageFetcher: server.

Introduced a new telemetry collector for the telemetry plugin which includes opt-in status, last report, and the type of fetcher the user is using.

Introduced user-agent tracking in ui-metrics to prevent losing this information if telemetry is sent from the server.

Replaced $http fetcher with kfetch as it $http shows a loader whenever we are sending telemetry from the browser. Finally Added a repository feature in telemetry to allow overriding configs via saved objects.

elasticmachine · 2019-11-04T17:45:10Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 27d6b67

elasticmachine · 2019-11-04T18:26:57Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 0037d75

elasticmachine · 2019-11-06T13:21:45Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 4436f3f

…ver_fetcher

elasticmachine · 2019-11-06T13:55:55Z

Pinging @elastic/kibana-stack-services (Team:Stack Services)

justinkambic

From my side, this doesn't break the functionality my previous changes to some of this code fixed, so I am OK. I'm not going to approve because I didn't perform a full code review, but I am happy to help in that regard if needed.

Thank you for keeping me in the loop.

elasticmachine · 2019-11-06T15:07:17Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: e0c10b6

Bamieh · 2019-11-12T21:59:40Z

@elasticmachine merge upstream

…nto telemetry/server_fetcher

gmmorris

LGTM, thanks for running through the changes with me

elasticmachine · 2019-11-12T22:54:04Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: c70becc

Bamieh · 2019-11-12T23:01:02Z

@elasticmachine merge upstream

elasticmachine · 2019-11-13T00:16:59Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 3961073

…nto telemetry/server_fetcher

…ver_fetcher

elasticmachine · 2019-11-13T07:15:38Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 6a077d3

* initial push * self code review * ignore node-fetch type * usageFetcher api * user agent metric * telemetry plugin collector * remove extra unused method * remove unused import * type check * fix collections tests * pass kfetch as dep * add ui metrics integration test for user agent * dont start ui metrics when not authenticated * user agent count always 1 * fix broken ui-metric integration tests * try using config.get * avoid fetching configs if sending * type unknown -> string * check if fetcher is causing the issue * disable ui_metric from functional tests * enable ui_metric back again * ignore keyword above 256 * check requesting app first * clean up after all the debugging :) * fix tests * always return 200 for ui metric reporting * remove boom import * logout after removing role/user * undo some changes in tests * inside try catch * prevent potential race conditions in priorities with = * use snake_case for telemetry plugin collection * usageFetcher -> sendUsageFrom * more replacements * remove extra unused route * config() -> config * Update src/legacy/core_plugins/telemetry/index.ts Co-Authored-By: Mike Côté <[email protected]> * Update src/legacy/core_plugins/ui_metric/server/routes/api/ui_metric.ts Co-Authored-By: Mike Côté <[email protected]> * config() -> config * fix SO update logic given the current changes * fix opt in check * triple check * check for non boolean * take into account older settings * import TelemetryOptInProvider * update test case

* upstream/master: Remove internal platform types exports (elastic#50427) [APM] Document `apm_oss.metricsIndices` and `apm_oss.sourcemap… (elastic#50312) [Telemetry] Server side fetcher (elastic#50015) [SIEM] Detection engine placeholders (elastic#50220) [Uptime] Donut chart loader position centered vertically (elastic#50219) update telemetry banner notice text (elastic#50403) Fix aborting when searching without batching (elastic#49966) [Telemetry] Remove telemetry splash page and add conditional messaging (elastic#50189) Revert chromedriver update (elastic#50324) Remove deprecated argument include_type_name from ES calls (elastic#50285) [Maps] add settings to maps telemetry (elastic#50161) remove visualize loader (elastic#46910) Fix misuse of react-router and react-router-dom (elastic#50120)

…-fallback * 'master' of github.com:elastic/kibana: Remove internal platform types exports (elastic#50427) [APM] Document `apm_oss.metricsIndices` and `apm_oss.sourcemap… (elastic#50312) [Telemetry] Server side fetcher (elastic#50015) [SIEM] Detection engine placeholders (elastic#50220) [Uptime] Donut chart loader position centered vertically (elastic#50219) update telemetry banner notice text (elastic#50403) Fix aborting when searching without batching (elastic#49966) [Telemetry] Remove telemetry splash page and add conditional messaging (elastic#50189) Revert chromedriver update (elastic#50324) Remove deprecated argument include_type_name from ES calls (elastic#50285) [Maps] add settings to maps telemetry (elastic#50161) remove visualize loader (elastic#46910) Fix misuse of react-router and react-router-dom (elastic#50120) Move table-list-view to kibana-react (elastic#50046) [ML] Stats bar for data frame analytics (elastic#49464)

…ger-ace-theme * 'master' of github.com:elastic/kibana: (56 commits) [ML] Server info service refactor (elastic#50302) Remove internal platform types exports (elastic#50427) [APM] Document `apm_oss.metricsIndices` and `apm_oss.sourcemap… (elastic#50312) [Telemetry] Server side fetcher (elastic#50015) [SIEM] Detection engine placeholders (elastic#50220) [Uptime] Donut chart loader position centered vertically (elastic#50219) update telemetry banner notice text (elastic#50403) Fix aborting when searching without batching (elastic#49966) [Telemetry] Remove telemetry splash page and add conditional messaging (elastic#50189) Revert chromedriver update (elastic#50324) Remove deprecated argument include_type_name from ES calls (elastic#50285) [Maps] add settings to maps telemetry (elastic#50161) remove visualize loader (elastic#46910) Fix misuse of react-router and react-router-dom (elastic#50120) Move table-list-view to kibana-react (elastic#50046) [ML] Stats bar for data frame analytics (elastic#49464) [ML] Make navigation in tests more stable (elastic#50132) Migrate authorization subsystem to the new platform. (elastic#46145) Bugfix: Interpreter conversion of string to number should throw on NaN elastic#27788 (elastic#50063) Update dependency @elastic/charts to v14 (elastic#49947) ...

* initial push * self code review * ignore node-fetch type * usageFetcher api * user agent metric * telemetry plugin collector * remove extra unused method * remove unused import * type check * fix collections tests * pass kfetch as dep * add ui metrics integration test for user agent * dont start ui metrics when not authenticated * user agent count always 1 * fix broken ui-metric integration tests * try using config.get * avoid fetching configs if sending * type unknown -> string * check if fetcher is causing the issue * disable ui_metric from functional tests * enable ui_metric back again * ignore keyword above 256 * check requesting app first * clean up after all the debugging :) * fix tests * always return 200 for ui metric reporting * remove boom import * logout after removing role/user * undo some changes in tests * inside try catch * prevent potential race conditions in priorities with = * use snake_case for telemetry plugin collection * usageFetcher -> sendUsageFrom * more replacements * remove extra unused route * config() -> config * Update src/legacy/core_plugins/telemetry/index.ts Co-Authored-By: Mike Côté <[email protected]> * Update src/legacy/core_plugins/ui_metric/server/routes/api/ui_metric.ts Co-Authored-By: Mike Côté <[email protected]> * config() -> config * fix SO update logic given the current changes * fix opt in check * triple check * check for non boolean * take into account older settings * import TelemetryOptInProvider * update test case

tsullivan · 2019-11-13T20:30:48Z

src/legacy/core_plugins/ui_metric/public/services/telemetry_analytics.ts

@@ -39,28 +41,36 @@ export const createUiStatsReporter = (appName: string) => (
  }
 };

+export const trackUserAgent = (appName: string) => {
+  if (telemetryReporter) {
+    return telemetryReporter.reportUserAgent(appName);


This popped up from someone's dev environment:

As a note, this was caused by someone not running bootstrap

Bamieh added 4 commits November 4, 2019 18:02

initial push

4860e53

self code review

d7f7f60

ignore node-fetch type

27d6b67

usageFetcher api

0037d75

user agent metric

4436f3f

Bamieh marked this pull request as ready for review November 6, 2019 12:12

Bamieh requested review from a team and justinkambic November 6, 2019 12:12

Bamieh added 2 commits November 6, 2019 15:54

merge master + add tests

2715b43

Merge branch 'master' of github.com:elastic/kibana into telemetry/ser…

e0c10b6

…ver_fetcher

Bamieh added Feature:Telemetry release_note:roadmap v7.5.0 v7.6.0 v8.0.0 labels Nov 6, 2019

Bamieh requested review from epixa, pmuellr and bmcconaghy November 6, 2019 13:56

justinkambic reviewed Nov 6, 2019

View reviewed changes

Bamieh added 4 commits November 6, 2019 17:11

telemetry plugin collector

0815769

remove extra unused method

df1f45f

remove unused import

dae375a

type check

01cfc26

Bamieh requested a review from a team as a code owner November 6, 2019 15:33

Merge branch 'master' into telemetry/server_fetcher

73bcafd

Bamieh requested a review from gmmorris November 12, 2019 22:00

Bamieh added 2 commits November 13, 2019 00:14

take into account older settings

cd0730e

Merge branch 'telemetry/server_fetcher' of github.com:Bamieh/kibana i…

1fbcd73

…nto telemetry/server_fetcher

gmmorris approved these changes Nov 12, 2019

View reviewed changes

import TelemetryOptInProvider

88f1b73

Merge branch 'master' into telemetry/server_fetcher

3961073

Bamieh added 3 commits November 13, 2019 07:51

update test case

2c12688

Merge branch 'telemetry/server_fetcher' of github.com:Bamieh/kibana i…

d88e765

…nto telemetry/server_fetcher

Merge branch 'master' of github.com:elastic/kibana into telemetry/ser…

6a077d3

…ver_fetcher

Bamieh merged commit 35a5b77 into elastic:master Nov 13, 2019

Bamieh deleted the telemetry/server_fetcher branch November 13, 2019 07:19

This was referenced Nov 13, 2019

[7.x] [Telemetry] Server side fetcher (#50015) #50454

Merged

[7.5] [Telemetry] Server side fetcher (#50015) #50455

Merged

cachedout mentioned this pull request Nov 13, 2019

[Metricbeat] kibana_stats.usage.telemetry.opt_in_status missing elastic/beats#14501

Closed

mikecote mentioned this pull request Nov 13, 2019

[Telemetry] server side fetcher #49711

Closed

tsullivan reviewed Nov 13, 2019

View reviewed changes

alexfrancoeur mentioned this pull request Nov 15, 2019

[DOCS] 7.5.0 Release Notes #50425

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Telemetry] Server side fetcher #50015

[Telemetry] Server side fetcher #50015

Bamieh commented Nov 4, 2019 •

edited

Loading

elasticmachine commented Nov 4, 2019

elasticmachine commented Nov 4, 2019

elasticmachine commented Nov 6, 2019

elasticmachine commented Nov 6, 2019

justinkambic left a comment

elasticmachine commented Nov 6, 2019

Bamieh commented Nov 12, 2019

gmmorris left a comment

elasticmachine commented Nov 12, 2019

Bamieh commented Nov 12, 2019

elasticmachine commented Nov 13, 2019

elasticmachine commented Nov 13, 2019

tsullivan Nov 13, 2019

tylersmalley Nov 14, 2019

[Telemetry] Server side fetcher #50015

[Telemetry] Server side fetcher #50015

Conversation

Bamieh commented Nov 4, 2019 • edited Loading

Details:

Notes

elasticmachine commented Nov 4, 2019

💔 Build Failed

elasticmachine commented Nov 4, 2019

💔 Build Failed

elasticmachine commented Nov 6, 2019

💔 Build Failed

elasticmachine commented Nov 6, 2019

justinkambic left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 6, 2019

💔 Build Failed

Bamieh commented Nov 12, 2019

gmmorris left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 12, 2019

💔 Build Failed

Bamieh commented Nov 12, 2019

elasticmachine commented Nov 13, 2019

💔 Build Failed

elasticmachine commented Nov 13, 2019

💚 Build Succeeded

tsullivan Nov 13, 2019

Choose a reason for hiding this comment

tylersmalley Nov 14, 2019

Choose a reason for hiding this comment

Bamieh commented Nov 4, 2019 •

edited

Loading