[APM] Improvements to data telemetry #70524

smith · 2020-07-01T22:18:13Z

Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index.

Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.)

Stop storing the mapping for the data telemetry in the Saved Object but instead use { dynamic: false }.

This reduces the number of fields used by APM in the .kibana index (as requested in #43673.)

Before:

> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
653

After:

> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
415

We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the upload-telemetry-data script uses that mapping when generating data.

For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts.

It's broken down into some variables that are put together into the same mapping object that was there before, but having it in this form should make it easier to update.

A new script, merge-telemetry-mapping, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping.

References #61583
Fixes #67032

Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in elastic#43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. Fixes elastic#61583 Fixes elastic#67032

smith · 2020-07-02T07:28:12Z

x-pack/plugins/apm/common/agent_name.ts

@@ -15,15 +15,14 @@ import { AgentName } from '../typings/es_schemas/ui/fields/agent';
 */

 export const AGENT_NAMES: AgentName[] = [
-  'java',


This has been alphabetized so the mapping output stays consistent with what already exists, and there were two "java"s here.

smith · 2020-07-02T07:28:58Z

x-pack/plugins/apm/public/plugin.ts

-  private readonly initializerContext: PluginInitializerContext<ConfigSchema>;
-  constructor(initializerContext: PluginInitializerContext<ConfigSchema>) {
+  constructor(
+    private readonly initializerContext: PluginInitializerContext<ConfigSchema>


This change is not related to anything else in this PR but lets us remove a line of code.

elasticmachine · 2020-07-02T07:29:53Z

Pinging @elastic/apm-ui (Team:apm)

dgieselaar

A quick look and no concerns from me, looks great. We should test it thoroughly though (obviously). Thanks so much @smith! I've felt bad about this code for a bit and it looks so much better now.

smith · 2020-07-07T01:48:15Z

retest

smith · 2020-07-07T04:40:25Z

retest

kibanamachine · 2020-07-07T08:45:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: daa72a7

Build metrics

✅ unchanged

History

💔 Build #59254 failed 6f447cd
💔 Build #59242 failed 6f447cd
💔 Build #59206 failed 6f447cd
💔 Build #59138 failed f27863b
💚 Build #58158 succeeded 710d640

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in elastic#43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References elastic#61583 Fixes elastic#67032

Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032

smith force-pushed the nls/cloud-telem branch from 03ca4c1 to 274a169 Compare July 2, 2020 06:55

smith force-pushed the nls/cloud-telem branch from 274a169 to 710d640 Compare July 2, 2020 07:26

smith changed the title ~~Data telemetry updates~~ [APM] Improvements to data telemetry Jul 2, 2020

smith commented Jul 2, 2020

View reviewed changes

smith marked this pull request as ready for review July 2, 2020 07:29

smith requested a review from a team as a code owner July 2, 2020 07:29

botelastic bot added the Team:APM All issues that need APM UI Team support label Jul 2, 2020

smith added v7.9.0 release_note:skip Skip the PR/issue when compiling release notes labels Jul 2, 2020

smith mentioned this pull request Jul 2, 2020

[APM] Improve telemetry DX #61583

Closed

dgieselaar approved these changes Jul 2, 2020

View reviewed changes

smith added 5 commits July 6, 2020 11:23

Merge remote-tracking branch 'upstream/master' into nls/cloud-telem

432536f

export merge function

abf6426

some changes to upload script

0f66097

Merge remote-tracking branch 'upstream/master' into nls/cloud-telem

f27863b

fix snapshot

6f447cd

Merge remote-tracking branch 'upstream/master' into nls/cloud-telem

daa72a7

smith merged commit 735d3ba into elastic:master Jul 7, 2020

smith deleted the nls/cloud-telem branch July 7, 2020 16:20

smith mentioned this pull request Jul 7, 2020

[7.x] [APM] Improvements to data telemetry (#70524) #70980

Merged

timroes mentioned this pull request Jul 7, 2020

Don't index so many saved object fields #43673

Closed

27 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Improvements to data telemetry #70524

[APM] Improvements to data telemetry #70524

smith commented Jul 1, 2020 •

edited

Loading

smith Jul 2, 2020

smith Jul 2, 2020

elasticmachine commented Jul 2, 2020

dgieselaar left a comment

smith commented Jul 7, 2020

smith commented Jul 7, 2020

kibanamachine commented Jul 7, 2020

[APM] Improvements to data telemetry #70524

[APM] Improvements to data telemetry #70524

Conversation

smith commented Jul 1, 2020 • edited Loading

smith Jul 2, 2020

Choose a reason for hiding this comment

smith Jul 2, 2020

Choose a reason for hiding this comment

elasticmachine commented Jul 2, 2020

dgieselaar left a comment

Choose a reason for hiding this comment

smith commented Jul 7, 2020

smith commented Jul 7, 2020

kibanamachine commented Jul 7, 2020

💚 Build Succeeded

Build metrics

History

smith commented Jul 1, 2020 •

edited

Loading