Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] Improve telemetry DX #61583

Closed
dgieselaar opened this issue Mar 27, 2020 · 3 comments
Closed

[APM] Improve telemetry DX #61583

dgieselaar opened this issue Mar 27, 2020 · 3 comments
Labels
apm:telemetry Feature:Telemetry Team:APM All issues that need APM UI Team support technical debt Improvement of the software architecture and operational architecture

Comments

@dgieselaar
Copy link
Member

Currently, changes in what telemetry we collect need to be persisted in several places:

  • the type definitions
  • the saved objects mapping
  • the tests (which we don't have yet)

This is tedious & prone to human error. We should investigate if we can have one definition, and use it to generate all that we need. One possibility is a io-ts runtime type.

@dgieselaar dgieselaar added Team:APM All issues that need APM UI Team support technical debt Improvement of the software architecture and operational architecture v7.7.0 labels Mar 27, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/apm-ui (Team:apm)

smith added a commit to smith/kibana that referenced this issue Jul 2, 2020
Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index.

Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.)

Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`.

This reduces the number of fields used by APM in the .kibana index (as requested in elastic#43673.)

Before:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
653
```

After:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
415
```

We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data.

For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts.

It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update.

A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping.

Fixes elastic#61583
Fixes elastic#67032
@smith
Copy link
Contributor

smith commented Jul 2, 2020

#70524 improves things some, but we still have the type definition and the mapping defining basically the same stuff in two different places.

An io-ts runtime type for ES mappings would let us define the mapping once and use it to generate the type definition.

smith added a commit that referenced this issue Jul 7, 2020
Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index.

Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.)

Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`.

This reduces the number of fields used by APM in the .kibana index (as requested in #43673.)

Before:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
653
```

After:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
415
```

We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data.

For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts.

It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update.

A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping.

References #61583
Fixes #67032
smith added a commit to smith/kibana that referenced this issue Jul 7, 2020
Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index.

Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.)

Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`.

This reduces the number of fields used by APM in the .kibana index (as requested in elastic#43673.)

Before:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
653
```

After:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
415
```

We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data.

For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts.

It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update.

A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping.

References elastic#61583
Fixes elastic#67032
smith added a commit that referenced this issue Jul 7, 2020
Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index.

Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.)

Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`.

This reduces the number of fields used by APM in the .kibana index (as requested in #43673.)

Before:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
653
```

After:

```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" |  jq '.fields|length'
415
```

We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data.

For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts.

It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update.

A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping.

References #61583
Fixes #67032
@dgieselaar
Copy link
Member Author

Not planned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apm:telemetry Feature:Telemetry Team:APM All issues that need APM UI Team support technical debt Improvement of the software architecture and operational architecture
Projects
None yet
Development

No branches or pull requests

7 participants