[Synthetics] Document changing retention on a per-data-stream basis #1944
Conversation
This patch adds a new docs page for the commonly requested task of altering retention periods for the space-hungry `synthetics-browser.network` and `synthetics-browser.screenshot` indices. While we will have better defaults in elastic/uptime#462 we need this doc before that change makes it into 8.4. Additionally, we will always need some level of documentation here, though we can simplify it a little after that. I've tried to keep the docs screenshot free for kibana index management, since I don't think we'll be able to track visual changes there well. It _is_ a complex process, but I've tried my best to describe it. Once the better defaults are merged it should be as simple as clicking on the default lifecycle policy and editing it, a lot of the complexity now is due to the fact that we share one policy across all dataset types
|
This pull request does not have a backport label. Could you fix it @andrewvc? 🙏
|
mergify
bot
added
the
backport-skip
|
A documentation preview will be available soon: |
mergify
bot
removed
the
backport-skip
A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a `heartbeat` data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use `synthetics*` data streams which breaks them out by type. We can't do this in any sort of simple way with the 'heartbeat' prefix. This shouldn't be a practical issue for most users since in 8.x all kibana clusters have synthetics templates pre-installed. Users who have explicitly set their kibana index pattern to only `heartbeat-*` with no `synthetics-*` will be affected. Given that we are in beta this potential breakage is worth it for the long term benefits to users. This also implies Kibana has been setup with Elasticsearch, which is overwhelmingly the case. If kibana is not setup then the mappings for this synthetics index will be incorrect, but this is likely not a real-world concern. See also elastic/observability-docs#1944
* [Heartbeat] Always use synthetics data streams for browser A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a `heartbeat` data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use `synthetics*` data streams which breaks them out by type. We can't do this in any sort of simple way with the 'heartbeat' prefix. This shouldn't be a practical issue for most users since in 8.x all kibana clusters have synthetics templates pre-installed. Users who have explicitly set their kibana index pattern to only `heartbeat-*` with no `synthetics-*` will be affected. Given that we are in beta this potential breakage is worth it for the long term benefits to users. This also implies Kibana has been setup with Elasticsearch, which is overwhelmingly the case. If kibana is not setup then the mappings for this synthetics index will be incorrect, but this is likely not a real-world concern. See also elastic/observability-docs#1944 * Add changelog, improve docs copy, and add deprecating warning to `index` setting
|
|
||
| To change the retention period of a given data stream simply edit the life cycle policy associated with it after ensuring that policy does not apply to addition data streams whose | ||
| retention you do not want to change. If the data stream you wish to change shares a life cycle policy with another, [create a new ILM policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html), then edit the relevant component template's | ||
| index settings. When editing the component template change the `index.lifecycle.name` value to point toward your new ILM policy. For `screenshot` documents edit the `synthetics-browser.screenshot@package` component template, for `network_info` documents edit the `synthetics-browser.network@package` template. Check that the settings have taken effect by visiting the data stream management |
There was a problem hiding this comment.
Some other documentation includes a bit more specifics about how to edit the component template, exactly what to change with example configuration. I think we should include that here.
There was a problem hiding this comment.
I am concerned about instructing customers to update the @package templates rather than the @custom templates for setting retention across package updates.
| To do so, first navigate to [Kibana index management](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html), then filter the list of data streams for | ||
| those containing the term 'synthetics'. In the UI there will be three sorts of data stream present `synthetics-browser-*`, `synthetics-browser.network-*`, and `synthetics-browser.screenshot-*`. From this page you can retrieve the size of each data stream on disk, as well as which [index life cycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) is associated with it. | ||
|
|
||
| To change the retention period of a given data stream simply edit the life cycle policy associated with it after ensuring that policy does not apply to addition data streams whose |
There was a problem hiding this comment.
simply edit the life cycle policy associated with it
Since our ILM policy defaults will be based off the integration package, we have to be careful with this. If the user edits existing ILM managed by the policy, their changes will be overwritten on the next update to the integration policy.
That's unfortunately not accurate. ILM policies owned by integrations are managed by those integrations, and will update to overwrite any changes to those default settings when the integration package itself is updated. The Fleet team documented how to apply a custom ILM policy to your data streams in a way that preserves changes here: https://github.com/elastic/observability-docs/pull/1194/files |
There was a problem hiding this comment.
@andrewvc I left some comments and suggestions below on the structure of this content. I'll come back and copy edit after those comments are resolved (to avoid editing things that might change).
This guide does a good job of describing why the problem happens and what to do to fix it. I think it's light on how a user would know this is the problem. When I'm writing troubleshooting content I usually try to guess what keywords the user is likely going to type into Google, especially related error messages (since my first troubleshooting step is almost always to copy and paste the error into Google 🙂).
Is there anything else we can add to make sure users using search terms that describe what they're seeing in a moment of frustration end up on this page (to get the answers they need)?
|
I agree with @dominiqueclarke that this documentation should instruct users to create Have you considered linking to our Fleet tutorial on custom ILM policies? It includes screenshots and walks through all the steps required for adding custom ILM policy. Then this documentation could instead focus on the names of the data streams to update and provide recommendations or suggestions based on common use cases. I'm not sure that it makes sense to have two different (but similar) ILM tutorials in our docs—that's double the screenshots and content to maintain. |
|
👋 @andrewvc what can I do to help here? |
|
Love your changes @colleenmcginnis . Can you LGTM this so we can merge it? I can't LGTM it as the author |
…1944) * [Synthetics] Document changing retention on a per-data-stream basis This patch adds a new docs page for the commonly requested task of altering retention periods for the space-hungry `synthetics-browser.network` and `synthetics-browser.screenshot` indices. While we will have better defaults in elastic/uptime#462 we need this doc before that change makes it into 8.4. Additionally, we will always need some level of documentation here, though we can simplify it a little after that. I've tried to keep the docs screenshot free for kibana index management, since I don't think we'll be able to track visual changes there well. It _is_ a complex process, but I've tried my best to describe it. Once the better defaults are merged it should be as simple as clicking on the default lifecycle policy and editing it, a lot of the complexity now is due to the fact that we share one policy across all dataset types * PR feedback * restructure to give context then hand off to fleet doc * clean up Co-authored-by: Colleen McGinnis <[email protected]> (cherry picked from commit df6aaf8)
…1944) * [Synthetics] Document changing retention on a per-data-stream basis This patch adds a new docs page for the commonly requested task of altering retention periods for the space-hungry `synthetics-browser.network` and `synthetics-browser.screenshot` indices. While we will have better defaults in elastic/uptime#462 we need this doc before that change makes it into 8.4. Additionally, we will always need some level of documentation here, though we can simplify it a little after that. I've tried to keep the docs screenshot free for kibana index management, since I don't think we'll be able to track visual changes there well. It _is_ a complex process, but I've tried my best to describe it. Once the better defaults are merged it should be as simple as clicking on the default lifecycle policy and editing it, a lot of the complexity now is due to the fact that we share one policy across all dataset types * PR feedback * restructure to give context then hand off to fleet doc * clean up Co-authored-by: Colleen McGinnis <[email protected]> (cherry picked from commit df6aaf8)
…1944) (#2113) * [Synthetics] Document changing retention on a per-data-stream basis This patch adds a new docs page for the commonly requested task of altering retention periods for the space-hungry `synthetics-browser.network` and `synthetics-browser.screenshot` indices. While we will have better defaults in elastic/uptime#462 we need this doc before that change makes it into 8.4. Additionally, we will always need some level of documentation here, though we can simplify it a little after that. I've tried to keep the docs screenshot free for kibana index management, since I don't think we'll be able to track visual changes there well. It _is_ a complex process, but I've tried my best to describe it. Once the better defaults are merged it should be as simple as clicking on the default lifecycle policy and editing it, a lot of the complexity now is due to the fact that we share one policy across all dataset types * PR feedback * restructure to give context then hand off to fleet doc * clean up Co-authored-by: Colleen McGinnis <[email protected]> (cherry picked from commit df6aaf8) Co-authored-by: Andrew Cholakian <[email protected]>
…1944) (#2112) * [Synthetics] Document changing retention on a per-data-stream basis This patch adds a new docs page for the commonly requested task of altering retention periods for the space-hungry `synthetics-browser.network` and `synthetics-browser.screenshot` indices. While we will have better defaults in elastic/uptime#462 we need this doc before that change makes it into 8.4. Additionally, we will always need some level of documentation here, though we can simplify it a little after that. I've tried to keep the docs screenshot free for kibana index management, since I don't think we'll be able to track visual changes there well. It _is_ a complex process, but I've tried my best to describe it. Once the better defaults are merged it should be as simple as clicking on the default lifecycle policy and editing it, a lot of the complexity now is due to the fact that we share one policy across all dataset types * PR feedback * restructure to give context then hand off to fleet doc * clean up Co-authored-by: Colleen McGinnis <[email protected]> (cherry picked from commit df6aaf8) Co-authored-by: Andrew Cholakian <[email protected]>
* [Heartbeat] Always use synthetics data streams for browser A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a `heartbeat` data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use `synthetics*` data streams which breaks them out by type. We can't do this in any sort of simple way with the 'heartbeat' prefix. This shouldn't be a practical issue for most users since in 8.x all kibana clusters have synthetics templates pre-installed. Users who have explicitly set their kibana index pattern to only `heartbeat-*` with no `synthetics-*` will be affected. Given that we are in beta this potential breakage is worth it for the long term benefits to users. This also implies Kibana has been setup with Elasticsearch, which is overwhelmingly the case. If kibana is not setup then the mappings for this synthetics index will be incorrect, but this is likely not a real-world concern. See also elastic/observability-docs#1944 * Add changelog, improve docs copy, and add deprecating warning to `index` setting
Fixes elastic/synthetics#286
This patch adds a new docs page for the commonly requested task of altering retention periods for the space-hungry
synthetics-browser.networkandsynthetics-browser.screenshotindices.While we will have better defaults in elastic/uptime#462 we need this doc before that change makes it into 8.4. Additionally, we will always need some level of documentation here, though we can simplify it a little after that.
I've tried to keep the docs screenshot free for kibana index management, since I don't think we'll be able to track visual changes there well.
It is a complex process, but I've tried my best to describe it.
Once the better defaults are merged it should be as simple as clicking on the default lifecycle policy and editing it, a lot of the complexity now is due to the fact that we share one policy across all dataset types.