From 8e95bdfa8b9fdad3a06bc700226885bffda34cd4 Mon Sep 17 00:00:00 2001 From: Andrei Dan Date: Thu, 7 Dec 2023 17:16:46 +0000 Subject: [PATCH 1/3] [DOCS] DSL downsampling docs --- .../lifecycle/apis/put-lifecycle.asciidoc | 36 +++++++++++++++++++ .../data-streams/lifecycle/index.asciidoc | 8 ++++- 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc index 89b8bbeb880c3..80951acc99239 100644 --- a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc +++ b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc @@ -59,6 +59,16 @@ duration the document could be deleted. When empty, every document in this data If defined, it turns data streqm lifecycle on/off (`true`/`false`) for this data stream. A data stream lifecycle that's disabled (`enabled: false`) will have no effect on the data stream. Defaults to `true`. + +`downsampling`:: +(Optional, array) +An optional array of downsampling configuration objects, each definig an `after` +interval representing when the backing index is meant to be downsampled (the time +frame is calculated since the index was rolled over i.e. generation time) and +an `fixed_interval` representing the downsampling interval (the minimum `fixed_interval` +value is `5m`). A maximum number of 10 downsampling rounds can be configured. +See <> below. + ==== [[data-streams-put-lifecycle-example]] @@ -84,3 +94,29 @@ When the lifecycle is successfully updated in all data streams, you receive the "acknowledged": true } -------------------------------------------------- + +[[data-streams-put-lifecycle-downsampling-example]] +==== {api-examples-title} + +The following example configures two downsampling rounds, the first one starting +one day after the backing index is rolled over (or later, if the index is still +within its write-accepting <>) with an interval +of `10m`, and a second round starting 7 days after rollover at an interval of `1d`: + +[source,console] +-------------------------------------------------------------------- +PUT _data_stream/my-weather-sensor-data-stream/_lifecycle +{ + "downsampling": [ + { + "after": "1d", + "fixed_interval": "10m" + }, + { + "after": "7d", + "fixed_interval": "1d" + } + ] +} +----------------------------------------------------------------- +//TEST[skip:downsampling requires waiting for indices to be out of time bounds] diff --git a/docs/reference/data-streams/lifecycle/index.asciidoc b/docs/reference/data-streams/lifecycle/index.asciidoc index 6c0220ef0a80f..653926294db4b 100644 --- a/docs/reference/data-streams/lifecycle/index.asciidoc +++ b/docs/reference/data-streams/lifecycle/index.asciidoc @@ -18,6 +18,10 @@ and backwards incompatible mapping changes. * Configurable retention, which allows you to configure the time period for which your data is guaranteed to be stored. {es} is allowed at a later time to delete data older than this time period. +Data stream lifecycle also support downsampling the data stream backing indices. +See <> for +more details. + [discrete] [[data-streams-lifecycle-how-it-works]] === How does it work? @@ -35,7 +39,9 @@ into tiers of exponential sizes, merging the long tail of small segments is only fraction of the cost of force mergeing to a single segment. The small segments would usually hold the most recent data so tail mergeing will focus the merging resources on the higher-value data that is most likely to keep being queried. -4. Applies retention to the remaining backing indices. This means deleting the backing indices whose +4. If <> is configured it will execute +all the configured downsampling rounds. +5. Applies retention to the remaining backing indices. This means deleting the backing indices whose `generation_time` is longer than the configured retention period. The `generation_time` is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the <> setting. From 282c6d70d3857309f8ae02b09b53d42876f662af Mon Sep 17 00:00:00 2001 From: Andrei Dan Date: Fri, 8 Dec 2023 10:56:57 +0000 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: Lee Hinman --- .../data-streams/lifecycle/apis/put-lifecycle.asciidoc | 6 +++--- docs/reference/data-streams/lifecycle/index.asciidoc | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc index 80951acc99239..26050076d43de 100644 --- a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc +++ b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc @@ -62,10 +62,10 @@ data stream. Defaults to `true`. `downsampling`:: (Optional, array) -An optional array of downsampling configuration objects, each definig an `after` +An optional array of downsampling configuration objects, each defining an `after` interval representing when the backing index is meant to be downsampled (the time -frame is calculated since the index was rolled over i.e. generation time) and -an `fixed_interval` representing the downsampling interval (the minimum `fixed_interval` +frame is calculated since the index was rolled over, i.e. generation time) and +a `fixed_interval` representing the downsampling interval (the minimum `fixed_interval` value is `5m`). A maximum number of 10 downsampling rounds can be configured. See <> below. diff --git a/docs/reference/data-streams/lifecycle/index.asciidoc b/docs/reference/data-streams/lifecycle/index.asciidoc index 653926294db4b..ef5558817885e 100644 --- a/docs/reference/data-streams/lifecycle/index.asciidoc +++ b/docs/reference/data-streams/lifecycle/index.asciidoc @@ -18,7 +18,7 @@ and backwards incompatible mapping changes. * Configurable retention, which allows you to configure the time period for which your data is guaranteed to be stored. {es} is allowed at a later time to delete data older than this time period. -Data stream lifecycle also support downsampling the data stream backing indices. +A data stream lifecycle also supports downsampling the data stream backing indices. See <> for more details. From b91cf7d82f73f7e8680cfc0c5ed81368e1469ecc Mon Sep 17 00:00:00 2001 From: Andrei Dan Date: Fri, 8 Dec 2023 11:33:34 +0000 Subject: [PATCH 3/3] codeblock --- .../data-streams/lifecycle/apis/put-lifecycle.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc index 26050076d43de..53bd3c2b96f0b 100644 --- a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc +++ b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc @@ -118,5 +118,5 @@ PUT _data_stream/my-weather-sensor-data-stream/_lifecycle } ] } ------------------------------------------------------------------ +-------------------------------------------------------------------- //TEST[skip:downsampling requires waiting for indices to be out of time bounds]