Skip to content

Commit

Permalink
[DOCS] DSL downsampling docs (#103148)
Browse files Browse the repository at this point in the history
  • Loading branch information
andreidan authored Dec 8, 2023
1 parent c6f7e2d commit 1781128
Show file tree
Hide file tree
Showing 2 changed files with 43 additions and 1 deletion.
36 changes: 36 additions & 0 deletions docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,16 @@ duration the document could be deleted. When empty, every document in this data
If defined, it turns data streqm lifecycle on/off (`true`/`false`) for this data stream.
A data stream lifecycle that's disabled (`enabled: false`) will have no effect on the
data stream. Defaults to `true`.
`downsampling`::
(Optional, array)
An optional array of downsampling configuration objects, each defining an `after`
interval representing when the backing index is meant to be downsampled (the time
frame is calculated since the index was rolled over, i.e. generation time) and
a `fixed_interval` representing the downsampling interval (the minimum `fixed_interval`
value is `5m`). A maximum number of 10 downsampling rounds can be configured.
See <<data-streams-put-lifecycle-downsampling-example, configuration example>> below.
====

[[data-streams-put-lifecycle-example]]
Expand All @@ -84,3 +94,29 @@ When the lifecycle is successfully updated in all data streams, you receive the
"acknowledged": true
}
--------------------------------------------------

[[data-streams-put-lifecycle-downsampling-example]]
==== {api-examples-title}

The following example configures two downsampling rounds, the first one starting
one day after the backing index is rolled over (or later, if the index is still
within its write-accepting <<time-bound-indices, time bounds>>) with an interval
of `10m`, and a second round starting 7 days after rollover at an interval of `1d`:

[source,console]
--------------------------------------------------------------------
PUT _data_stream/my-weather-sensor-data-stream/_lifecycle
{
"downsampling": [
{
"after": "1d",
"fixed_interval": "10m"
},
{
"after": "7d",
"fixed_interval": "1d"
}
]
}
--------------------------------------------------------------------
//TEST[skip:downsampling requires waiting for indices to be out of time bounds]
8 changes: 7 additions & 1 deletion docs/reference/data-streams/lifecycle/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ and backwards incompatible mapping changes.
* Configurable retention, which allows you to configure the time period for which your data is guaranteed to be stored.
{es} is allowed at a later time to delete data older than this time period.

A data stream lifecycle also supports downsampling the data stream backing indices.
See <<data-streams-put-lifecycle-downsampling-example, the downsampling example>> for
more details.

[discrete]
[[data-streams-lifecycle-how-it-works]]
=== How does it work?
Expand All @@ -35,7 +39,9 @@ into tiers of exponential sizes, merging the long tail of small segments is only
fraction of the cost of force mergeing to a single segment. The small segments would usually
hold the most recent data so tail mergeing will focus the merging resources on the higher-value
data that is most likely to keep being queried.
4. Applies retention to the remaining backing indices. This means deleting the backing indices whose
4. If <<data-streams-put-lifecycle-downsampling-example, downsampling>> is configured it will execute
all the configured downsampling rounds.
5. Applies retention to the remaining backing indices. This means deleting the backing indices whose
`generation_time` is longer than the configured retention period. The `generation_time` is only applicable to rolled over backing
indices and it is either the time since the backing index got rolled over, or the time optionally configured in the
<<index-data-stream-lifecycle-origination-date,`index.lifecycle.origination_date`>> setting.
Expand Down

0 comments on commit 1781128

Please sign in to comment.