Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] All Rollup docs experimental, better agg limitations, clarify DeleteJob #31299

Merged
merged 1 commit into from
Jun 13, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions x-pack/docs/en/rest-api/rollup/delete-job.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,37 @@
<titleabbrev>Delete Job</titleabbrev>
++++

experimental[]

This API deletes an existing rollup job. The job can be started or stopped, in both cases it will be deleted. Attempting
to delete a non-existing job will throw an exception

.Deleting the job does not delete rolled up data
**********************************
When a job is deleted, that only removes the process that is actively monitoring and rolling up data.
It does not delete any previously rolled up data. This is by design; a user may wish to roll up a static dataset. Because
the dataset is static, once it has been fully rolled up there is no need to keep the indexing Rollup job around (as there
will be no new data). So the job may be deleted, leaving behind the rolled up data for analysis.

If you wish to also remove the rollup data, and the rollup index only contains the data for a single job, you can simply
delete the whole rollup index. If the rollup index stores data from several jobs, you must issue a Delete-By-Query that
targets the Rollup job's ID in the rollup index:


[source,js]
--------------------------------------------------
POST my_rollup_index/_delete_by_query
{
"query": {
"term": {
"_rollup.id": "the_rollup_job_id"
}
}
}
--------------------------------------------------
// NOTCONSOLE

**********************************
==== Request

`DELETE _xpack/rollup/job/<job_id>`
Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rest-api/rollup/get-job.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
<titleabbrev>Get Job</titleabbrev>
++++

experimental[]

This API returns the configuration, stats and status of rollup jobs. The API can return the details for a single job,
or for all jobs.

Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rest-api/rollup/put-job.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
<titleabbrev>Create Job</titleabbrev>
++++

experimental[]

This API enables you to create a rollup job. The job will be created in a `STOPPED` state, and must be
started with the <<rollup-start-job,Start Job API>>.

Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rest-api/rollup/rollup-caps.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
<titleabbrev>Get Rollup Caps</titleabbrev>
++++

experimental[]

This API returns the rollup capabilities that have been configured for an index or index pattern. This API is useful
because a rollup job is often configured to rollup only a subset of fields from the source index. Furthermore, only
certain aggregations can be configured for various fields, leading to a limited subset of functionality depending on
Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rest-api/rollup/rollup-job-config.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
[[rollup-job-config]]
=== Rollup Job Configuration

experimental[]

The Rollup Job Configuration contains all the details about how the rollup job should run, when it indexes documents,
and what future queries will be able to execute against the rollup index.

Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rest-api/rollup/rollup-search.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
<titleabbrev>Rollup Search</titleabbrev>
++++

experimental[]

The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint
is needed because, internally, rolled-up documents utilize a different document structure than the original data. The
Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response
Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rest-api/rollup/start-job.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
<titleabbrev>Start Job</titleabbrev>
++++

experimental[]

This API starts an existing, stopped rollup job. If the job does not exist an exception will be thrown.
Starting an already started job has no action.

Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rest-api/rollup/stop-job.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
<titleabbrev>Stop Job</titleabbrev>
++++

experimental[]

This API stops an existing, started rollup job. If the job does not exist an exception will be thrown.
Stopping an already stopped job has no action.

Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rollup/api-quickref.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[[rollup-api-quickref]]
== API Quick Reference

experimental[]

Most {rollup} endpoints have the following base:

[source,js]
Expand Down
4 changes: 3 additions & 1 deletion x-pack/docs/en/rollup/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ for analysis, but at a fraction of the storage cost of raw data.
* <<rollup-getting-started,Getting Started>>
* <<rollup-api-quickref, API Quick Reference>>
* <<rollup-understanding-groups,Understanding Rollup Grouping>>
* <<rollup-search-limitations,Limitations of Rollup Search>>
* <<rollup-agg-limitations,Rollup aggregation limitations>>
* <<rollup-search-limitations,Rollup Search limitations>>


--
Expand All @@ -27,4 +28,5 @@ include::overview.asciidoc[]
include::api-quickref.asciidoc[]
include::rollup-getting-started.asciidoc[]
include::understanding-groups.asciidoc[]
include::rollup-agg-limitations.asciidoc[]
include::rollup-search-limitations.asciidoc[]
2 changes: 2 additions & 0 deletions x-pack/docs/en/rollup/overview.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[[rollup-overview]]
== Overview

experimental[]

Time-based data (documents that are predominantly identified by their timestamp) often have associated retention policies
to manage data growth. For example, your system may be generating 500,000 documents every second. That will generate
43 million documents per day, and nearly 16 billion documents a year.
Expand Down
24 changes: 24 additions & 0 deletions x-pack/docs/en/rollup/rollup-agg-limitations.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[[rollup-agg-limitations]]
== Rollup Aggregation Limitations

experimental[]

There are some limitations to how fields can be rolled up / aggregated. This page highlights the major limitations so that
you are aware of them.

[float]
=== Limited aggregation components

The Rollup functionality allows fields to be grouped with the following aggregations:

- Date Histogram aggregation
- Histogram aggregation
- Terms aggregation

And the following metrics are allowed to be specified for numeric fields:

- Min aggregation
- Max aggregation
- Sum aggregation
- Average aggregation
- Value Count aggregation
2 changes: 2 additions & 0 deletions x-pack/docs/en/rollup/rollup-getting-started.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[[rollup-getting-started]]
== Getting Started

experimental[]

To use the Rollup feature, you need to create one or more "Rollup Jobs". These jobs run continuously in the background
and rollup the index or indices that you specify, placing the rolled documents in a secondary index (also of your choosing).

Expand Down
6 changes: 4 additions & 2 deletions x-pack/docs/en/rollup/rollup-search-limitations.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[[rollup-search-limitations]]
== Rollup Search Limitations

experimental[]

While we feel the Rollup function is extremely flexible, the nature of summarizing data means there will be some limitations. Once
live data is thrown away, you will always lose some flexibility.

Expand Down Expand Up @@ -100,8 +102,8 @@ The Rollup functionality allows `query`'s in the search request, but with a limi
- MatchAll Query
- Any compound query (Boolean, Boosting, ConstantScore, etc)

Furthermore, these queries can only use fields that were also saved in the rollup job. If you wish to filter on a keyword `hostname` field,
that field must have been configured in the rollup job under a `terms` grouping.
Furthermore, these queries can only use fields that were also saved in the rollup job as a `group`.
If you wish to filter on a keyword `hostname` field, that field must have been configured in the rollup job under a `terms` grouping.

If you attempt to use an unsupported query, or the query references a field that wasn't configured in the rollup job, an exception will be
thrown. We expect the list of support queries to grow over time as more are implemented.
Expand Down
2 changes: 2 additions & 0 deletions x-pack/docs/en/rollup/understanding-groups.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[[rollup-understanding-groups]]
== Understanding Groups

experimental[]

To preserve flexibility, Rollup Jobs are defined based on how future queries may need to use the data. Traditionally, systems force
the admin to make decisions about what metrics to rollup and on what interval. E.g. The average of `cpu_time` on an hourly basis. This
is limiting; if, at a future date, the admin wishes to see the average of `cpu_time` on an hourly basis _and partitioned by `host_name`_,
Expand Down