Skip to content

Commit

Permalink
Adds query DSL section to 1.3 (#3000)
Browse files Browse the repository at this point in the history
Signed-off-by: Fanit Kolchina <[email protected]>
  • Loading branch information
kolchfa-aws authored Feb 21, 2023
1 parent 2d0a17f commit a5341b2
Show file tree
Hide file tree
Showing 20 changed files with 655 additions and 314 deletions.
10 changes: 5 additions & 5 deletions _im-plugin/index-rollups/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ has_toc: false

# Index rollups

Time series data increases storage costs, strains cluster health, and slows down aggregations over time. Index rollup lets you periodically reduce data granularity by rolling up old data into summarized indices.
Time series data increases storage costs, strains cluster health, and slows down aggregations over time. Index rollup lets you periodically reduce data granularity by rolling up old data into summarized indexes.

You pick the fields that interest you and use index rollup to create a new index with only those fields aggregated into coarser time buckets. You can store months or years of historical data at a fraction of the cost with the same query performance.

Expand All @@ -18,15 +18,15 @@ For example, say you collect CPU consumption data every five seconds and store i
You can use index rollup in three ways:

1. Use the index rollup API for an on-demand index rollup job that operates on an index that's not being actively ingested such as a rolled-over index. For example, you can perform an index rollup operation to reduce data collected at a five minute interval to a weekly average for trend analysis.
2. Use the OpenSearch Dashboards UI to create an index rollup job that runs on a defined schedule. You can also set it up to roll up your indices as it’s being actively ingested. For example, you can continuously roll up Logstash indices from a five second interval to a one hour interval.
2. Use the OpenSearch Dashboards UI to create an index rollup job that runs on a defined schedule. You can also set it up to roll up your indexes as it’s being actively ingested. For example, you can continuously roll up Logstash indexes from a five second interval to a one hour interval.
3. Specify the index rollup job as an ISM action for complete index management. This allows you to roll up an index after a certain event such as a rollover, index age reaching a certain point, index becoming read-only, and so on. You can also have rollover and index rollup jobs running in sequence, where the rollover first moves the current index to a warm node and then the index rollup job creates a new index with the minimized data on the hot node.

## Create an Index Rollup Job

To get started, choose **Index Management** in OpenSearch Dashboards.
Select **Rollup Jobs** and choose **Create rollup job**.

### Step 1: Set up indices
### Step 1: Set up indexes

1. In the **Job name and description** section, specify a unique name and an optional description for the index rollup job.
2. In the **Indices** section, select the source and target index. The source index is the one that you want to roll up. The source index remains as is, the index rollup job creates a new index referred to as a target index. The target index is where the index rollup results are saved. For target index, you can either type in a name for a new index or you select an existing index.
Expand All @@ -48,7 +48,7 @@ The order in which you select attributes is critical. A city followed by a demog

### Step 3: Specify schedule

Specify a schedule to roll up your indices as it’s being ingested. The index rollup job is enabled by default.
Specify a schedule to roll up your indexes as it’s being ingested. The index rollup job is enabled by default.

1. Specify if the data is continuous or not.
3. For roll up execution frequency, select **Define by fixed interval** and specify the **Rollup interval** and the time unit or **Define by cron expression** and add in a cron expression to select the interval. To learn how to define a cron expression, see [Alerting]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/).
Expand Down Expand Up @@ -303,7 +303,7 @@ PUT _plugins/_rollup/jobs/example
```

You can query the `example_rollup` index for the terms aggregations on the fields set up in the rollup job.
You get back the same response that you would on the original `opensearch_dashboards_sample_data_ecommerce` source index.
You get back the same response that you would on the original `opensearch_dashboards_sample_data_ecommerce` source index:

```json
POST example_rollup/_search
Expand Down
4 changes: 2 additions & 2 deletions _observing-your-data/alerting/monitors.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Whereas query-level monitors run your specified query and then check whether the

- Visual definition works well for monitors that you can define as "some value is above or below some threshold for some amount of time."

- Query definition gives you flexibility in terms of what you query for (using [the OpenSearch query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text)) and how you evaluate the results of that query (Painless scripting).
- Query definition gives you flexibility in terms of what you query for (using [OpenSearch query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text/index)) and how you evaluate the results of that query (Painless scripting).

This example averages the `cpu_usage` field:

Expand Down Expand Up @@ -164,7 +164,7 @@ Whereas query-level monitors run your specified query and then check whether the

If you use the Security plugin, you can only choose indexes that you have permission to access. For details, see [Alerting security]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/security/).

To use a query, choose **Extraction query editor**, add your query (using [the OpenSearch query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text/)), and test it using the **Run** button.
To use a query, choose **Extraction query editor**, add your query (using [OpenSearch query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text/index)), and test it using the **Run** button.

The monitor makes this query to OpenSearch as often as the schedule dictates; check the **Query Performance** section and make sure you're comfortable with the performance implications.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
---
layout: default
title: Boolean queries
parent: Query DSL
nav_order: 45
parent: Compound queries
grand_parent: Query DSL
nav_order: 10
---

# Boolean queries
Expand Down
19 changes: 19 additions & 0 deletions _opensearch/query-dsl/compound/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
layout: default
title: Compound queries
parent: Query DSL
has_children: true
nav_order: 40
---

# Compound queries

Compound queries serve as wrappers for multiple leaf or compound clauses either to combine their results or to modify their behavior.

OpenSearch supports the following compound query types:

- **Boolean**: Combines multiple query clauses with Boolean logic. To learn more, see [Boolean queries]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/compound/bool/).
- **Constant score**: Wraps a query or a filter and assigns a constant score to all matching documents. This score is equal to the `boost` value.
- **Disjunction max**: Returns documents that match one or more query clauses. If a document matches multiple query clauses, it is assigned a higher relevance score. The relevance score is calculated using the highest score from any matching clause and, optionally, the scores from the other matching clauses multiplied by the tiebreaker value.
- **Function score**: Recalculates the relevance score of documents that are returned by a query using a function that you define.
- **Boosting**: Changes the relevance score of documents without removing them from the search results. Returns documents that match a `positive` query, but downgrades the relevance of documents in the results that match a `negative` query.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
layout: default
title: Full-text queries
parent: Query DSL
nav_order: 40
has_children: true
nav_order: 30
---

# Full-text queries
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
---
layout: default
title: Query string queries
parent: Query DSL
nav_order: 70
parent: Full-text queries
grand_parent: Query DSL
nav_order: 25
---

# Query string queries
Expand Down Expand Up @@ -41,7 +42,7 @@ Parameter | Data type | Description
`phrase_slop` | Integer | The maximum number of words that are allowed between the matched words. If `phrase_slop` is 2, a maximum of two words is allowed between matched words in a phrase. Transposed words have a slop of 2. Default is 0 (an exact phrase match where matched words must be next to each other).
`minimum_should_match` | Positive or negative integer, positive or negative percentage, combination | If the query string contains multiple search terms and you used the `or` operator, the number of terms that need to match for the document to be considered a match. For example, if `minimum_should_match` is 2, "wind often rising" does not match "The Wind Rises." If `minimum_should_match` is 1, it matches.
`rewrite` | String | Determines how OpenSearch rewrites and scores multi-term queries. Valid values are `constant_score`, `scoring_boolean`, `constant_score_boolean`, `top_terms_N`, `top_terms_boost_N`, and `top_terms_blended_freqs_N`. Default is `constant_score`.
`auto_generate_synonyms_phrase_query` | Boolean | Specifies whether to create [match queries]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text#match) automatically for multi-term synonyms. Default is `true`.
`auto_generate_synonyms_phrase_query` | Boolean | Specifies whether to create [match queries]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/full-text/index#match) automatically for multi-term synonyms. Default is `true`.
`boost` | Floating-point | Boosts the clause by the given multiplier. Values less than 1.0 decrease relevance, and values greater than 1.0 increase relevance. Default is 1.0.
`default_operator`| String | The default Boolean operator used if no operators are specified. Valid values are:<br>- `OR`: The string `to be` is interpreted as `to OR be`<br>- `AND`: The string `to be` is interpreted as `to AND be`<br> Default is `OR`.
`enable_position_increments` | Boolean | When true, resulting queries are aware of position increments. This setting is useful when the removal of stop words leaves an unwanted "gap" between terms. Default is `true`.
Expand Down
Empty file.
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
---
layout: default
title: Geo-bounding box queries
parent: Query DSL
nav_order: 55
parent: Geographic and xy queries
grand_parent: Query DSL
nav_order: 10
---

# Geo-bounding box queries
Expand Down
32 changes: 32 additions & 0 deletions _opensearch/query-dsl/geo-and-xy/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
layout: default
title: Geographic and xy queries
parent: Query DSL
has_children: true
nav_order: 50
---

# Geographic and xy queries

Geographic and xy queries let you search fields that contain points and shapes on a map or coordinate plane. Geographic queries work on geospatial data, while xy queries work on two-dimensional coordinate data. Out of all geographic queries, the geoshape query is very similar to the xy query, but the former searches [geographic fields]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/geographic), while the latter searches [Cartesian fields]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/xy).

## xy queries

[xy queries]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/geo-and-xy/xy) search for documents that contain geometries in a Cartesian coordinate system. These geometries can be specified in [`xy_point`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/xy-point) fields, which support points, and [`xy_shape`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/xy-shape) fields, which support points, lines, circles, and polygons.

xy queries return documents that contain:
- xy shapes and xy points that have one of four spatial relations to the provided shape: `INTERSECTS`, `DISJOINT`, `WITHIN`, or `CONTAINS`.
- xy points that intersect the provided shape.

## Geographic queries

Geographic queries search for documents that contain geospatial geometries. These geometries can be specified in [`geo_point`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/geo-point) fields, which support points on a map, and [`geo_shape`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/geo-shape) fields, which support points, lines, circles, and polygons.

OpenSearch provides the following geographic query types:

- [**Geo-bounding box queries**]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/geo-and-xy/geo-bounding-box/): Return documents with geopoint field values that are within a bounding box.
- **Geodistance queries** return documents with geopoints that are within a specified distance from the provided geopoint.
- **Geopolygon queries** return documents with geopoints that are within a polygon.
- **Geoshape queries** return documents that contain:
- geoshapes and geopoints that have one of four spatial relations to the provided shape: `INTERSECTS`, `DISJOINT`, `WITHIN`, or `CONTAINS`.
- geopoints that intersect the provided shape.
Loading

0 comments on commit a5341b2

Please sign in to comment.