Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Update ingest pipelines content and page navigation #4285

Closed
wants to merge 41 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
4fafff4
Update content and page navigation
vagimeli Jun 7, 2023
0f99dbc
Update create-update-ingest.md
vagimeli Jun 7, 2023
e878d11
Update index.md
vagimeli Jun 7, 2023
897bf65
Update get-ingest.md
vagimeli Jun 7, 2023
ac54479
Update simulate-ingest.md
vagimeli Jun 7, 2023
9edac34
Address tech input
vagimeli Jun 9, 2023
005a2bf
Update _api-reference/ingest-apis/create-update-ingest.md
vagimeli Jun 9, 2023
e941b09
Update _api-reference/ingest-apis/create-update-ingest.md
vagimeli Jun 9, 2023
073211b
Update _api-reference/ingest-apis/delete-ingest.md
vagimeli Jun 9, 2023
44eae15
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
714f758
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
837552a
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
69cf081
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
410a3f7
Update _api-reference/ingest-apis/delete-ingest.md
vagimeli Jun 9, 2023
d68d5ae
Update _api-reference/ingest-apis/index.md
vagimeli Jun 9, 2023
ce55172
Update _api-reference/ingest-apis/get-ingest.md
vagimeli Jun 9, 2023
4330ca4
Update _api-reference/ingest-apis/index.md
vagimeli Jun 9, 2023
2366474
Address tech input
vagimeli Jun 9, 2023
d77c1e9
Rework pipelines documentation
vagimeli Jun 13, 2023
ba0157d
Revisions to address tech input
vagimeli Jun 14, 2023
3527111
Refresh index page
vagimeli Jun 14, 2023
168ef53
Update index.md
vagimeli Jun 14, 2023
1121548
Update index.md
vagimeli Jun 14, 2023
556880c
Modified example requests and responses
kolchfa-aws Jun 21, 2023
5c839a3
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
398bf05
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
4f9e963
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
65e72e0
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
8cb9425
Update ingest-pipelines.md
vagimeli Jun 21, 2023
940b7f9
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
79ee9e9
Update index.md
vagimeli Jun 21, 2023
77892e8
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
b07d196
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
8c33a4a
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
66e0fd3
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
6d538d8
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
1c6cc05
Update ingest-pipelines.md
vagimeli Jun 21, 2023
d7ec21e
Copy edits
vagimeli Jul 31, 2023
6f8ce5c
Copy edits
vagimeli Jul 31, 2023
b5e3839
Copy edits
vagimeli Jul 31, 2023
4970c58
Copy edits
vagimeli Jul 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 0 additions & 79 deletions _api-reference/ingest-apis/create-update-ingest.md

This file was deleted.

44 changes: 0 additions & 44 deletions _api-reference/ingest-apis/delete-ingest.md

This file was deleted.

59 changes: 0 additions & 59 deletions _api-reference/ingest-apis/get-ingest.md

This file was deleted.

155 changes: 153 additions & 2 deletions _api-reference/ingest-apis/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,157 @@ redirect_from:

# Ingest APIs

vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Before you index your data, OpenSearch's ingest APIs help transform your data by creating and managing ingest pipelines. Pipelines consist of **processors**, customizable tasks that run in the order they appear in the request body. The transformed data appears in your index after each of the processor completes.
OpenSearch ingest APIs simplify the data ingestion process with a standard and structured way to process input data. Particularly, ingest APIs manage tasks for ingest pipelines and processors. Ingest pipelines preprocess and transform data before it's indexed in OpenSearch. A pipeline consists of processors, customizable tasks that run specific changes sequentially on incoming documents. The transformed data is then ingested in your index.

Ingest pipelines in OpenSearch can only be managed using ingest API operations. When using ingest in production environments, your cluster should contain at least one node with the node roles permission set to `ingest`. For more information on setting up node roles within a cluster, see [Cluster Formation]({{site.url}}{{site.baseurl}}/opensearch/cluster/).
The key fields typically used in a pipeline are `description` and `processors`. The `description` is an optional field that provides a description of the pipeline's purpose or functionality. The `processors` field is an array that defines the sequence of processing stages within the pipeline. The output of one processor becomes the input for the next.

```json
{
"description" : "...",
"processors" : [ ... ]
}
```

## OpenSearch ingest processor types

To see the list of available processors in your version of OpenSearch, run the following command:

```json
GET /_nodes/ingest
```
{% include copy-curl.html %}

Your response should be similar to the following:

```json
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "opensearch-cluster",
"nodes" : {
"i9hLyCxYTqOpa22BcJkKyg" : {
"name" : "opensearch-node1",
"transport_address" : "172.18.0.3:9300",
"host" : "172.18.0.3",
"ip" : "172.18.0.3",
"version" : "2.7.0",
"build_type" : "tar",
"build_hash" : "b7a6e09e492b1e965d827525f7863b366ef0e304",
"roles" : [
"cluster_manager",
"data",
"ingest",
"remote_cluster_client"
],
"attributes" : {
"shard_indexing_pressure_enabled" : "true"
},
"ingest" : {
"processors" : [
{
"type" : "append"
},
{
"type" : "bytes"
},
{
"type" : "convert"
},
{
"type" : "csv"
},
{
"type" : "date"
},
{
"type" : "date_index_name"
},
{
"type" : "dissect"
},
{
"type" : "dot_expander"
},
{
"type" : "drop"
},
{
"type" : "fail"
},
{
"type" : "foreach"
},
{
"type" : "geoip"
},
{
"type" : "geojson-feature"
},
{
"type" : "grok"
},
{
"type" : "gsub"
},
{
"type" : "html_strip"
},
{
"type" : "join"
},
{
"type" : "json"
},
{
"type" : "kv"
},
{
"type" : "lowercase"
},
{
"type" : "pipeline"
},
{
"type" : "remove"
},
{
"type" : "rename"
},
{
"type" : "script"
},
{
"type" : "set"
},
{
"type" : "sort"
},
{
"type" : "split"
},
{
"type" : "text_embedding"
},
{
"type" : "trim"
},
{
"type" : "uppercase"
},
{
"type" : "urldecode"
},
{
"type" : "user_agent"
}
]
}
}
}
}
```

vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Get started with [creating an ingest pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/ingest-pipelines/) in OpenSearch, or learn more about OpenSearch [ingest processors]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/ingest-processors/).
Loading