Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Update ingest pipelines content and page navigation #4285

Closed
wants to merge 41 commits into from
Closed
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
4fafff4
Update content and page navigation
vagimeli Jun 7, 2023
0f99dbc
Update create-update-ingest.md
vagimeli Jun 7, 2023
e878d11
Update index.md
vagimeli Jun 7, 2023
897bf65
Update get-ingest.md
vagimeli Jun 7, 2023
ac54479
Update simulate-ingest.md
vagimeli Jun 7, 2023
9edac34
Address tech input
vagimeli Jun 9, 2023
005a2bf
Update _api-reference/ingest-apis/create-update-ingest.md
vagimeli Jun 9, 2023
e941b09
Update _api-reference/ingest-apis/create-update-ingest.md
vagimeli Jun 9, 2023
073211b
Update _api-reference/ingest-apis/delete-ingest.md
vagimeli Jun 9, 2023
44eae15
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
714f758
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
837552a
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
69cf081
Update _api-reference/ingest-apis/simulate-ingest.md
vagimeli Jun 9, 2023
410a3f7
Update _api-reference/ingest-apis/delete-ingest.md
vagimeli Jun 9, 2023
d68d5ae
Update _api-reference/ingest-apis/index.md
vagimeli Jun 9, 2023
ce55172
Update _api-reference/ingest-apis/get-ingest.md
vagimeli Jun 9, 2023
4330ca4
Update _api-reference/ingest-apis/index.md
vagimeli Jun 9, 2023
2366474
Address tech input
vagimeli Jun 9, 2023
d77c1e9
Rework pipelines documentation
vagimeli Jun 13, 2023
ba0157d
Revisions to address tech input
vagimeli Jun 14, 2023
3527111
Refresh index page
vagimeli Jun 14, 2023
168ef53
Update index.md
vagimeli Jun 14, 2023
1121548
Update index.md
vagimeli Jun 14, 2023
556880c
Modified example requests and responses
kolchfa-aws Jun 21, 2023
5c839a3
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
398bf05
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
4f9e963
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
65e72e0
Update _api-reference/ingest-apis/index.md
vagimeli Jun 21, 2023
8cb9425
Update ingest-pipelines.md
vagimeli Jun 21, 2023
940b7f9
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
79ee9e9
Update index.md
vagimeli Jun 21, 2023
77892e8
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
b07d196
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
8c33a4a
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
66e0fd3
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
6d538d8
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Jun 21, 2023
1c6cc05
Update ingest-pipelines.md
vagimeli Jun 21, 2023
d7ec21e
Copy edits
vagimeli Jul 31, 2023
6f8ce5c
Copy edits
vagimeli Jul 31, 2023
b5e3839
Copy edits
vagimeli Jul 31, 2023
4970c58
Copy edits
vagimeli Jul 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 22 additions & 43 deletions _api-reference/ingest-apis/create-update-ingest.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,21 @@
layout: default
title: Create or update ingest pipeline
parent: Ingest APIs
nav_order: 11
nav_order: 10
redirect_from:
- /opensearch/rest-api/ingest-apis/create-update-ingest/
---

# Create and update a pipeline
# Create or update a pipeline

The create ingest pipeline API operation creates or updates an ingest pipeline. Each pipeline requires an ingest definition defining how each processor transforms your documents.
The create ingest pipeline API operation creates or updates an ingest pipeline. Each pipeline requires an ingest definition defining how each processor transforms your data.

## Example
The following is an example of a create pipeline API request:

```
PUT _ingest/pipeline/12345
```json
PUT _ingest/pipeline/my-pipeline-id
{
"description" : "A description for your pipeline",
"description" : "Example pipeline",
"processors" : [
{
"set" : {
Expand All @@ -29,51 +29,30 @@ PUT _ingest/pipeline/12345
```
{% include copy-curl.html %}

## Path and HTTP methods
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
```
PUT _ingest/pipeline/{id}
```

## Request body fields

Field | Required | Type | Description
:--- | :--- | :--- | :---
description | Optional | string | Description of your ingest pipeline.
processors | Required | Array of processor objects | A processor that transforms documents. Runs in the order specified. Appears in index once ran.

```json
{
"description" : "A description for your pipeline",
"processors" : [
{
"set" : {
"field": "field-name",
"value": "value"
}
}
]
}
```
Field | Type | Description
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
:--- | :--- | :---
`description` | String | Description of the ingest pipeline. Optional.
`processors` | Array | The processor that performs an ingest action on the data. Processors run sequentially. Required.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## URL parameters
## Path parameters
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

All URL parameters are optional.
Path parameters are required.

Parameter | Type | Description
:--- | :--- | :---
master_timeout | time | How long to wait for a connection to the master node.
timeout | time | How long to wait for the request to return.

## Response

```json
{
"acknowledged" : true
}
```

`pipeline` | String | Pipeline ID or wildcard expression of pipeline IDs used to limit the request.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Query parameters

Query parameters are optional.

Parameter | Type | Description
:--- | :--- | :---
master_timeout | Time | Period to wait for a connection to the primary node. Defaults to #s.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
timeout | Time | Period to wait for a response. Defaults to #s.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Next steps

Once you've created or updated a pipeline, use the [get ingest pipeline]({{site.url}}{{site.baseurl}}//api-reference/ingest-apis/get-ingest/) API operation to return all the information about the pipeline.
52 changes: 34 additions & 18 deletions _api-reference/ingest-apis/delete-ingest.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,43 +2,59 @@
layout: default
title: Delete a pipeline
parent: Ingest APIs
nav_order: 14
nav_order: 25
redirect_from:
- /opensearch/rest-api/ingest-apis/delete-ingest/
---

# Delete a pipeline

If you no longer want to use an ingest pipeline, use the delete ingest pipeline API operation.
The delete ingest pipeline API deletes a pipeline.

## Example
#### Request

```
DELETE _ingest/pipeline/12345
```json
DELETE _ingest/pipeline/<pipeline>
```
{% include copy-curl.html %}

## Path and HTTP methods
## Path parameters

Delete an ingest pipeline based on that pipeline's ID.
Path parameters are required.

```
DELETE _ingest/pipeline/
```
Parameter | Type | Description
:--- | :--- | :---
`pipeline` | String | Pipeline ID or wildcard expression of pipeline IDs used to limit the request.

To delete all ingest pipelines in a cluster, use a value of *.

## URL parameters
## Query parameters

All URL parameters are optional.
Query parameters are optional.

Parameter | Type | Description
:--- | :--- | :---
master_timeout | time | How long to wait for a connection to the master node.
timeout | time | How long to wait for the request to return.
`master_timeout` | Time | Period to wait for a connection to the primary node. Defaults to #s.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`timeout` | Time | Period to wait for a response. Defaults to #s.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Examples

The following are delete pipeline examples.

### Delete a specific pipeline

```json
DELETE /_ingest/pipeline/pipeline-specific
```

### Delete pipelines using a wildcard expression

```json
DELETE /_ingest/pipeline/pipeline-*
```

## Response
### Delete all pipelines

```json
{
"acknowledged" : true
}
DELETE /_ingest/pipeline/*
```
47 changes: 22 additions & 25 deletions _api-reference/ingest-apis/get-ingest.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,50 +2,39 @@
layout: default
title: Get ingest pipeline
parent: Ingest APIs
nav_order: 10
nav_order: 15
redirect_from:
- /opensearch/rest-api/ingest-apis/get-ingest/
---

## Get ingest pipeline
# Get ingest pipeline

After you create a pipeline, use the get ingest pipeline API operation to return all the information about a specific ingest pipeline.
After creating a pipeline, use the get ingest pipeline API operation to return all the information about the pipeline.

## Example
## Examples

```
GET _ingest/pipeline/12345
```
{% include copy-curl.html %}
The following examples return a specific pipeline or all pipelines.

## Path and HTTP methods

Return all ingest pipelines.
### Return a specific pipeline

```
GET _ingest/pipeline
GET _ingest/pipeline/pipeline-id
```
{% include copy-curl.html %}

Returns a single ingest pipeline based on the pipeline's ID.
### Return all ingest pipelines

```
GET _ingest/pipeline/{id}
GET _ingest/pipeline
```
{% include copy-curl.html %}

## URL parameters

All parameters are optional.

Parameter | Type | Description
:--- | :--- | :---
master_timeout | time | How long to wait for a connection to the master node.

## Response
#### Example response

```json
{
"pipeline-id" : {
"description" : "A description for your pipeline",
"description" : "Example description",
"processors" : [
{
"set" : {
Expand All @@ -56,4 +45,12 @@ master_timeout | time | How long to wait for a connection to the master node.
]
}
}
```
```

## Query parameters

Query parameters are optional.

Parameter | Type | Description
:--- | :--- | :---
master_timeout | Time | Period to wait for a connection to the primary node. Defaults to #s.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
19 changes: 17 additions & 2 deletions _api-reference/ingest-apis/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,21 @@ redirect_from:

# Ingest APIs

vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Before you index your data, OpenSearch's ingest APIs help transform your data by creating and managing ingest pipelines. Pipelines consist of **processors**, customizable tasks that run in the order they appear in the request body. The transformed data appears in your index after each of the processor completes.
Ingest pipelines in OpenSearch can only be managed using ingest API operations. When using ingest in production environments, your cluster should contain at least one node with the node roles permission set to `ingest`. For more information about setting up node roles within a cluster, see [Cluster formation]({{site.url}}{{site.baseurl}}/opensearch/cluster/).
{: .note}

Ingest pipelines in OpenSearch can only be managed using ingest API operations. When using ingest in production environments, your cluster should contain at least one node with the node roles permission set to `ingest`. For more information on setting up node roles within a cluster, see [Cluster Formation]({{site.url}}{{site.baseurl}}/opensearch/cluster/).
The OpenSearch ingest APIs simplify the data integration process with a standard and structured way to process large volumes of data, and ingest pipelines pre-process and transform data before it's indexed in OpenSearch. Pipelines consist of processors, customizable tasks that run in the order they appear in the request body. The transformed data appears in your index after each processor completes.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

The key fields typically used in a pipeline are `description` and `processors`. The `description` is an optional field that provides a description of the pipeline's purpose or funcationality. The `processors` field is an array that defines the sequence of processing stages within in the pipeline. The output of one processor becomes the input for the next.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
{
"description" : "...",
"processors" : [ ... ]
}
```

## Next steps

- Start first with [creating a pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/create-update-ingest/).
- Learn more about OpenSearch [ingest processors]({{site.url}}{{site.baseurl}}/<index-page-in-progess>).
Loading