Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Add ingest processors documentation #4299

Merged
merged 286 commits into from
Aug 29, 2023
Merged
Show file tree
Hide file tree
Changes from 194 commits
Commits
Show all changes
286 commits
Select commit Hold shift + click to select a range
49722a8
Writing processors backlog documentation
vagimeli Jun 22, 2023
e56cab1
Update append.md
vagimeli Jun 22, 2023
0437b2b
Update searchable snapshot documentation to be more correct (#4203)
adaisley Jun 14, 2023
7e41042
Update plugins.md (#4353)
wbeckler Jun 16, 2023
698fff6
Add date nanoseconds field type (#4348)
kolchfa-aws Jun 19, 2023
3bb298b
Add model access control documentation for ML Commons (#4223)
kolchfa-aws Jun 19, 2023
bc02ea2
Update nested.md (#4363)
kolchfa-aws Jun 20, 2023
c1ddaa7
Reformat supported field types index page (#4349)
kolchfa-aws Jun 21, 2023
c30af78
Update links in field types index (#4379)
kolchfa-aws Jun 21, 2023
a2cf71e
updatestomigrationdoc (#4343)
hdhalter Jun 22, 2023
fe7943f
Writing processors backlog documentation
vagimeli Jun 22, 2023
47bbb69
Writing processors backlog documentation
vagimeli Jun 22, 2023
fabc15e
Writing processors backlog documentation
vagimeli Jun 22, 2023
2d7205e
Writing processors backlog documentation
vagimeli Jun 22, 2023
bc20602
Writing processors backlog documentation
vagimeli Jun 22, 2023
1827fae
Writing processors backlog documentation
vagimeli Jun 22, 2023
765c287
Writing processors backlog documentation
vagimeli Jun 22, 2023
417373a
Writing processors backlog documentation
vagimeli Jun 22, 2023
3e73879
Writing processors backlog documentation
vagimeli Jun 22, 2023
f499ddb
Writing processors backlog documentation
vagimeli Jun 22, 2023
1451142
Writing processors backlog documentation
vagimeli Jun 22, 2023
8605786
Writing processors backlog documentation
vagimeli Jun 22, 2023
f338778
Writing processors backlog documentation
vagimeli Jun 22, 2023
0fa8fd6
Revert "updatestomigrationdoc (#4343)"
vagimeli Jun 23, 2023
9f6913c
Revert "Update links in field types index (#4379)"
vagimeli Jun 23, 2023
44533ad
Revert "Reformat supported field types index page (#4349)"
vagimeli Jun 23, 2023
5756737
Revert "Update nested.md (#4363)"
vagimeli Jun 23, 2023
e50c25b
Revert "Add model access control documentation for ML Commons (#4223)"
vagimeli Jun 23, 2023
2a2821f
Revert "Add date nanoseconds field type (#4348)"
vagimeli Jun 23, 2023
05ae4aa
Revert "Update plugins.md (#4353)"
vagimeli Jun 23, 2023
d282c22
Revert "Update searchable snapshot documentation to be more correct (…
vagimeli Jun 23, 2023
8e88c6e
Update ingest-processors.md
vagimeli Jun 23, 2023
478bc16
Move files to separate PRs
vagimeli Jun 23, 2023
714e2ba
Move files to separate PRs
vagimeli Jun 23, 2023
a23449e
Update ingest-processors.md
vagimeli Jun 23, 2023
3495404
Update information architecture
vagimeli Jun 23, 2023
55cfa3b
Add first draft remove processor
vagimeli Jun 27, 2023
a907c5f
Add first draft remove processor
vagimeli Jun 27, 2023
6db6827
Address tech review feedback
vagimeli Jun 27, 2023
a89ca6a
Address tech review feedback
vagimeli Jun 27, 2023
7f7c807
Address tech review feedback
vagimeli Jun 27, 2023
2ae1cc4
Address tech review feedback
vagimeli Jun 27, 2023
ad8133b
Address tech review feedback
vagimeli Jun 27, 2023
ae1c7e8
Update _api-reference/ingest-apis/ingest-processors.md
vagimeli Jun 27, 2023
18c5d5e
Update _api-reference/ingest-apis/ingest-processors.md
vagimeli Jun 27, 2023
428bfd9
Update ingest-processors.md
vagimeli Jun 27, 2023
7c2a1d9
Update ingest-processors.md
vagimeli Jun 28, 2023
b0060da
Address tech review feedback
vagimeli Jun 27, 2023
85b57a7
Update pipeline example
vagimeli Jun 29, 2023
308ba02
Update pipeline example
vagimeli Jun 29, 2023
057a146
Update pipeline example
vagimeli Jun 29, 2023
1a60a13
Update pipeline example
vagimeli Jun 30, 2023
2d36887
Update pipeline example
vagimeli Jun 30, 2023
5fd0a64
Update pipeline example
vagimeli Jun 30, 2023
37fa2c0
Update pipeline example
vagimeli Jun 30, 2023
1b1535c
Update pipeline example
vagimeli Jun 30, 2023
807431c
Update pipeline example
vagimeli Jun 30, 2023
444123d
Update pipeline example
vagimeli Jul 3, 2023
9612019
Update pipeline example
vagimeli Jul 3, 2023
8c97658
Update pipeline example
vagimeli Jul 3, 2023
3f66760
Update append.md
vagimeli Jul 3, 2023
5dd3154
Copy edits
vagimeli Jul 5, 2023
56be338
Update _api-reference/ingest-apis/ingest-processors.md
vagimeli Jul 24, 2023
b48f124
Update _api-reference/ingest-apis/ingest-processors.md
vagimeli Jul 24, 2023
3825a13
Update _api-reference/ingest-apis/ingest-processors.md
vagimeli Jul 24, 2023
a3940bf
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
f907ec9
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
5bf0146
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
07c0776
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
d2e0aa8
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
0ca8746
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
5c14e8c
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
dc86e10
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
34bbd60
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
3ed7c02
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
ee1dbb5
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
badbb42
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
11814e1
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
71f249c
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
dedc3f7
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
a4b23af
Update _api-reference/ingest-apis/processors-reference/append.md
vagimeli Jul 24, 2023
86af726
Update _api-reference/ingest-apis/processors-reference/bytes.md
vagimeli Jul 24, 2023
3ea45e8
Update _api-reference/ingest-apis/processors-reference/bytes.md
vagimeli Jul 24, 2023
5bcf36e
Update _api-reference/ingest-apis/processors-reference/bytes.md
vagimeli Jul 24, 2023
41cce56
Update _api-reference/ingest-apis/processors-reference/bytes.md
vagimeli Jul 24, 2023
67a2c29
Update _api-reference/ingest-apis/processors-reference/bytes.md
vagimeli Jul 24, 2023
ef609b1
Update _api-reference/ingest-apis/processors-reference/convert.md
vagimeli Jul 24, 2023
6d4eae3
Update _api-reference/ingest-apis/processors-reference/convert.md
vagimeli Jul 24, 2023
392cef1
Update _api-reference/ingest-apis/processors-reference/convert.md
vagimeli Jul 24, 2023
97e0df5
Update _api-reference/ingest-apis/processors-reference/csv.md
vagimeli Jul 24, 2023
9b3fd4d
Update _api-reference/ingest-apis/processors-reference/csv.md
vagimeli Jul 24, 2023
07c78de
Update _api-reference/ingest-apis/processors-reference/date.md
vagimeli Jul 24, 2023
04b7f33
Update _api-reference/ingest-apis/processors-reference/date.md
vagimeli Jul 24, 2023
26de00d
Update _api-reference/ingest-apis/processors-reference/date.md
vagimeli Jul 24, 2023
523c441
Continue writing
vagimeli Jul 24, 2023
5166d1f
Continue writing
vagimeli Jul 24, 2023
26b0dc2
Continue writing
vagimeli Jul 25, 2023
f3dcef1
Continue writing
vagimeli Jul 26, 2023
67a4d50
Revised to include SME feedback
vagimeli Jul 31, 2023
ee0d394
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 1, 2023
dd17c5d
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 1, 2023
5276b90
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 1, 2023
91ef17b
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 1, 2023
ca51a56
Update _api-reference/ingest-apis/processors-reference/date.md
vagimeli Aug 1, 2023
1f1a0d4
Address SME and doc reviewer input
vagimeli Aug 1, 2023
6b28932
Add simulate content
vagimeli Aug 1, 2023
0385a7e
Writing
vagimeli Aug 1, 2023
3207dad
Writing
vagimeli Aug 1, 2023
ce5c972
Copy edits
vagimeli Aug 2, 2023
ea6110f
Writing
vagimeli Aug 2, 2023
cb908a9
Writing
vagimeli Aug 2, 2023
4d078ae
Revised content
vagimeli Aug 2, 2023
042204e
copy edit
vagimeli Aug 3, 2023
3a38b2e
Update file name
vagimeli Aug 3, 2023
4e4a932
Update file name
vagimeli Aug 3, 2023
f2622a8
Update file name
vagimeli Aug 3, 2023
41562a5
Update file name
vagimeli Aug 3, 2023
ec4a354
Update file name
vagimeli Aug 3, 2023
fdc6a1b
Update file name
vagimeli Aug 3, 2023
55861a8
Add bytes content and examples
vagimeli Aug 3, 2023
9591dcc
Add bytes content and examples
vagimeli Aug 3, 2023
ef4cb53
Testing code snippets and optional configuration parameters
vagimeli Aug 4, 2023
5aae19f
Testing code snippets and optional configuration parameters
vagimeli Aug 4, 2023
5f6dd0e
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 18, 2023
f91ccf5
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 18, 2023
94a86c3
QA pipeline testing
vagimeli Aug 22, 2023
05e3ed6
Update _api-reference/ingest-apis/pipeline-failures.md
vagimeli Aug 22, 2023
74fb7f1
Address SME comments
vagimeli Aug 22, 2023
96616d8
Add copy label
vagimeli Aug 22, 2023
88d3805
Update append.md
vagimeli Aug 22, 2023
4412bfc
Fix broken links
vagimeli Aug 23, 2023
3766c7b
Fix broken links
vagimeli Aug 23, 2023
dac1701
Update create-ingest.md
vagimeli Aug 23, 2023
17d4211
Update append.md
vagimeli Aug 23, 2023
417242c
Update bytes.md
vagimeli Aug 23, 2023
d95391c
Update convert.md
vagimeli Aug 23, 2023
b253f3a
Update convert.md
vagimeli Aug 23, 2023
28482a9
Update csv.md
vagimeli Aug 23, 2023
7b48521
Update date.md
vagimeli Aug 23, 2023
e51b1f7
Update lowercase.md
vagimeli Aug 23, 2023
26e7db0
Update remove.md
vagimeli Aug 23, 2023
06ad0e1
Update uppercase.md
vagimeli Aug 23, 2023
753ea67
Update append.md
vagimeli Aug 24, 2023
9548191
Update bytes.md
vagimeli Aug 24, 2023
2280597
Update convert.md
vagimeli Aug 24, 2023
1cda6ba
Update csv.md
vagimeli Aug 24, 2023
d7d3e68
Update date.md
vagimeli Aug 24, 2023
64d4d30
Update lowercase.md
vagimeli Aug 24, 2023
de5c1d0
Update remove.md
vagimeli Aug 24, 2023
9c76cc1
Update uppercase.md
vagimeli Aug 24, 2023
056ea7b
Address doc review feedback
vagimeli Aug 24, 2023
7e096db
Copy edits
vagimeli Aug 24, 2023
55357a8
Copy edits
vagimeli Aug 24, 2023
14294a3
Copy edits
vagimeli Aug 24, 2023
c879c9d
Copy edits
vagimeli Aug 24, 2023
9b467cc
Revise order of steps
vagimeli Aug 25, 2023
12347ec
Update processor steps
vagimeli Aug 25, 2023
a8f4ceb
Copy edits
vagimeli Aug 25, 2023
9d29c86
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 28, 2023
6fe30f5
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 28, 2023
a20ac44
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 28, 2023
fc95582
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 28, 2023
80eb62f
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 28, 2023
967bf9c
Update _api-reference/ingest-apis/create-ingest.md
vagimeli Aug 28, 2023
aafd5d0
Update _api-reference/ingest-apis/delete-ingest.md
vagimeli Aug 28, 2023
7f0a5ff
Update _api-reference/ingest-apis/index.md
vagimeli Aug 28, 2023
9402e7b
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 28, 2023
47a7040
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 28, 2023
65ba5e0
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 28, 2023
f531945
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 28, 2023
32846b1
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 28, 2023
904f386
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 28, 2023
f4add7c
Update _api-reference/ingest-apis/ingest-pipelines.md
vagimeli Aug 28, 2023
69a05d8
Update _api-reference/ingest-apis/ingest-processors.md
vagimeli Aug 28, 2023
c405e98
Update _api-reference/ingest-apis/pipeline-failures.md
vagimeli Aug 28, 2023
63effc5
Update _api-reference/ingest-apis/pipeline-failures.md
vagimeli Aug 28, 2023
1dbf2a6
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
baad585
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
ced1072
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
4065a79
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
761c6c6
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
c73de52
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
e3616bc
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
3f8e587
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
867a2ba
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
8a7e49f
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
1e235c7
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 28, 2023
fd30afb
Update _api-reference/ingest-apis/processors/bytes.md
vagimeli Aug 28, 2023
ae5ab17
Update _api-reference/ingest-apis/processors/bytes.md
vagimeli Aug 28, 2023
9985ddd
Update _api-reference/ingest-apis/processors/bytes.md
vagimeli Aug 28, 2023
b6b9f22
Update _api-reference/ingest-apis/processors/bytes.md
vagimeli Aug 28, 2023
e53d1cc
Address editorial feedback
vagimeli Aug 28, 2023
f588107
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
da0a4b6
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
07ac669
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
bf9b669
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
1aa34dc
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
a166ab1
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
e19c406
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
9ae3dc7
Address editorial feedback
vagimeli Aug 28, 2023
bde9ba9
Address editorial changes
vagimeli Aug 28, 2023
9e62879
Update _api-reference/ingest-apis/processors/convert.md
vagimeli Aug 28, 2023
be5056c
Update _api-reference/ingest-apis/processors/csv.md
vagimeli Aug 28, 2023
8212b0a
Update _api-reference/ingest-apis/processors/csv.md
vagimeli Aug 28, 2023
482b854
Update _api-reference/ingest-apis/processors/csv.md
vagimeli Aug 28, 2023
c9310ca
Update _api-reference/ingest-apis/processors/csv.md
vagimeli Aug 28, 2023
9967966
Update _api-reference/ingest-apis/processors/csv.md
vagimeli Aug 28, 2023
042207e
Update _api-reference/ingest-apis/processors/csv.md
vagimeli Aug 28, 2023
ca48924
Update _api-reference/ingest-apis/processors/csv.md
vagimeli Aug 28, 2023
c282647
Address editorial feedback
vagimeli Aug 28, 2023
2ade25c
Address editorial feedback
vagimeli Aug 28, 2023
60af276
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
ce1030c
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
2fc7870
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
eecc24a
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
88a4373
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
0e92ea4
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
e23e4d7
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
11a591e
Update _api-reference/ingest-apis/processors/date.md
vagimeli Aug 28, 2023
0f5b034
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
749a549
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
4beaca8
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
04f4fac
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
7ce2840
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
d3dc5cf
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
5c34ca7
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
3e3fa5b
Update _api-reference/ingest-apis/processors/lowercase.md
vagimeli Aug 28, 2023
eb258db
Address editorial feedback
vagimeli Aug 28, 2023
bf8e9a6
Update _api-reference/ingest-apis/processors/remove.md
vagimeli Aug 28, 2023
ecd54df
Update _api-reference/ingest-apis/processors/remove.md
vagimeli Aug 28, 2023
1853ec2
Update _api-reference/ingest-apis/processors/remove.md
vagimeli Aug 28, 2023
b528834
Update _api-reference/ingest-apis/processors/remove.md
vagimeli Aug 28, 2023
1a32fc9
Update _api-reference/ingest-apis/processors/remove.md
vagimeli Aug 28, 2023
ef80080
Update _api-reference/ingest-apis/processors/remove.md
vagimeli Aug 28, 2023
9ff0234
Address editorial review feedback
vagimeli Aug 28, 2023
98726c7
Update _api-reference/ingest-apis/processors/uppercase.md
vagimeli Aug 28, 2023
93d3028
Update _api-reference/ingest-apis/processors/uppercase.md
vagimeli Aug 28, 2023
d2e6261
Update _api-reference/ingest-apis/processors/uppercase.md
vagimeli Aug 28, 2023
115ca45
Update _api-reference/ingest-apis/processors/uppercase.md
vagimeli Aug 28, 2023
55b45ac
Update _api-reference/ingest-apis/processors/uppercase.md
vagimeli Aug 28, 2023
3869d73
Update _api-reference/ingest-apis/processors/uppercase.md
vagimeli Aug 28, 2023
477b5b5
Update _api-reference/ingest-apis/processors/uppercase.md
vagimeli Aug 28, 2023
2d6281a
Address editorial review feedback
vagimeli Aug 28, 2023
9445229
Address editorial review feedback
vagimeli Aug 28, 2023
f7d8a6d
Address editorial review feedback
vagimeli Aug 28, 2023
09bc013
Address editorial review feedback
vagimeli Aug 28, 2023
3cf22cd
Address editorial review feedback
vagimeli Aug 28, 2023
74f887b
Address editorial review feedback
vagimeli Aug 28, 2023
36f3c53
Address editorial review feedback
vagimeli Aug 28, 2023
7e9c29d
Update _api-reference/ingest-apis/processors/append.md
vagimeli Aug 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions _api-reference/ingest-apis/create-ingest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
layout: default
title: Create pipeline
parent: Ingest pipelines
grand_parent: Ingest APIs
nav_order: 10
redirect_from:
- /opensearch/rest-api/ingest-apis/create-update-ingest/
---

# Create pipeline

Use the create pipeline API operation to create or update pipelines in OpenSearch. Note that the pipeline requires you to define at least one processor that specifies how to change the documents.

## Path and HTTP method

Replace `<pipeline-id>` with your pipeline ID.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT _ingest/pipeline/<pipeline-id>
```
#### Example request

Here is an example in JSON format that creates an ingest pipeline with two `set` processors and an `uppercase` processor. The first `set` processor sets the the `grad_year` to `2023`, the second `set` processor sets `graduated` to `true`. The `uppercase` processor converts the `name` field to uppercase.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT _ingest/pipeline/my-pipeline
{
"description": "This pipeline processes student data",
"processors": [
{
"set": {
"description": "Sets the graduation year to 2023",
"field": "grad_year",
"value": 2023
}
},
{
"set": {
"description": "Sets graduated to true",
"field": "graduated",
"value": true
}
},
{
"uppercase": {
"field": "name"
}
}
]
}
```
{% include copy-curl.html %}

To learn more about error handling, see [Handling pipelines failures]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/pipeline-failures/).
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Request body fields

The following table lists the request body fields used to create, or update, a pipeline.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

Parameter | Required | Type | Description
:--- | :--- | :--- | :---
`processors` | Required | Array of processor objects | An array of processors, each of which transforms documents. Processors are run sequentially in the order specified.
`description` | Optional | String | Description of your ingest pipeline.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Path parameters

Parameter | Required | Type | Description
:--- | :--- | :--- | :---
`pipeline-id` | Required | String | The unique identifier, or pipeline ID, assigned to the ingest pipeline.

Parameter | Required | Type | Description
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
:--- | :--- | :--- | :---
`cluster_manager_timeout` | Optional | Time | Period to wait for a connection to the cluster manager node. Defaults to 30 seconds.
`timeout` | Optional | Time | Period to wait for a response. Defaults to 30 seconds.

## Template snippets

Some processor parameters support [Mustache](https://mustache.github.io/) template snippets. To get the value of a field, surround the field name in three curly braces, for example, `{% raw %}{{{field-name}}}{% endraw %}`.

#### Example: `set` ingest processor using Mustache template snippet

The following example sets the field `{% raw %}{{{role}}}{% endraw %}` with a value `{% raw %}{{{tenure}}}{% endraw %}`.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT _ingest/pipeline/my-pipeline
{
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
"processors": [
{
"set": {
"field": "{% raw %}{{{role}}}{% endraw %}",
"value": "{% raw %}{{{tenure}}}{% endraw %}"
}
}
]
}
```
{% include copy-curl.html %}
79 changes: 0 additions & 79 deletions _api-reference/ingest-apis/create-update-ingest.md

This file was deleted.

43 changes: 13 additions & 30 deletions _api-reference/ingest-apis/delete-ingest.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,27 @@
---
layout: default
title: Delete a pipeline
parent: Ingest APIs
nav_order: 14
title: Delete pipeline
parent: Ingest pipelines
grand_parent: Ingest APIs
nav_order: 13
redirect_from:
- /opensearch/rest-api/ingest-apis/delete-ingest/
---

# Delete a pipeline
# Delete pipeline

If you no longer want to use an ingest pipeline, use the delete ingest pipeline API operation.
Use the following request to delete pipelines.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Example
To delete a specific pipeline, pass the pipeline ID as a parameter:

```
DELETE _ingest/pipeline/12345
```json
DELETE /_ingest/pipeline/<pipeline-id>
```
{% include copy-curl.html %}

## Path and HTTP methods

Delete an ingest pipeline based on that pipeline's ID.

```
DELETE _ingest/pipeline/
```

## URL parameters

All URL parameters are optional.

Parameter | Type | Description
:--- | :--- | :---
master_timeout | time | How long to wait for a connection to the master node.
timeout | time | How long to wait for the request to return.

## Response
To delete all pipelines in a cluster, use the wildcard character (`*`):

```json
{
"acknowledged" : true
}
```
DELETE /_ingest/pipeline/*
```
{% include copy-curl.html %}
71 changes: 37 additions & 34 deletions _api-reference/ingest-apis/get-ingest.md
Original file line number Diff line number Diff line change
@@ -1,59 +1,62 @@
---
layout: default
title: Get ingest pipeline
parent: Ingest APIs
nav_order: 10
title: Get pipeline
parent: Ingest pipelines
grand_parent: Ingest APIs
nav_order: 12
redirect_from:
- /opensearch/rest-api/ingest-apis/get-ingest/
---

## Get ingest pipeline
# Get pipeline

After you create a pipeline, use the get ingest pipeline API operation to return all the information about a specific ingest pipeline.
Use the get ingest pipeline API operation to retrieve all the information about the pipeline.

## Example
## Retrieving information about all pipelines

```
GET _ingest/pipeline/12345
The following example request returns information about all ingest pipelines:

```json
GET _ingest/pipeline/
```
{% include copy-curl.html %}

## Path and HTTP methods
## Retrieving information about a specific pipeline

Return all ingest pipelines.
The following example request returns information about a specific pipeline, which for this example is `my-pipeline`:

```json
GET _ingest/pipeline/my-pipeline
```
GET _ingest/pipeline
```

Returns a single ingest pipeline based on the pipeline's ID.

```
GET _ingest/pipeline/{id}
```

## URL parameters

All parameters are optional.

Parameter | Type | Description
:--- | :--- | :---
master_timeout | time | How long to wait for a connection to the master node.
{% include copy-curl.html %}

## Response
The response contains the pipeline information:

```json
{
"pipeline-id" : {
"description" : "A description for your pipeline",
"processors" : [
"my-pipeline": {
"description": "This pipeline processes student data",
"processors": [
{
"set" : {
"field" : "field-name",
"value" : "value"
"set": {
"description": "Sets the graduation year to 2023",
"field": "grad_year",
"value": 2023
}
},
{
"set": {
"description": "Sets graduated to true",
"field": "graduated",
"value": true
}
},
{
"uppercase": {
"field": "name"
}
}
]
}
}
```
```
11 changes: 9 additions & 2 deletions _api-reference/ingest-apis/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@ redirect_from:

# Ingest APIs

Before you index your data, OpenSearch's ingest APIs help transform your data by creating and managing ingest pipelines. Pipelines consist of **processors**, customizable tasks that run in the order they appear in the request body. The transformed data appears in your index after each of the processor completes.
Ingest APIs are a valuable tool for loading data into a system. Ingest APIs work together with [ingest pipelines]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/ingest-pipelines/) and [ingest processors]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/ingest-processors/) to process or transform data from a variety of sources and in a variety of formats.

Ingest pipelines in OpenSearch can only be managed using ingest API operations. When using ingest in production environments, your cluster should contain at least one node with the node roles permission set to `ingest`. For more information on setting up node roles within a cluster, see [Cluster Formation]({{site.url}}{{site.baseurl}}/opensearch/cluster/).
## Ingest pipeline APIs

Simplify, secure, and scale your data ingestion in OpenSearch with the following APIs:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

- [Create pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/create-ingest/): Use this API to create or update a pipeline configuration.
- [Get pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/get-ingest/): Use this API to retrieve a pipeline configuration.
- [Simulate pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/simulate-ingest/): Use this pipeline to test a pipeline configuration.
- [Delete pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/delete-ingest/): Use this API to delete a pipeline configuration.
Loading