Add DLQ to OpenSearch sink configuration options. #3743

carolxob · 2023-04-11T16:38:12Z

Description

Adds DLQ to OpenSearch sink configuration options.

Issues Resolved

No affected issues.

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: carolxob <[email protected]>

cmanning09 · 2023-04-17T15:21:28Z

We will update the link provided to the documentation website when we add the DLQ plugin, right?

lizsnyder · 2023-04-20T23:53:34Z

@Naarcha-AWS can we get this approved and merged? I need to reference it in my docs.

_data-prepper/pipelines/configuration/sinks/opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

hdhalter · 2023-04-26T14:32:48Z

_data-prepper/pipelines/configuration/sinks/opensearch.md

@@ -68,7 +68,8 @@ index | Conditionally | String | Name of the export index. Applicable and requir
 index_type | No | String | This index type tells the Sink plugin what type of data it is handling. Valid values: `custom`, `trace-analytics-raw`, `trace-analytics-service-map`, `management-disabled`. Default value is `custom`.
 template_file | No | String | Path to a JSON [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/) file (for example, `/your/local/template-file.json`) if `index_type` is `custom`. See [otel-v1-apm-span-index-template.json](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/otel-v1-apm-span-index-template.json) for an example.
 document_id_field | No | String | The field from the source data to use for the OpenSearch document ID (for example, `"my-field"`) if `index_type` is `custom`.
-dlq_file | No | String | The path to your preferred dead letter queue (DLQ) file (for example, `/your/local/dlq-file`). Data Prepper writes to this file when it fails to index a document on the OpenSearch cluster.
+dlq_file | No | String | The path to your preferred dead letter queue file (for example, `/your/local/dlq-file`). Data Prepper writes to this file when it fails to index a document on the OpenSearch cluster.
+dlq | No | N/A | DLQ configurations. See [Dead Letter Queues (DLQ)](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/failures-common/src/main/java/org/opensearch/dataprepper/plugins/dlq/README.md) for details. If the `dlq_file` option is also available, the sink will fail.


small nit: "...the sink fails."

* Add DLQ to OpenSearch sink configuration options. Signed-off-by: carolxob <[email protected]> * Update _data-prepper/pipelines/configuration/sinks/opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: carolxob <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> (cherry picked from commit 66f30d7) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add DLQ to OpenSearch sink configuration options. * Update _data-prepper/pipelines/configuration/sinks/opensearch.md --------- (cherry picked from commit 66f30d7) Signed-off-by: carolxob <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Naarcha-AWS <[email protected]>

vagimeli · 2023-04-26T22:03:01Z

_data-prepper/pipelines/configuration/sinks/opensearch.md

@@ -68,7 +68,8 @@ index | Conditionally | String | Name of the export index. Applicable and requir
 index_type | No | String | This index type tells the Sink plugin what type of data it is handling. Valid values: `custom`, `trace-analytics-raw`, `trace-analytics-service-map`, `management-disabled`. Default value is `custom`.
 template_file | No | String | Path to a JSON [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/) file (for example, `/your/local/template-file.json`) if `index_type` is `custom`. See [otel-v1-apm-span-index-template.json](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/otel-v1-apm-span-index-template.json) for an example.
 document_id_field | No | String | The field from the source data to use for the OpenSearch document ID (for example, `"my-field"`) if `index_type` is `custom`.
-dlq_file | No | String | The path to your preferred dead letter queue (DLQ) file (for example, `/your/local/dlq-file`). Data Prepper writes to this file when it fails to index a document on the OpenSearch cluster.
+dlq_file | No | String | The path to your preferred dead letter queue file (for example, `/your/local/dlq-file`). Data Prepper writes to this file when it fails to index a document on the OpenSearch cluster.
+dlq | No | N/A | DLQ configurations. See [Dead Letter Queues (DLQ)](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/failures-common/src/main/java/org/opensearch/dataprepper/plugins/dlq/README.md) for details. If the `dlq_file` option is also available, the sink will fail.
 bulk_size | No | Integer (long) | The maximum size (in MiB) of bulk requests sent to the OpenSearch cluster. Values below 0 indicate an unlimited size. If a single document exceeds the maximum bulk request size, Data Prepper sends it individually. Default value is 5.
 ism_policy_file | No | String | The absolute file path for an ISM (Index State Management) policy JSON file. This policy file is effective only when there is no built-in policy file for the index type. For example, `custom` index type is currently the only one without a built-in policy file, thus it would use the policy file here if it's provided through this parameter. For more information, see [ISM policies]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/).
 number_of_shards | No | Integer | The number of primary shards that an index should have on the destination OpenSearch server. This parameter is effective only when `template_file` is either explicitly provided in Sink configuration or built-in. If this parameter is set, it would override the value in index template file. For more information, see [Create index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/).


Should "Sink" be capitalized?

"...it would override the value in the index template file.

* Add DLQ to OpenSearch sink configuration options. Signed-off-by: carolxob <[email protected]> * Update _data-prepper/pipelines/configuration/sinks/opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: carolxob <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]>

This reverts commit a070189.

…#3743) * Add DLQ to OpenSearch sink configuration options. Signed-off-by: carolxob <[email protected]> * Update _data-prepper/pipelines/configuration/sinks/opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: carolxob <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]>

Add DLQ to OpenSearch sink configuration options.

0a5707d

Signed-off-by: carolxob <[email protected]>

carolxob assigned carolxob and Naarcha-AWS and unassigned carolxob Apr 11, 2023

cmanning09 approved these changes Apr 17, 2023

View reviewed changes

lizsnyder approved these changes Apr 20, 2023

View reviewed changes

carolxob removed their assignment Apr 25, 2023

Naarcha-AWS reviewed Apr 26, 2023

View reviewed changes

_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved

Update _data-prepper/pipelines/configuration/sinks/opensearch.md

d8ee509

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS approved these changes Apr 26, 2023

View reviewed changes

Merge branch 'main' into opensearch-sink-s3

18b07a1

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS removed backport 2.0 PR: Backport label for v2.0.x backport 2.1 PR: Backport label for 2.1 backport 2.2 PR: Backport label for 2.2 backport 2.3 PR: Backport label for 2.3 backport 2.4 PR: Backport label for 2.4 labels Apr 26, 2023

hdhalter approved these changes Apr 26, 2023

View reviewed changes

Naarcha-AWS merged commit 66f30d7 into main Apr 26, 2023

Naarcha-AWS deleted the opensearch-sink-s3 branch April 26, 2023 16:36

This was referenced Apr 26, 2023

[Backport 2.5] Add DLQ to OpenSearch sink configuration options. #3878

Merged

[Backport 2.6] Add DLQ to OpenSearch sink configuration options. #3879

Merged

vagimeli reviewed Apr 26, 2023

View reviewed changes

vagimeli added a commit that referenced this pull request May 4, 2023

Revert "Add DLQ to OpenSearch sink configuration options. (#3743)"

1962d2f

This reverts commit a070189.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DLQ to OpenSearch sink configuration options. #3743

Add DLQ to OpenSearch sink configuration options. #3743

carolxob commented Apr 11, 2023

cmanning09 commented Apr 17, 2023

lizsnyder commented Apr 20, 2023

hdhalter Apr 26, 2023

vagimeli Apr 26, 2023

Add DLQ to OpenSearch sink configuration options. #3743

Add DLQ to OpenSearch sink configuration options. #3743

Conversation

carolxob commented Apr 11, 2023

Description

Issues Resolved

Checklist

cmanning09 commented Apr 17, 2023

lizsnyder commented Apr 20, 2023

hdhalter Apr 26, 2023

Choose a reason for hiding this comment

vagimeli Apr 26, 2023

Choose a reason for hiding this comment