From a0ac70e51b82e32701970cc55fc490e0a8536a68 Mon Sep 17 00:00:00 2001 From: George Chen Date: Fri, 13 Oct 2023 12:49:33 -0500 Subject: [PATCH 1/7] ADD: extension docs in data-prepper-config Signed-off-by: George Chen --- .../configuring-data-prepper.md | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+) diff --git a/_data-prepper/managing-data-prepper/configuring-data-prepper.md b/_data-prepper/managing-data-prepper/configuring-data-prepper.md index b27ba8e49d..d807016f11 100644 --- a/_data-prepper/managing-data-prepper/configuring-data-prepper.md +++ b/_data-prepper/managing-data-prepper/configuring-data-prepper.md @@ -31,6 +31,7 @@ processorShutdownTimeout | No | Duration | The time given to processors to clear sinkShutdownTimeout | No | Duration | The time given to sinks to clear any in-flight data and gracefully shut down. Default is 30s. peer_forwarder | No | Object | Peer forwarder configurations. See [Peer forwarder options](#peer-forwarder-options) for more details. circuit_breakers | No | [circuit_breakers](#circuit-breakers) | Configures a circuit breaker on incoming data. +extensions | No | Object | The pipline extension plugin configurations. See [Extension plugins](#extension-plugins) for more details. ### Peer forwarder options @@ -100,3 +101,97 @@ usage | Yes | Bytes | Specifies the JVM heap usage at which to trip a circuit br reset | No | Duration | After tripping the circuit breaker, no new checks are made until after this time has passed. This effectively sets the minimum time for a breaker to remain open to allow for clearing memory. Defaults to `1s`. check_interval | No | Duration | Specifies the time between checks of the heap size. Defaults to `500ms`. +### Extension plugins + +Since 2.5, Data Prepper provides support for user configurable extension plugins. Extension plugins are shared common +configurations shared across pipeline plugins, i.e. [source/buffer/processor/sink](../index.md#data-prepper). + +#### AWS extension plugins + +Collection of AWS resource related extension plugins. All such plugin configuration objects are under `aws:` + +Option | Required | Type | Description +:--- |:---|:-------| :--- +aws | No | Object | The AWS extension plugins configuration. + +##### AWS secrets extension plugin + +Configures [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) extension plugin to be +referenced in pipeline plugin configurations, e.g. + +``` +extensions: + aws: + secrets: + : + secret_id: + region: + sts_role_arn: + refresh_interval: + : + ... +``` + +Option | Required | Type | Description +:--- |:---|:-------| :--- +secrets | No | Object | The AWS Secrets Manager extension plugin configuration. See [Secrets](#secrets) for details. + +###### Secrets + +Multiple secrets configuration objects can be defined with unique id for each. + +Option | Required | Type | Description +:--- |:---|:---------| :--- +secret_id | Yes | String | The AWS secret name or ARN. +region | No | String | The AWS region of the secret. Defaults to `us-east-1`. +sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). +refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. Defaults to `PT1H`. + +###### Reference secrets + +In `pipelines.yaml`, secret values can be referenced within pipeline plugins using the following formats: + +* plaintext: `${{aws_secrets:}}`. +* json (key-value pairs): `${{aws_secrets::}}` + +The secret value reference string format can be interpreted for the following plugin setting data types: + +* String +* Number +* Long +* Short +* Integer +* Double +* Float +* Boolean +* Character + +The following snippet in `pipelines.yaml` uses opensearch sink as an example + +``` +sink: + - opensearch: + hosts: [ "${{aws_secrets:host-secret-config}}" ] + username: "${{aws_secrets:credential-secret-config:username}}" + password: "${{aws_secrets:credential-secret-config:password}}" + index: "test-migration" +``` + +Note that the above snippet reference assumes plaintext secret value in `host-secret-config` and json key-value pairs in `credential-secret-config` with both `username` and `password` present as keys. +The corresponding AWS secrets extension plugin in `data-prepper-config.yaml` is as follows: + +``` +extensions: + aws: + secrets: + host-secret-config: + secret_id: + region: + sts_role_arn: + refresh_interval: + credential-secret-config: + secret_id: + region: + sts_role_arn: + refresh_interval: +``` \ No newline at end of file From 292895d09d83e974f425b4ee3d4308b6d3923389 Mon Sep 17 00:00:00 2001 From: George Chen Date: Fri, 13 Oct 2023 22:09:32 -0500 Subject: [PATCH 2/7] MAINT: updating secrets extension doc Signed-off-by: George Chen --- .../configuring-data-prepper.md | 35 +++++++++++-------- .../configuration/sources/opensearch.md | 4 +-- _data-prepper/pipelines/pipelines.md | 32 +++++++++++++++++ 3 files changed, 54 insertions(+), 17 deletions(-) diff --git a/_data-prepper/managing-data-prepper/configuring-data-prepper.md b/_data-prepper/managing-data-prepper/configuring-data-prepper.md index d807016f11..39d2cccf94 100644 --- a/_data-prepper/managing-data-prepper/configuring-data-prepper.md +++ b/_data-prepper/managing-data-prepper/configuring-data-prepper.md @@ -103,16 +103,16 @@ check_interval | No | Duration | Specifies the time between checks of the heap s ### Extension plugins -Since 2.5, Data Prepper provides support for user configurable extension plugins. Extension plugins are shared common -configurations shared across pipeline plugins, i.e. [source/buffer/processor/sink](../index.md#data-prepper). +Since Data Prepper 2.5, Data Prepper provides support for user configurable extension plugins. Extension plugins are shared common +configurations shared across pipeline plugins, i.e. [source/buffer/processor/sink](../index.md#concepts). #### AWS extension plugins Collection of AWS resource related extension plugins. All such plugin configuration objects are under `aws:` -Option | Required | Type | Description -:--- |:---|:-------| :--- -aws | No | Object | The AWS extension plugins configuration. +| Option | Required | Type | Description | +|:-------|:---------|:-------|:-----------------------------------------| +| aws | No | Object | The AWS extension plugins configuration. | ##### AWS secrets extension plugin @@ -132,20 +132,20 @@ extensions: ... ``` -Option | Required | Type | Description -:--- |:---|:-------| :--- -secrets | No | Object | The AWS Secrets Manager extension plugin configuration. See [Secrets](#secrets) for details. +| Option | Required | Type | Description | +|:--------|:---------|:-------|:---------------------------------------------------------------------------------------------| +| secrets | No | Object | The AWS Secrets Manager extension plugin configuration. See [Secrets](#secrets) for details. | ###### Secrets Multiple secrets configuration objects can be defined with unique id for each. -Option | Required | Type | Description -:--- |:---|:---------| :--- -secret_id | Yes | String | The AWS secret name or ARN. -region | No | String | The AWS region of the secret. Defaults to `us-east-1`. -sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). -refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. Defaults to `PT1H`. +| Option | Required | Type | Description | +|:-----------------|:---------|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| secret_id | Yes | String | The AWS secret name or ARN. | +| region | No | String | The AWS region of the secret. Defaults to `us-east-1`. | +| sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). | +| refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. See [Secrets refreshment](#secrets-refreshment) for details. Defaults to `PT1H`. | ###### Reference secrets @@ -194,4 +194,9 @@ extensions: region: sts_role_arn: refresh_interval: -``` \ No newline at end of file +``` + +###### Secrets refreshment + +For each individual secret configuration, the latest secret value is polled on a regular interval to support secrets refreshment in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, e.g. connection and authentication to the backend service. +For multiple secret configurations, jitter within 60s will be applied across them on the initial secrets polling. \ No newline at end of file diff --git a/_data-prepper/pipelines/configuration/sources/opensearch.md b/_data-prepper/pipelines/configuration/sources/opensearch.md index faa5b0b68b..fda99f1ce5 100644 --- a/_data-prepper/pipelines/configuration/sources/opensearch.md +++ b/_data-prepper/pipelines/configuration/sources/opensearch.md @@ -97,8 +97,8 @@ The following table describes options you can configure for the `opensearch` sou Option | Required | Type | Description :--- | :--- |:--------| :--- `hosts` | Yes | List | A list of OpenSearch hosts to write to, for example, `["https://localhost:9200", "https://remote-cluster:9200"]`. -`username` | No | String | The username for HTTP basic authentication. -`password` | No | String | The password for HTTP basic authentication. +`username` | No | String | The username for HTTP basic authentication. Since Data Prepper 2.5, this setting can be refreshed at runtime if [AWS secrets reference](../../../managing-data-prepper/configuring-data-prepper.md#reference-secrets) is applied. +`password` | No | String | The password for HTTP basic authentication. Since Data Prepper 2.5, this setting can be refreshed at runtime if [AWS secrets reference](../../../managing-data-prepper/configuring-data-prepper.md#reference-secrets) is applied. `disable_authentication` | No | Boolean | Whether authentication is disabled. Defaults to `false`. `aws` | No | Object | The AWS configuration. For more information, see [aws](#aws). `acknowledgments` | No | Boolean | When `true`, enables the `opensearch` source to receive [end-to-end acknowledgments]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/pipelines/#end-to-end-acknowledgments) when events are received by OpenSearch sinks. Default is `false`. diff --git a/_data-prepper/pipelines/pipelines.md b/_data-prepper/pipelines/pipelines.md index 50063079e7..7d9e56ddbe 100644 --- a/_data-prepper/pipelines/pipelines.md +++ b/_data-prepper/pipelines/pipelines.md @@ -326,3 +326,35 @@ peer_forwarder: ``` +# Pipeline Configurations + +Since Data Prepper 2.5, shared pipeline components can be configured under the reserved section `pipeline_configurations` when all pipelines are defined in a single pipeline configuration YAML file. +Shared pipeline configurations can include certain components within [Extension Plugins](../managing-data-prepper/configuring-data-prepper.md#extension-plugins), e.g. + +``` +pipeline_configurations: + aws: + secrets: + credential-secret-config: + secret_id: + region: + sts_role_arn: +simple-sample-pipeline: + ... + sink: + - opensearch: + hosts: [ "${{aws_secrets:host-secret-config}}" ] + username: "${{aws_secrets:credential-secret-config:username}}" + password: "${{aws_secrets:credential-secret-config:password}}" + index: "test-migration" +``` + +The above snippet in `pipelines.yaml` defines [AWS secrets extension plugin](../managing-data-prepper/configuring-data-prepper.md#aws-extension-plugins) within `pipeline_configurations` and refers to secrets configurations in opensearch sink. +Note that when the same component is defined in both `pipelines.yaml` and `data-prepper-config.yaml`, the definition in the `pipelines.yaml` will overwrite the counterpart in `data-prepper-config.yaml`. In the above example, `pipeline_configurations/aws/secrets` +in `pipelines.yaml` will overwrite `extensions/aws/secrets` if defined in `data-prepper-config.yaml`. + +The collection of shared pipeline components are as follows. + +## AWS secrets extension + +See [AWS secrets extension plugin](../managing-data-prepper/configuring-data-prepper.md#aws-secrets-extension-plugin) for details. \ No newline at end of file From 4cd2ade42876706b32d03cf4525781ad649ce3bf Mon Sep 17 00:00:00 2001 From: George Chen Date: Fri, 13 Oct 2023 22:31:57 -0500 Subject: [PATCH 3/7] MAINT: fix links Signed-off-by: George Chen --- _data-prepper/pipelines/configuration/sources/opensearch.md | 4 ++-- _data-prepper/pipelines/pipelines.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/opensearch.md b/_data-prepper/pipelines/configuration/sources/opensearch.md index fda99f1ce5..6056d10943 100644 --- a/_data-prepper/pipelines/configuration/sources/opensearch.md +++ b/_data-prepper/pipelines/configuration/sources/opensearch.md @@ -97,8 +97,8 @@ The following table describes options you can configure for the `opensearch` sou Option | Required | Type | Description :--- | :--- |:--------| :--- `hosts` | Yes | List | A list of OpenSearch hosts to write to, for example, `["https://localhost:9200", "https://remote-cluster:9200"]`. -`username` | No | String | The username for HTTP basic authentication. Since Data Prepper 2.5, this setting can be refreshed at runtime if [AWS secrets reference](../../../managing-data-prepper/configuring-data-prepper.md#reference-secrets) is applied. -`password` | No | String | The password for HTTP basic authentication. Since Data Prepper 2.5, this setting can be refreshed at runtime if [AWS secrets reference](../../../managing-data-prepper/configuring-data-prepper.md#reference-secrets) is applied. +`username` | No | String | The username for HTTP basic authentication. Since Data Prepper 2.5, this setting can be refreshed at runtime if [AWS secrets reference]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#reference-secrets) is applied. +`password` | No | String | The password for HTTP basic authentication. Since Data Prepper 2.5, this setting can be refreshed at runtime if [AWS secrets reference]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#reference-secrets) is applied. `disable_authentication` | No | Boolean | Whether authentication is disabled. Defaults to `false`. `aws` | No | Object | The AWS configuration. For more information, see [aws](#aws). `acknowledgments` | No | Boolean | When `true`, enables the `opensearch` source to receive [end-to-end acknowledgments]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/pipelines/#end-to-end-acknowledgments) when events are received by OpenSearch sinks. Default is `false`. diff --git a/_data-prepper/pipelines/pipelines.md b/_data-prepper/pipelines/pipelines.md index 7d9e56ddbe..e104d9fa09 100644 --- a/_data-prepper/pipelines/pipelines.md +++ b/_data-prepper/pipelines/pipelines.md @@ -329,7 +329,7 @@ peer_forwarder: # Pipeline Configurations Since Data Prepper 2.5, shared pipeline components can be configured under the reserved section `pipeline_configurations` when all pipelines are defined in a single pipeline configuration YAML file. -Shared pipeline configurations can include certain components within [Extension Plugins](../managing-data-prepper/configuring-data-prepper.md#extension-plugins), e.g. +Shared pipeline configurations can include certain components within [Extension Plugins]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#extension-plugins), e.g. ``` pipeline_configurations: @@ -349,7 +349,7 @@ simple-sample-pipeline: index: "test-migration" ``` -The above snippet in `pipelines.yaml` defines [AWS secrets extension plugin](../managing-data-prepper/configuring-data-prepper.md#aws-extension-plugins) within `pipeline_configurations` and refers to secrets configurations in opensearch sink. +The above snippet in `pipelines.yaml` defines [AWS secrets extension plugin]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#aws-extension-plugins) within `pipeline_configurations` and refers to secrets configurations in opensearch sink. Note that when the same component is defined in both `pipelines.yaml` and `data-prepper-config.yaml`, the definition in the `pipelines.yaml` will overwrite the counterpart in `data-prepper-config.yaml`. In the above example, `pipeline_configurations/aws/secrets` in `pipelines.yaml` will overwrite `extensions/aws/secrets` if defined in `data-prepper-config.yaml`. @@ -357,4 +357,4 @@ The collection of shared pipeline components are as follows. ## AWS secrets extension -See [AWS secrets extension plugin](../managing-data-prepper/configuring-data-prepper.md#aws-secrets-extension-plugin) for details. \ No newline at end of file +See [AWS secrets extension plugin]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#aws-secrets-extension-plugin) for details. \ No newline at end of file From 1b9dd1898fe809311a94dc656cfaf4efa874a8ca Mon Sep 17 00:00:00 2001 From: George Chen Date: Fri, 13 Oct 2023 22:37:42 -0500 Subject: [PATCH 4/7] MAINT: fix one more dead link Signed-off-by: George Chen --- _data-prepper/managing-data-prepper/configuring-data-prepper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/managing-data-prepper/configuring-data-prepper.md b/_data-prepper/managing-data-prepper/configuring-data-prepper.md index 39d2cccf94..d9a48eaeee 100644 --- a/_data-prepper/managing-data-prepper/configuring-data-prepper.md +++ b/_data-prepper/managing-data-prepper/configuring-data-prepper.md @@ -104,7 +104,7 @@ check_interval | No | Duration | Specifies the time between checks of the heap s ### Extension plugins Since Data Prepper 2.5, Data Prepper provides support for user configurable extension plugins. Extension plugins are shared common -configurations shared across pipeline plugins, i.e. [source/buffer/processor/sink](../index.md#concepts). +configurations shared across pipeline plugins, i.e. [source/buffer/processor/sink]({{site.url}}{{site.baseurl}}/data-prepper/index/#concepts). #### AWS extension plugins From 09e9f33857de828cf781ca57aada64b3e4b664d8 Mon Sep 17 00:00:00 2001 From: George Chen Date: Tue, 24 Oct 2023 10:53:24 -0500 Subject: [PATCH 5/7] MAINT: renaming Signed-off-by: George Chen --- .../configuring-data-prepper.md | 14 +++++++------- .../configuration/processors/rename-keys.md | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/_data-prepper/managing-data-prepper/configuring-data-prepper.md b/_data-prepper/managing-data-prepper/configuring-data-prepper.md index d9a48eaeee..9c01311090 100644 --- a/_data-prepper/managing-data-prepper/configuring-data-prepper.md +++ b/_data-prepper/managing-data-prepper/configuring-data-prepper.md @@ -31,7 +31,7 @@ processorShutdownTimeout | No | Duration | The time given to processors to clear sinkShutdownTimeout | No | Duration | The time given to sinks to clear any in-flight data and gracefully shut down. Default is 30s. peer_forwarder | No | Object | Peer forwarder configurations. See [Peer forwarder options](#peer-forwarder-options) for more details. circuit_breakers | No | [circuit_breakers](#circuit-breakers) | Configures a circuit breaker on incoming data. -extensions | No | Object | The pipline extension plugin configurations. See [Extension plugins](#extension-plugins) for more details. +extensions | No | Object | The pipeline extension plugin configurations. See [Extension plugins](#extension-plugins) for more details. ### Peer forwarder options @@ -145,16 +145,16 @@ Multiple secrets configuration objects can be defined with unique id for each. | secret_id | Yes | String | The AWS secret name or ARN. | | region | No | String | The AWS region of the secret. Defaults to `us-east-1`. | | sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). | -| refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. See [Secrets refreshment](#secrets-refreshment) for details. Defaults to `PT1H`. | +| refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. See [Secrets refreshment](#automatically-refreshing-secrets) for details. Defaults to `PT1H`. | ###### Reference secrets In `pipelines.yaml`, secret values can be referenced within pipeline plugins using the following formats: * plaintext: `${{aws_secrets:}}`. -* json (key-value pairs): `${{aws_secrets::}}` +* JSON (key-value pairs): `${{aws_secrets::}}` -The secret value reference string format can be interpreted for the following plugin setting data types: +Note that `` should be replaced with the corresponding secret config ID under `/extensions/aws/secrets` and `` should be replaced with the desired key in the secret JSON value. The secret value reference string format can be interpreted for the following plugin setting data types: * String * Number @@ -166,7 +166,7 @@ The secret value reference string format can be interpreted for the following pl * Boolean * Character -The following snippet in `pipelines.yaml` uses opensearch sink as an example +The following snippet in `pipelines.yaml` uses an OpenSearch sink as an example ``` sink: @@ -196,7 +196,7 @@ extensions: refresh_interval: ``` -###### Secrets refreshment +###### Automatically refreshing secrets -For each individual secret configuration, the latest secret value is polled on a regular interval to support secrets refreshment in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, e.g. connection and authentication to the backend service. +For each individual secret configuration, the latest secret value is polled on a regular interval to support refreshing secrets in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, e.g. connection and authentication to the backend service. For multiple secret configurations, jitter within 60s will be applied across them on the initial secrets polling. \ No newline at end of file diff --git a/_data-prepper/pipelines/configuration/processors/rename-keys.md b/_data-prepper/pipelines/configuration/processors/rename-keys.md index d2c892d745..f57b4e509f 100644 --- a/_data-prepper/pipelines/configuration/processors/rename-keys.md +++ b/_data-prepper/pipelines/configuration/processors/rename-keys.md @@ -64,7 +64,7 @@ When you run the `rename_keys` processor, it parses the message into the followi ## Special considerations -Renaming operations occur in the order that the key-value pair entries are listed in the `pipeline.yaml` file. This means that chaining (where key-value pairs are renamed in sequence) is implicit in the `rename_keys` processor. See the following example `pipline.yaml` file: +Renaming operations occur in the order that the key-value pair entries are listed in the `pipeline.yaml` file. This means that chaining (where key-value pairs are renamed in sequence) is implicit in the `rename_keys` processor. See the following example `pipeline.yaml` file: ```yaml pipeline: From 02480554715eb84d1f960df94e4af1aff6513f3b Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 3 Nov 2023 11:59:21 -0500 Subject: [PATCH 6/7] Update configuring-data-prepper.md --- .../configuring-data-prepper.md | 86 ++++++++++--------- 1 file changed, 46 insertions(+), 40 deletions(-) diff --git a/_data-prepper/managing-data-prepper/configuring-data-prepper.md b/_data-prepper/managing-data-prepper/configuring-data-prepper.md index 9c01311090..0c91b37e2c 100644 --- a/_data-prepper/managing-data-prepper/configuring-data-prepper.md +++ b/_data-prepper/managing-data-prepper/configuring-data-prepper.md @@ -104,22 +104,22 @@ check_interval | No | Duration | Specifies the time between checks of the heap s ### Extension plugins Since Data Prepper 2.5, Data Prepper provides support for user configurable extension plugins. Extension plugins are shared common -configurations shared across pipeline plugins, i.e. [source/buffer/processor/sink]({{site.url}}{{site.baseurl}}/data-prepper/index/#concepts). +configurations shared across pipeline plugins, such as [sources, buffers, processors, and sinks]({{site.url}}{{site.baseurl}}/data-prepper/index/#concepts). -#### AWS extension plugins +### AWS extension plugins -Collection of AWS resource related extension plugins. All such plugin configuration objects are under `aws:` +To use the AWS extension plugin, add the following setting to your `data-prepper-config.yaml` under `aws`. -| Option | Required | Type | Description | -|:-------|:---------|:-------|:-----------------------------------------| -| aws | No | Object | The AWS extension plugins configuration. | +Option | Required | Type | Description +:--- |:---|:---| :--- +aws | No | Object | The AWS extension plugins configuration. -##### AWS secrets extension plugin +#### AWS secrets extension plugin -Configures [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) extension plugin to be -referenced in pipeline plugin configurations, e.g. +The AWS secrets extension plugin configures the [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) to be +referenced in pipeline plugin configurations, as shown in the following example: -``` +```json extensions: aws: secrets: @@ -132,29 +132,33 @@ extensions: ... ``` -| Option | Required | Type | Description | -|:--------|:---------|:-------|:---------------------------------------------------------------------------------------------| -| secrets | No | Object | The AWS Secrets Manager extension plugin configuration. See [Secrets](#secrets) for details. | +To use the secrets extension plugin, add the following setting to your `pipeline.yaml` under `extensions` > `aws`. + +Option | Required | Type | Description +:--- |:---|:---| :--- +secrets | No | Object | The AWS Secrets Manager extension plugin configuration. See [Secrets](#secrets) for more details. -###### Secrets +### Secrets -Multiple secrets configuration objects can be defined with unique id for each. +Use the following settings under the `secrets` extension setting. -| Option | Required | Type | Description | -|:-----------------|:---------|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| secret_id | Yes | String | The AWS secret name or ARN. | -| region | No | String | The AWS region of the secret. Defaults to `us-east-1`. | -| sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). | -| refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. See [Secrets refreshment](#automatically-refreshing-secrets) for details. Defaults to `PT1H`. | -###### Reference secrets +Option | Required | Type | Description +:--- |:---|:---| :--- +secret_id | Yes | String | The AWS secret name or ARN. | +region | No | String | The AWS region of the secret. Defaults to `us-east-1`. +sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to the AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). +refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. Defaults to `PT1H`. See [Automatically refreshing secrets](#automatically-refreshing-secrets) for details. + +#### Reference secrets -In `pipelines.yaml`, secret values can be referenced within pipeline plugins using the following formats: +In `pipelines.yaml`, secret values can be referenced within the pipeline plugins using the following formats: * plaintext: `${{aws_secrets:}}`. * JSON (key-value pairs): `${{aws_secrets::}}` -Note that `` should be replaced with the corresponding secret config ID under `/extensions/aws/secrets` and `` should be replaced with the desired key in the secret JSON value. The secret value reference string format can be interpreted for the following plugin setting data types: + +Replace `` with the corresponding secret config ID under `/extensions/aws/secrets`. Replace `` with the desired key in the secret JSON value. The secret value reference string format can be interpreted for the following plugin setting data types: * String * Number @@ -166,21 +170,10 @@ Note that `` should be replaced with the corresponding se * Boolean * Character -The following snippet in `pipelines.yaml` uses an OpenSearch sink as an example +The following example section of `data-prepper-config.yaml` names two secret config IDs, `host-secret-config` and `credential-secret-config`: -``` -sink: - - opensearch: - hosts: [ "${{aws_secrets:host-secret-config}}" ] - username: "${{aws_secrets:credential-secret-config:username}}" - password: "${{aws_secrets:credential-secret-config:password}}" - index: "test-migration" -``` - -Note that the above snippet reference assumes plaintext secret value in `host-secret-config` and json key-value pairs in `credential-secret-config` with both `username` and `password` present as keys. -The corresponding AWS secrets extension plugin in `data-prepper-config.yaml` is as follows: -``` +```json extensions: aws: secrets: @@ -196,7 +189,20 @@ extensions: refresh_interval: ``` -###### Automatically refreshing secrets +After `` is configured, you can reference the IDs in your `pipelines.yaml`: + +``` +sink: + - opensearch: + hosts: [ "${{aws_secrets:host-secret-config}}" ] + username: "${{aws_secrets:credential-secret-config:username}}" + password: "${{aws_secrets:credential-secret-config:password}}" + index: "test-migration" +``` + + +#### Automatically refreshing secrets + +For each individual secret configuration, the latest secret value is polled on a regular interval to support refreshing secrets in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, such as connection and authentication to the backend service. -For each individual secret configuration, the latest secret value is polled on a regular interval to support refreshing secrets in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, e.g. connection and authentication to the backend service. -For multiple secret configurations, jitter within 60s will be applied across them on the initial secrets polling. \ No newline at end of file +For multiple secret configurations, jitter within `60s` will be applied across all configurations during the initial secrets polling. From 543fde75bc648bb25d6d0641f404a3a5bc8c54b1 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 3 Nov 2023 12:04:00 -0500 Subject: [PATCH 7/7] Update pipelines.md --- _data-prepper/pipelines/pipelines.md | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/_data-prepper/pipelines/pipelines.md b/_data-prepper/pipelines/pipelines.md index e104d9fa09..87c8ce5755 100644 --- a/_data-prepper/pipelines/pipelines.md +++ b/_data-prepper/pipelines/pipelines.md @@ -326,12 +326,12 @@ peer_forwarder: ``` -# Pipeline Configurations +## Pipeline Configurations Since Data Prepper 2.5, shared pipeline components can be configured under the reserved section `pipeline_configurations` when all pipelines are defined in a single pipeline configuration YAML file. -Shared pipeline configurations can include certain components within [Extension Plugins]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#extension-plugins), e.g. +Shared pipeline configurations can include certain components within [Extension Plugins]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#extension-plugins), as shown in the following example that refers to secrets configurations for an `opensearch` sink: -``` +```json pipeline_configurations: aws: secrets: @@ -349,12 +349,4 @@ simple-sample-pipeline: index: "test-migration" ``` -The above snippet in `pipelines.yaml` defines [AWS secrets extension plugin]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#aws-extension-plugins) within `pipeline_configurations` and refers to secrets configurations in opensearch sink. -Note that when the same component is defined in both `pipelines.yaml` and `data-prepper-config.yaml`, the definition in the `pipelines.yaml` will overwrite the counterpart in `data-prepper-config.yaml`. In the above example, `pipeline_configurations/aws/secrets` -in `pipelines.yaml` will overwrite `extensions/aws/secrets` if defined in `data-prepper-config.yaml`. - -The collection of shared pipeline components are as follows. - -## AWS secrets extension - -See [AWS secrets extension plugin]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#aws-secrets-extension-plugin) for details. \ No newline at end of file +When the same component is defined in both `pipelines.yaml` and `data-prepper-config.yaml`, the definition in the `pipelines.yaml` will overwrite the counterpart in `data-prepper-config.yaml`. For more information on shared pipeline components, see [AWS secrets extension plugin]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#aws-secrets-extension-plugin) for details.