[DOC] data prepper secret extensions #5202
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: George Chen <[email protected]>
Signed-off-by: George Chen <[email protected]>
chenqi0805
requested review from
hdhalter,
kolchfa-aws,
Naarcha-AWS,
vagimeli,
ananzh,
seanneumann,
AMoo-Miki and
natebower
as code owners
Signed-off-by: George Chen <[email protected]>
Signed-off-by: George Chen <[email protected]>
Signed-off-by: George Chen <[email protected]>
dlvenable
requested changes
Oct 24, 2023
| | Option | Required | Type | Description | | ||
| |:-----------------|:---------|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | secret_id | Yes | String | The AWS secret name or ARN. | | ||
| | region | No | String | The AWS region of the secret. Defaults to `us-east-1`. | |
There was a problem hiding this comment.
@chenqi0805 , Does this default to us-east-1? I'd think it uses the AWS SDK's default. This can be the AWS_REGION environment variable or us-east-1. Do we know for sure?
There was a problem hiding this comment.
Yes. We do default to us-east-1 the same way as the S3 DLQ in opensearch sink
| refresh_interval: <YOUR_REFRESH_INTERVAL_2> | ||
| ``` | ||
|
|
||
| ###### Secrets refreshment |
There was a problem hiding this comment.
Let's not use "refreshment" here. A drink may be a refreshment. :)
I'd prefer "Refreshing secrets" or "Automatically refreshing secrets".
|
|
||
| ###### Secrets refreshment | ||
|
|
||
| For each individual secret configuration, the latest secret value is polled on a regular interval to support secrets refreshment in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, e.g. connection and authentication to the backend service. |
There was a problem hiding this comment.
support secrets refreshment in
change to
support refreshing secrets in
| | secret_id | Yes | String | The AWS secret name or ARN. | | ||
| | region | No | String | The AWS region of the secret. Defaults to `us-east-1`. | | ||
| | sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). | | ||
| | refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. See [Secrets refreshment](#secrets-refreshment) for details. Defaults to `PT1H`. | |
There was a problem hiding this comment.
See my note below. Let's change the link from "Secrets refreshment" to "Automatically refreshing secrets"
|
|
||
| In `pipelines.yaml`, secret values can be referenced within pipeline plugins using the following formats: | ||
|
|
||
| * plaintext: `${{aws_secrets:<YOUR_SECRET_CONFIG_ID>}}`. |
There was a problem hiding this comment.
We should be clearly state in the preceding paragraph that users need to replace <YOUR_SECRET_CONFIG_ID> including the <>.
Signed-off-by: George Chen <[email protected]>
Naarcha-AWS
approved these changes
Nov 3, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Nov 3, 2023
* ADD: extension docs in data-prepper-config Signed-off-by: George Chen <[email protected]> * MAINT: updating secrets extension doc Signed-off-by: George Chen <[email protected]> * MAINT: fix links Signed-off-by: George Chen <[email protected]> * MAINT: fix one more dead link Signed-off-by: George Chen <[email protected]> * MAINT: renaming Signed-off-by: George Chen <[email protected]> * Update configuring-data-prepper.md * Update pipelines.md --------- Signed-off-by: George Chen <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> (cherry picked from commit 9617548) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Naarcha-AWS
added a commit
that referenced
this pull request
Nov 3, 2023
* ADD: extension docs in data-prepper-config * MAINT: updating secrets extension doc * MAINT: fix links * MAINT: fix one more dead link * MAINT: renaming * Update configuring-data-prepper.md * Update pipelines.md --------- (cherry picked from commit 9617548) Signed-off-by: George Chen <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Naarcha-AWS <[email protected]>
vagimeli
pushed a commit
that referenced
this pull request
Dec 21, 2023
* ADD: extension docs in data-prepper-config Signed-off-by: George Chen <[email protected]> * MAINT: updating secrets extension doc Signed-off-by: George Chen <[email protected]> * MAINT: fix links Signed-off-by: George Chen <[email protected]> * MAINT: fix one more dead link Signed-off-by: George Chen <[email protected]> * MAINT: renaming Signed-off-by: George Chen <[email protected]> * Update configuring-data-prepper.md * Update pipelines.md --------- Signed-off-by: George Chen <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds Data Prepper 2.5 documentation on AWS secrets extension plugin details into both data-prepper-config YAML and pipelines configuration YAML.
Issues Resolved
Closes #5115 .
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.