Data Prepper documentation for the SNS to SQS fan out pattern in S3 #3915
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pper. Signed-off-by: David Venable <[email protected]>
dlvenable
requested review from
cwillum,
hdhalter,
kolchfa-aws,
Naarcha-AWS,
vagimeli,
ananzh,
seanneumann,
AMoo-Miki and
natebower
as code owners
|
Looking over this now. |
Naarcha-AWS
approved these changes
May 5, 2023
dlvenable
commented
May 5, 2023
|
|
||
| To meet the scale of logs produced in S3, some users require multiple SQS queues for their logs. You can use the [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications go directly to a single SNS topic, where you can subscribe to multiple SQS queues. | ||
|
|
||
| To make sure that Data Prepper can parse the event from the SNS topic directly, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. If you want Data Prepper to ignore an SQS queue that sends notifications to your SNS topic, do not enable raw message delivery those queues. |
There was a problem hiding this comment.
If you want Data Prepper to ignore an SQS queue
I don't think this properly conveys the situation. Data Prepper isn't ignoring these SQS queues. They are not being used at all by Data Prepper. Also, the user can configure those other queues to use raw message delivery or not. It doesn't affect Data Prepper and Data Prepper doesn't affect them.
The point of my original wording was to help with the objection that setting raw message delivery will affect other consumers.
There was a problem hiding this comment.
How about this?
"When raw message delivery is not enabled on an SQS queue, Data Prepper does not use that SQS queue."
There was a problem hiding this comment.
I'm still not sure if that really conveys what I'm trying to say here.
This is my original text:
The raw message delivery option is on the connection between SNS and SQS.
Thus, you can have some SQS queues which do not enable raw message delivery.
Using Data Prepper will not interfere with those queues.
Perhaps we can simply say:
Setting this option will not affect other SQS queues which are subscribed to that SNS topic.
Naarcha-AWS
added
3 - Tech review
Naarcha-AWS
added
5 - Editorial review
natebower
approved these changes
May 12, 2023
Naarcha-AWS
approved these changes
May 15, 2023
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
May 15, 2023
…3915) * Adds documentation on how to use the SNS to SQS fan out with Data Prepper. Signed-off-by: David Venable <[email protected]> * Grammar edits. * Update s3-logs.md * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit a9a47ed) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Naarcha-AWS
added a commit
that referenced
this pull request
May 15, 2023
…3915) (#4082) * Adds documentation on how to use the SNS to SQS fan out with Data Prepper. * Grammar edits. * Update s3-logs.md * Update _data-prepper/common-use-cases/s3-logs.md * Update _data-prepper/common-use-cases/s3-logs.md --------- (cherry picked from commit a9a47ed) Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
harshavamsi
pushed a commit
to harshavamsi/documentation-website
that referenced
this pull request
Oct 31, 2023
…pensearch-project#3915) * Adds documentation on how to use the SNS to SQS fan out with Data Prepper. Signed-off-by: David Venable <[email protected]> * Grammar edits. * Update s3-logs.md * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Some users have hit an issue using Data Prepper with an SNS/SQS fan out. This can be overcome and this documentation change conveys what a user needs to do for it.
Issues Resolved
N/A
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.