Skip to content

Commit

Permalink
Data Prepper documentation for the SNS to SQS fan out pattern in S3 (#…
Browse files Browse the repository at this point in the history
…3915)

* Adds documentation on how to use the SNS to SQS fan out with Data Prepper.

Signed-off-by: David Venable <[email protected]>

* Grammar edits.

* Update s3-logs.md

* Update _data-prepper/common-use-cases/s3-logs.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/common-use-cases/s3-logs.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
  • Loading branch information
3 people authored May 15, 2023
1 parent ba8039c commit a9a47ed
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions _data-prepper/common-use-cases/s3-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,10 @@ We recommend that you have one SQS queue per Data Prepper pipeline. In addition,

If you have multiple pipelines, you must create multiple SQS queues for each pipeline, even if both pipelines use the same S3 bucket.

## Amazon SNS fanout pattern

To meet the scale of logs produced by S3, some users require multiple SQS queues for their logs. You can use [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (Amazon SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications are sent directly to a single SNS topic, where you can subscribe to multiple SQS queues.

To make sure that Data Prepper can directly parse the event from the SNS topic, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. Setting this option will not affect other SQS queues that are subscribed to that SNS topic.


0 comments on commit a9a47ed

Please sign in to comment.