From a9a47ed945b7355ec93f9f48aaee83aa6d281cab Mon Sep 17 00:00:00 2001 From: David Venable Date: Mon, 15 May 2023 12:55:45 -0500 Subject: [PATCH] Data Prepper documentation for the SNS to SQS fan out pattern in S3 (#3915) * Adds documentation on how to use the SNS to SQS fan out with Data Prepper. Signed-off-by: David Venable * Grammar edits. * Update s3-logs.md * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: David Venable Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower --- _data-prepper/common-use-cases/s3-logs.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_data-prepper/common-use-cases/s3-logs.md b/_data-prepper/common-use-cases/s3-logs.md index fa3f74b029..2987c9a677 100644 --- a/_data-prepper/common-use-cases/s3-logs.md +++ b/_data-prepper/common-use-cases/s3-logs.md @@ -171,6 +171,10 @@ We recommend that you have one SQS queue per Data Prepper pipeline. In addition, If you have multiple pipelines, you must create multiple SQS queues for each pipeline, even if both pipelines use the same S3 bucket. +## Amazon SNS fanout pattern +To meet the scale of logs produced by S3, some users require multiple SQS queues for their logs. You can use [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (Amazon SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications are sent directly to a single SNS topic, where you can subscribe to multiple SQS queues. + +To make sure that Data Prepper can directly parse the event from the SNS topic, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. Setting this option will not affect other SQS queues that are subscribed to that SNS topic.