From 098e8854095b5176a8c891b5c9e35a0e71a176fc Mon Sep 17 00:00:00 2001 From: David Venable Date: Tue, 2 May 2023 09:32:21 -0500 Subject: [PATCH 1/5] Adds documentation on how to use the SNS to SQS fan out with Data Prepper. Signed-off-by: David Venable --- _data-prepper/common-use-cases/s3-logs.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/_data-prepper/common-use-cases/s3-logs.md b/_data-prepper/common-use-cases/s3-logs.md index fa3f74b029..97648f9154 100644 --- a/_data-prepper/common-use-cases/s3-logs.md +++ b/_data-prepper/common-use-cases/s3-logs.md @@ -171,6 +171,16 @@ We recommend that you have one SQS queue per Data Prepper pipeline. In addition, If you have multiple pipelines, you must create multiple SQS queues for each pipeline, even if both pipelines use the same S3 bucket. - - +## Amazon SNS fan out pattern + +Often a team needs to have multiple event handlers for an S3 bucket. +A standard solution here is to use an [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (SNS) to SQS [fan out pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). +In this setup, you will have S3 event notifications go directly to a single SNS topic. +Then you can subscribe multiple SQS queues to the SNS topic. + +In order for this to work with Data Prepper, you must configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. +This configuration allows Data Prepper to parse the incoming data correctly. +The raw message delivery option is on the connection between SNS and SQS. +Thus, you can have some SQS queues which do not enable raw message delivery. +Using Data Prepper will not interfere with those queues. From 3818cc2fe57d63cab709175102aaaaba3865fc70 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 5 May 2023 13:46:42 -0500 Subject: [PATCH 2/5] Grammar edits. --- _data-prepper/common-use-cases/s3-logs.md | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/_data-prepper/common-use-cases/s3-logs.md b/_data-prepper/common-use-cases/s3-logs.md index 97648f9154..487cf4d95a 100644 --- a/_data-prepper/common-use-cases/s3-logs.md +++ b/_data-prepper/common-use-cases/s3-logs.md @@ -171,16 +171,10 @@ We recommend that you have one SQS queue per Data Prepper pipeline. In addition, If you have multiple pipelines, you must create multiple SQS queues for each pipeline, even if both pipelines use the same S3 bucket. -## Amazon SNS fan out pattern - -Often a team needs to have multiple event handlers for an S3 bucket. -A standard solution here is to use an [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (SNS) to SQS [fan out pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). -In this setup, you will have S3 event notifications go directly to a single SNS topic. -Then you can subscribe multiple SQS queues to the SNS topic. - -In order for this to work with Data Prepper, you must configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. -This configuration allows Data Prepper to parse the incoming data correctly. -The raw message delivery option is on the connection between SNS and SQS. -Thus, you can have some SQS queues which do not enable raw message delivery. -Using Data Prepper will not interfere with those queues. +## Amazon SNS fanout pattern + +To meet the scale of logs produced in S3, some users require multiple SQS queues for their logs. You can use the [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications go directly to a single SNS topic, where you can subscribe to multiple SQS queues. + +To make sure that Data Prepper can parse the event from the SNS topic directly, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. If you want Data Prepper to ignore an SQS queue that sends notifications to your SNS topic, do not enable raw message delivery those queues. + From 30ce15116c31720df2a8314d852b4bd590398061 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 9 May 2023 16:03:59 -0500 Subject: [PATCH 3/5] Update s3-logs.md --- _data-prepper/common-use-cases/s3-logs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/common-use-cases/s3-logs.md b/_data-prepper/common-use-cases/s3-logs.md index 487cf4d95a..f45fc10ce3 100644 --- a/_data-prepper/common-use-cases/s3-logs.md +++ b/_data-prepper/common-use-cases/s3-logs.md @@ -175,6 +175,6 @@ If you have multiple pipelines, you must create multiple SQS queues for each pip To meet the scale of logs produced in S3, some users require multiple SQS queues for their logs. You can use the [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications go directly to a single SNS topic, where you can subscribe to multiple SQS queues. -To make sure that Data Prepper can parse the event from the SNS topic directly, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. If you want Data Prepper to ignore an SQS queue that sends notifications to your SNS topic, do not enable raw message delivery those queues. +To make sure that Data Prepper can parse the event from the SNS topic directly, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. Setting this option will not affect other SQS queues which are subscribed to that SNS topic. From 324b8d8260e1f851db162e897d90c8080f8b32d2 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 15 May 2023 12:55:17 -0500 Subject: [PATCH 4/5] Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _data-prepper/common-use-cases/s3-logs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/common-use-cases/s3-logs.md b/_data-prepper/common-use-cases/s3-logs.md index f45fc10ce3..785376f37b 100644 --- a/_data-prepper/common-use-cases/s3-logs.md +++ b/_data-prepper/common-use-cases/s3-logs.md @@ -175,6 +175,6 @@ If you have multiple pipelines, you must create multiple SQS queues for each pip To meet the scale of logs produced in S3, some users require multiple SQS queues for their logs. You can use the [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications go directly to a single SNS topic, where you can subscribe to multiple SQS queues. -To make sure that Data Prepper can parse the event from the SNS topic directly, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. Setting this option will not affect other SQS queues which are subscribed to that SNS topic. +To make sure that Data Prepper can directly parse the event from the SNS topic, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. Setting this option will not affect other SQS queues that are subscribed to that SNS topic. From 38b9253330339ae6681c9756f7efd845ad45ca08 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 15 May 2023 12:55:27 -0500 Subject: [PATCH 5/5] Update _data-prepper/common-use-cases/s3-logs.md Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _data-prepper/common-use-cases/s3-logs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/common-use-cases/s3-logs.md b/_data-prepper/common-use-cases/s3-logs.md index 785376f37b..2987c9a677 100644 --- a/_data-prepper/common-use-cases/s3-logs.md +++ b/_data-prepper/common-use-cases/s3-logs.md @@ -173,7 +173,7 @@ If you have multiple pipelines, you must create multiple SQS queues for each pip ## Amazon SNS fanout pattern -To meet the scale of logs produced in S3, some users require multiple SQS queues for their logs. You can use the [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications go directly to a single SNS topic, where you can subscribe to multiple SQS queues. +To meet the scale of logs produced by S3, some users require multiple SQS queues for their logs. You can use [Amazon Simple Notification Service](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) (Amazon SNS) to route event notifications from S3 to an SQS [fanout pattern](https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html). Using SNS, all S3 event notifications are sent directly to a single SNS topic, where you can subscribe to multiple SQS queues. To make sure that Data Prepper can directly parse the event from the SNS topic, configure [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS to SQS subscription. Setting this option will not affect other SQS queues that are subscribed to that SNS topic.