Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SNS messages in S3 SQS queues #2604

Closed
dlvenable opened this issue Apr 28, 2023 · 1 comment · Fixed by #2622
Closed

Support SNS messages in S3 SQS queues #2604

dlvenable opened this issue Apr 28, 2023 · 1 comment · Fixed by #2622
Assignees
Milestone

Comments

@dlvenable
Copy link
Member

Is your feature request related to a problem? Please describe.

Presently, the s3 source parses SQS messages for when S3 writes directly to the SQS queue. However, in some cases (e.g. fan-out pattern), an architecture has S3 events go to SNS, and then SNS to SQS. When this happens, the message is wrapped in another layer of formatting.

Presently, we get messages like the following:

ERROR org.opensearch.dataprepper.plugins.source.SqsWorker - SQS message with message ID:*** has invalid body which cannot be parsed into S3EventNotification. Unrecognized field "Type" (class org.opensearch.dataprepper.plugins.source.S3EventNotification), not marked as ignorable (one known property: "Records"])
 at [Source: (String)"{
  "Type" : "Notification",
  "MessageId" : "***",
  "TopicArn" : "arn:aws:sns:us-east-1:123456789012:my-topic",
  "Subject" : "Amazon S3 Notification",
  "Message" : "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2023-04-27T12:32:12.908Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"***\"[truncated 1060 chars]; line: 12, column: 2] (through reference chain: org.opensearch.dataprepper.plugins.source.S3EventNotification["Type"])

Describe the solution you'd like

Automatically detect when the input is an SNS message. Extract the Message from that. Then parse the body of Message just as we currently do.

Describe alternatives you've considered (Optional)

We could provide a user-configure here. But, it seems that there are only two common scenarios that need to be supported and they can be automatically detected.

Scenario 1: S3 -> SQS (already supported)
Scenario 2: S3 -> SNS -> SQS

@dlvenable
Copy link
Member Author

I created #2622 to add support to Data Prepper.

Users can still use the fan out pattern by enabling raw message delivery on their SNS to SQS subscription. I created a PR to document this for users: opensearch-project/documentation-website#3915

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant