You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Presently, the s3 source parses SQS messages for when S3 writes directly to the SQS queue. However, in some cases (e.g. fan-out pattern), an architecture has S3 events go to SNS, and then SNS to SQS. When this happens, the message is wrapped in another layer of formatting.
Presently, we get messages like the following:
ERROR org.opensearch.dataprepper.plugins.source.SqsWorker - SQS message with message ID:*** has invalid body which cannot be parsed into S3EventNotification. Unrecognized field "Type" (class org.opensearch.dataprepper.plugins.source.S3EventNotification), not marked as ignorable (one known property: "Records"])
at [Source: (String)"{
"Type" : "Notification",
"MessageId" : "***",
"TopicArn" : "arn:aws:sns:us-east-1:123456789012:my-topic",
"Subject" : "Amazon S3 Notification",
"Message" : "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2023-04-27T12:32:12.908Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"***\"[truncated 1060 chars]; line: 12, column: 2] (through reference chain: org.opensearch.dataprepper.plugins.source.S3EventNotification["Type"])
Describe the solution you'd like
Automatically detect when the input is an SNS message. Extract the Message from that. Then parse the body of Message just as we currently do.
Describe alternatives you've considered (Optional)
We could provide a user-configure here. But, it seems that there are only two common scenarios that need to be supported and they can be automatically detected.
Users can still use the fan out pattern by enabling raw message delivery on their SNS to SQS subscription. I created a PR to document this for users: opensearch-project/documentation-website#3915
Is your feature request related to a problem? Please describe.
Presently, the
s3
source parses SQS messages for when S3 writes directly to the SQS queue. However, in some cases (e.g. fan-out pattern), an architecture has S3 events go to SNS, and then SNS to SQS. When this happens, the message is wrapped in another layer of formatting.Presently, we get messages like the following:
Describe the solution you'd like
Automatically detect when the input is an SNS message. Extract the
Message
from that. Then parse the body ofMessage
just as we currently do.Describe alternatives you've considered (Optional)
We could provide a user-configure here. But, it seems that there are only two common scenarios that need to be supported and they can be automatically detected.
Scenario 1: S3 -> SQS (already supported)
Scenario 2: S3 -> SNS -> SQS
The text was updated successfully, but these errors were encountered: