Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Amazon Security Lake data events in S3 source with SQS queue #2788

Closed
dlvenable opened this issue May 31, 2023 · 1 comment · Fixed by #2861
Closed

Support Amazon Security Lake data events in S3 source with SQS queue #2788

dlvenable opened this issue May 31, 2023 · 1 comment · Fixed by #2861
Labels
enhancement New feature or request plugin - source A plugin to receive data from a service or location.

Comments

@dlvenable
Copy link
Member

Is your feature request related to a problem? Please describe.

Amazon Security Lake provides events over Amazon EventBridge. The format is different from S3 directly to SQS or via SNS as shown in this sample configuration.

Describe the solution you'd like

Automatically detect events from Amazon Security Lake and read the S3 object accordingly.

Additional context

See #2604 for another feature that was able to parse SQS messages which came via SNS.

@dlvenable
Copy link
Member Author

It appears that the solution for this may also work for general Amazon EventBridge events. However, I see at least one clear difference. With Amazon Security Lake, the notifications are for create object only (which is what Data Prepper wants). But, with Event Bridge, there is a detail-type which indicates if it was created. See the Event Bridge sample structure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin - source A plugin to receive data from a service or location.
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants