Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a different queue for retroactive analysis #134

Open
austinbyers opened this issue Sep 5, 2018 · 1 comment
Open

Create a different queue for retroactive analysis #134

austinbyers opened this issue Sep 5, 2018 · 1 comment

Comments

@austinbyers
Copy link
Collaborator

Background

Related to #102 , it's always felt awkward to have the same SQS queue for both regular operation and for retroactive analysis:

  • New binaries added to the BinaryAlert bucket might not be scanned for hours if a large retro scan is underway
  • A live_test will fail for the same reason if a retro scan is running
  • A purge_queue to stop a runaway retro scan will also drop new object events from the queue
  • The number of concurrent analyzers needed for a retro scan vs. normal operation could be vastly different

These problems could all be fixed if there were just a different SQS queue for retroactive analysis!

Desired Change

Create a new SQS queue specifically for retro scans. It's not clear whether Lambda will allow 2 different SQS event source mappings, but it's worth a shot.

@jdheyburn
Copy link

On top of this, might it be helpful to publish retrospective no_yara_matches to a different SNS topic?

Background behind that is, users might have a flow whereby they subscribe to no_yara_match for continued file processing. If retrospective scanning were to publish messages on that topic once again, they could be double-processed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants