[Feature Request] Dynamic Routing of Alerts to Output

[jack1902]

Background

I would like to configure one output on a rule but have it send to different destinations based on something inside the alert. (Eg, account "a" slack channel a, account "b" slack channel "b")
NOTE: Before filing this issue, please consider the following:

Have you tried pinging us on Slack?
https://streamalert.herokuapp.com/

Are you on the latest version of StreamAlert?

Description

I would like to write one rule, and have that route the alert to an output based on information within the alert.
The current way i am thinking of implementing this is to use the context field and have a placeholder output, such as "slack:placeholder".

1. The rule will use a lookup table (new feature in release 3-0-0) to use the account_id on the record and find out which team owns that account (i currently work with alot of AWS Accounts).
2. Use this team name as the descriptor (i plan to add each team as an output, to store the secrets relevant for that output in the s3 bucket) and carry on the output.

This would be amazing, as using matchers is relevant most of the time but not in a case when you want a rule to trigger regardless of the account, but only notify the team that owns it. (I don't like the idea of notifying people who don't need to be notified)

[Ryxias]

This is a great feature idea. We also have the (upcoming) requirement of dynamic routing based upon AWS Accounts.

What do you think of the following 2 solutions:

Solution 1, dynamic_output=

When writing a rule, you can now specify a new dynamic_outputs= option:

@rule(
  logs=['blah'],
  outputs=['default-output'].
  dynamic_outputs=[dynamic_output_function],
  # ..
)
def blahblahblah(record):
  # ... zzzz

def dynamic_output_function(record, context):
  account_id = record.get('recipientaccountid')
 
  return slack_output = LookupTables.get(
    'my_lookup_table',
    'aws-account-owner-slack:{}'.format(account_id), 
    None
  )

You can pass dynamic_outputs an array of functions. Each function accepts a record & context. It returns a string corresponding to an output, or None.

Then in the rules_engine.py code we run this function whenever there is a rule match to generate the output. The implementation sounds pretty easy—we'd just need to change how RulesEngine._configure_outputs() is implemented.

Solution2, overloaded outputs=

Same as solution 1, except instead of a separate option, we overload the existing outputs= option to respect both strings and functions. Tradeoff here is it may be a bit more confusing to read?

@rule(
  logs=['blah'],
  outputs=['slack:default', dynamic_output_function].
  # ..
)
def blahblahblah(record):
  # ... zzzz

def dynamic_output_function(record):
  account_id = record.get('recipientaccountid')

  return slack_output = LookupTables.get(
    'my_lookup_table',
    'aws-account-owner-slack:{}'.format(account_id), 
    None
  )

Thoughts?

[jack1902]

so i like the new dynamic_outputs (solution 1) a lot more. Its less confusing and the intention is obvious from the get go. The example you gave is very similar to what we have to do with alerts.

I think the main thing to consider is the ordering of what is called first, personally i think this order is best:

1. Rule is evaluated
2. dynamic_output is configured only if the rule becomes an alert

Also, if we add a new dynamic_outputs, should we then consider the ability to ask the user for additional outputs when configuring a slack channel? Ie, i can just put the descriptor and url multiple times from one call to the ./manage.py output slack (this is more of the detail, but the lookup tables are the major feature that allows for this type of stuff)

Edit:

Also i like the separation because i know i always want to send alerts to one specific place (central security team), but dynamically send them to the owners of the account

[0xdabbad00]

When I had a need for something similar to this previously (#416), we had a requirement that under some circumstances the alert went to multiple locations. Specifically, all alerts would go to the security team, but if the alert was for a certain production AWS account, it would additionally go to the production team (so we didn't have the security team acting as a middle man for this important account). I think it would be useful to consider that use case for this.

[Ryxias]

@0xdabbad00 maybe for the dynamic_outputs, each function can return:

• None — No outputs. Nothing is added to the "current" list of outputs.
• str — One output. The single string is added to the "current" list of outputs, if it is not already there.
• list[str] — Multiple outputs. Every element of the list is merged in with the current list of outputs, with deduplication.

@jack1902 I agree with the ordering. In my suggested implementation, we'd be passing in a function reference, which only gets executed when RulesEngine._configure_outputs() is called. This is only called after a rule matches.

[jack1902]

@Ryxias Noted, i see where it would be called now :D.

Also i like the idea of None, str and list[str]. That works nicely, although would the dynamic outputs be checked for their existence before being merged to throw a different error message to notify the user their dynamic function is returning an unknown output?

[jack1902]

Closing as work is now merged