Skip to content

A Downtime Notifier Stack

Jump to bottom
Dave Liggat edited this page Jul 9, 2016 · 4 revisions

downtime_notifier is a Lambda function that polls a list of websites, and attempts to connect. If it cannot reach a particular site, it will post a notification to an SNS topic. The function is built as part of a CloudFormation stack which also includes:

  • A AWS::Events::Rule to periodically trigger the function
  • A DynamoDB table to store results; the previous outcome is queried each run to detect state changes
  • A SNS topic for reporting failures in the event that a state change has occurred
  • An IAM role for the function to execute under; permits publishing to the topic, among other things
  • An optional KMS stack for encrypted secrets that can be decrypted at runtime by the Lambda function

Assuming fab_aws has been set up properly, this stack can be easily deployed:

1) Create the downtime_notifier stack (dn_stack)

This creates the elements listed above.

Some basic configuration:

# cloudformation_config/dn_stack.yaml
display_name: Downtime Notifier Bot  # Sender email address for the SNS topic
description: A function to monitor websites, and alert on unavailability
schedule_expression: rate(5 minutes)  # How often to run the Lambda function
memory_size: 256
timeout: 180

Optionally, supply a list of email addresses and/or SMS targets to be added to the SNS topic subscription list on creation. These will require confirmation of an opt-in email that will be sent.

# cloudformation_config/dn_stack.local.yaml
subscriptions:
  - endpoint: [email protected]
    protocol: email
  - endpoint: +177812345678
    protocol: sms

Create the actual stack:

fab render validate provision:template_name=dn_stack,stack_name=downtime-notifier-stack

2) Optional: Create a kms-stack

Supply the ARN of the Lambda execution IAM role (from the dn_stack outputs) in the kms_stack configuration so the KMS key-policy will permit a decryption by the Lambda function. We can use this to encrypt Lambda configuration 'secrets' such as the SNS topic to post to in the event of failures.

# cloudformation_config/kms_stack.local.yaml

parameters:
  PermitDecryptionByRoleArn0: arn:aws:iam::111111111111:role/downtime-notifier-stack-LambdaFunctionExecutionRol-INGNY1TAFVJB

Provision the actual KMS stack:

fab render validate provision:template_name=kms_stack,stack_name=config-key-stack

2a) Make a Key Alias

This will ease interaction with the key when we use it to encrypt values.

# Using the AWS CLI here - could be a `fab` task, but haven't gotten around to that yet.
aws kms create-alias --alias-name alias/config-key --target-key-id cd0823ad-fb8d-44ad-91fe-3e26e0cf5308

2b) Use the key to encrypt the SNS topic

# Using the AWS CLI here - could be a `fab` task, but haven't gotten around to that yet.
aws kms encrypt --key-id alias/config-key --plaintext "arn:aws:sns:us-west-2:111111111111:downtime-notifier-stack-NotificationTopic-10B8FJFDDASDF"

3) Update the Lambda configuration

The Lambda function needs to know what SNS topic to post to, and the name of the DynamoDB table. Place these values into into the Lambda config:

If encrypted:

# lambda/downtime_notifier/lambda_config/env.yaml
encrypted_dynamo_table: CAQAwgaMGCSqGSIb3DQEHATA.....PCmiliO
encrypted_topic_arn: CiDuilNJaTYlHPZi/1mDo4XjI0FC3....DAFSFQ=
subject_prefix: Site downtime detected!
sites:
  - url: https://www.google.ca
    name: Google

Else:

# lambda/downtime_notifier/lambda_config/env.yaml
dynamo_table: downtime-notifier-stack-ResultTable-KE5YN8THMI23
topic_arn: arn:aws:sns:us-west-2:111111111111:downtime-notifier-stack-NotificationTopic-110T6X2GKWVN0
subject_prefix: Site downtime detected!
sites:
  - url: https://www.google.ca
    name: Google

4) Override sites to be monitored in env.local.yaml:

# lambda/downtime_notifier/lambda_config/env.local.yaml
sites:
  - url: https://liggat.org
    name: Liggat.org
    expected_text: <h2>by Dave Liggat</h2>
  - url: http://liggat.org
    name: Liggat.org (http => https redirect)
    expected_code: 301

5) Build the Lambda package

This constructs a zip file containing everything the Lambda function needs to execute on AWS.

fab build:function_name=downtime_notifier

6) Deploy the lambda package to the ARN

This replaces the currently-deployed Lambda code with the latest build.

FUNCTION_NAME=downtime_notifier
ARN=arn=arn:aws:lambda:us-west-2:111111111111:function:downtime-notifier-stack-LambdaFunction-J3R
fab deploy:function_name=$FUNCTION_NAME,arn=$ARN

7) Done

With all of the above deployed, the Lambda function will spin up at the specified interval and check the list of sites. It will notify the SNS topic on any failure.

Clone this wiki locally