Backend NPE2 Feature (#520)

* Add initial lambda for retrieving manifest * Add npe2 to requirements * Add comment about requirements * Remove extension copying to see if function otherwise runs * Rename folder to plugins * Add back extension * Move out of backend * Add dockerfile and requirements * Update path to command * Docker Edit * Rename npe2 file * Docker Edit * Rename file * get_plugin_manifest * testing * update pip install location and add boto3 dependency * testing * testing * testing * testing * add sys path for installation * adding failure handler * adding failure handler * adding details to failureHandler * adding details to failureHandler * testing deletion of s3 files * testing deletion of s3 files * testing deletion of s3 files * testing recreation of s3 files * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * testing manifest attributes * updated code for noe2 * add npe2 attributes to plugin and index * add npe2 attributes to plugin and index * add npe2 attributes to plugin and index * add npe2 attributes to plugin and index * fixing writing to manifest * fixing writing to manifest * fixing writing to manifest * fixing writing to manifest * fixing writing to manifest * fixing writing to manifest * testing npe1 contributions * testing npe1 contributions * testing npe1 contributions * updating preview stuff * updating preview stuff * add docstrings to added methods * Update preview page discovery logic (#519) * Try to rely on already installed plugin * Clean up * testing * Resolved Github PR comments * testing current branch * testing current branch * testing current branch * completing feature for branch * Revert "testing current branch" This reverts commit 2eca36d. * Got rid of npe2 depedency in backend folder * fixing errors * fixing errors * fixing errors * fixing errors * fixing errors * adding npe2 * fixing internal error * fixing internal error * fixing imports * adding debug message * adding debug message * Move back to parsing manifest in the backend * Change parse manifest to use dict attributes * update preview page to use parse manifest with manifest dictionary * Fix key access for manifest dict * Add newline for manifest writing * Add 2 lambda function in terraform modules to process manifest separately from lambda processing backend requests * Add 2 lambda function in terraform modules to process manifest separately from lambda processing backend requests * Add 2 lambda function in terraform modules to process manifest separately from lambda processing backend requests * Add 2 lambda function in terraform modules to process manifest separately from lambda processing backend requests * Add 2 lambda function in terraform modules to process manifest separately from lambda processing backend requests * Add 2 lambda function in terraform modules to process manifest separately from lambda processing backend requests * Add 2 lambda function in terraform modules to process manifest separately from lambda processing backend requests * testing terraform * testing terraform * adding policy for lambdas * adding terraform configs * adding terraform configs * adding terraform configs * adding terraform configs * adding terraform configs * Merge plugin discovery changes into original npe2 work (#551) * testing npe2 discovery * adding pyqt5 in req * import statement * import statement * import statement * import statement * import statement * testing * testing * test * styling changes * styling changes * styling changes * styling changes * moved file * adding error message to index.json and plugin.yaml * fixing styling * changing error string * changing error string * changing error string * changing error string * changing error string * changing error string * changing error string * changing error string * changing error string Co-authored-by: klai95 <[email protected]> * adding terraform configs * adding terraform configs * add async permission * add async permission * add async permission * add async permission * adding process_count and error_message in exception clause * adding process_count and error_message in exception clause * merge with main * add platform for plugins image * stlying * fixing errors and pr comments * fixing pr comments * fixing index.json error * fixing index.json error * delete test manifest * change yaml content type * always update lambda because of image update * always update lambda because of image update Co-authored-by: Draga Doncila Pop <[email protected]> Co-authored-by: Draga Doncila Pop <[email protected]> Co-authored-by: Ziyang Liu <[email protected]>
chanzuckerberg · Jun 15, 2022 · b9203f1 · b9203f1
1 parent e5776f7
commit b9203f1
Show file tree

Hide file tree

Showing 19 changed files with 442 additions and 40 deletions.
diff --git a/.happy/terraform/envs/dev/providers.tf b/.happy/terraform/envs/dev/providers.tf
@@ -1,5 +1,5 @@
 provider aws {
-  version = "~> 3.39.0"
+  version = "~> 4.18.0"
   region  = "us-west-2"
   assume_role {
     role_arn = "arn:aws:iam::${var.aws_account_id}:role/${var.aws_role}"

diff --git a/.happy/terraform/envs/prod/providers.tf b/.happy/terraform/envs/prod/providers.tf
@@ -1,5 +1,5 @@
 provider aws {
-  version = "~> 3.39.0"
+  version = "~> 4.18.0"
   region  = "us-west-2"
   assume_role {
     role_arn = "arn:aws:iam::${var.aws_account_id}:role/${var.aws_role}"

diff --git a/.happy/terraform/envs/staging/providers.tf b/.happy/terraform/envs/staging/providers.tf
@@ -1,5 +1,5 @@
 provider aws {
-  version = "~> 3.39.0"
+  version = "~> 4.18.0"
   region  = "us-west-2"
   assume_role {
     role_arn = "arn:aws:iam::${var.aws_account_id}:role/${var.aws_role}"

diff --git a/.happy/terraform/modules/ecs-stack/main.tf b/.happy/terraform/modules/ecs-stack/main.tf
@@ -16,12 +16,15 @@ locals {
 
   frontend_cmd = []
   backend_cmd  = []
+  plugins_cmd  = []
+  failure_cmd  = ["get_plugin_manifest.failure_handler"]
 
   security_groups     = local.secret["security_groups"]
   zone                = local.secret["zone_id"]
   cluster             = local.secret["cluster_arn"]
   frontend_image_repo = local.secret["ecrs"]["frontend"]["url"]
   backend_image_repo  = local.secret["ecrs"]["backend"]["url"]
+  plugins_image_repo  = local.secret["ecrs"]["plugins"]["url"]
   external_dns        = local.secret["external_zone_name"]
   internal_dns        = local.secret["internal_zone_name"]
   rest_api_id         = local.secret["api_gateway"]["rest_api_id"]
@@ -48,6 +51,8 @@ locals {
 
   frontend_url = var.frontend_url != "" ? var.frontend_url: try(join("", ["https://", module.frontend_dns.dns_prefix, ".", local.external_dns]), var.frontend_url)
   backend_function_name = "${local.custom_stack_name}-backend"
+  plugins_function_name = "${local.custom_stack_name}-plugins"
+  failure_function_name = "${local.custom_stack_name}-failure"
 }
 
 module frontend_dns {
@@ -118,6 +123,54 @@ module backend_lambda {
   timeout               = 300
 }
 
+module plugins_lambda {
+  source             = "../lambda-container"
+  function_name      = local.plugins_function_name
+  image              = "${local.plugins_image_repo}:${local.image_tag}"
+  cmd                = local.plugins_cmd
+  tags               = var.tags
+
+  vpc_config = {
+    subnet_ids         = local.cloud_env.private_subnets
+    security_group_ids = local.security_groups
+  }
+
+  environment = {
+    "BUCKET" = local.data_bucket_name
+    "BUCKET_PATH" = var.env == "dev" ? local.custom_stack_name : ""
+  }
+
+  log_retention_in_days = 14
+  timeout               = 900
+  memory_size           = 10240
+  ephemeral_storage_size = 10240
+  maximum_retry_attempts = 0
+  create_async_event_config = true
+  destination_on_failure = module.failure_lambda.function_arn
+}
+
+module failure_lambda {
+  source             = "../lambda-container"
+  function_name      = local.failure_function_name
+  image              = "${local.plugins_image_repo}:${local.image_tag}"
+  cmd                = local.failure_cmd
+  tags               = var.tags
+
+  vpc_config = {
+    subnet_ids         = local.cloud_env.private_subnets
+    security_group_ids = local.security_groups
+  }
+
+  environment = {
+    "BUCKET" = local.data_bucket_name
+    "BUCKET_PATH" = var.env == "dev" ? local.custom_stack_name : ""
+  }
+
+  log_retention_in_days = 14
+  timeout               = 900
+  maximum_retry_attempts = 0
+}
+
 module api_gateway_proxy_stage {
   source               = "../api-gateway-proxy-stage"
   lambda_function_name = local.backend_function_name
@@ -137,6 +190,27 @@ resource "aws_cloudwatch_event_rule" "update_rule" {
   tags                = var.tags
 }
 
+resource "aws_lambda_permission" "allow_bucket" {
+  statement_id  = "AllowExecutionFromS3Bucket"
+  action        = "lambda:InvokeFunction"
+  function_name = module.plugins_lambda.function_arn
+  principal     = "s3.amazonaws.com"
+  source_arn    = local.data_bucket_arn
+}
+
+resource "aws_s3_bucket_notification" "plugins_notification" {
+  bucket = local.data_bucket_name
+
+  lambda_function {
+    lambda_function_arn = module.plugins_lambda.function_arn
+    events              = ["s3:ObjectCreated:*"]
+    filter_prefix       = var.env == "dev" ? local.custom_stack_name : ""
+    filter_suffix       = ".yaml"
+  }
+
+  depends_on = [aws_lambda_permission.allow_bucket]
+}
+
 resource "aws_cloudwatch_event_target" "update_target" {
     rule = aws_cloudwatch_event_rule.update_rule.name
     arn = module.backend_lambda.function_arn
@@ -184,12 +258,57 @@ data aws_iam_policy_document backend_policy {
   }
 }
 
+data aws_iam_policy_document plugins_policy {
+  statement {
+    actions = [
+      "s3:PutObject",
+      "s3:GetObject",
+      "s3:DeleteObject",
+    ]
+
+    resources = ["${local.data_bucket_arn}/*"]
+  }
+
+  statement {
+    actions = [
+      "lambda:InvokeFunction",
+      "lambda:InvokeAsync",
+    ]
+
+    resources = [module.failure_lambda.function_arn]
+  }
+}
+
+data aws_iam_policy_document failure_policy {
+  statement {
+    actions = [
+      "s3:PutObject",
+      "s3:GetObject",
+      "s3:DeleteObject",
+    ]
+
+    resources = ["${local.data_bucket_arn}/*"]
+  }
+}
+
 resource aws_iam_role_policy policy {
   name     = "${local.custom_stack_name}-${var.env}-policy"
   role     = module.backend_lambda.role_name
   policy   = data.aws_iam_policy_document.backend_policy.json
 }
 
+resource aws_iam_role_policy plugins_lambda_policy {
+  name     = "${local.custom_stack_name}-${var.env}-plugins-lambda-policy"
+  role     = module.plugins_lambda.role_name
+  policy   = data.aws_iam_policy_document.plugins_policy.json
+}
+
+resource aws_iam_role_policy failure_lambda_policy {
+  name     = "${local.custom_stack_name}-${var.env}-failure-lambda-policy"
+  role     = module.failure_lambda.role_name
+  policy   = data.aws_iam_policy_document.failure_policy.json
+}
+
 resource aws_acm_certificate cert {
   domain_name               = "${module.frontend_dns.dns_prefix}.${local.external_dns}"
   validation_method         = "DNS"

diff --git a/.happy/terraform/modules/lambda-container/main.tf b/.happy/terraform/modules/lambda-container/main.tf
@@ -1,6 +1,6 @@
 module lambda {
   source  = "terraform-aws-modules/lambda/aws"
-  version = "2.0.0"
+  version = "3.2.1"
   publish = var.provisioned_lambda == -1 ? false : true
 
   function_name          = var.function_name
@@ -16,6 +16,8 @@ module lambda {
   vpc_security_group_ids = var.vpc_config == null ? null : var.vpc_config.security_group_ids
 
   memory_size                       = var.memory_size
+  ephemeral_storage_size            = var.ephemeral_storage_size
+  maximum_retry_attempts            = var.maximum_retry_attempts
   kms_key_arn                       = var.kms_key_arn
   role_name                         = var.function_name
   role_path                         = var.lambda_role_path
@@ -24,6 +26,10 @@ module lambda {
   attach_network_policy             = true
   reserved_concurrent_executions    = var.reserved_concurrent_executions
   allowed_triggers                  = var.allowed_triggers
+  destination_on_failure            = var.destination_on_failure
+  create_async_event_config         = var.create_async_event_config
+
+  hash_extra                        = random_string.random_md5.result
 }
 
 resource "aws_lambda_provisioned_concurrency_config" "provisioned" {
@@ -35,4 +41,10 @@ resource "aws_lambda_provisioned_concurrency_config" "provisioned" {
   lifecycle {
     create_before_destroy = true
   }
+}
+
+# create random md5 so that the lambda would be refreshed
+resource "random_string" "random_md5" {
+  length           = 64
+  special          = false
 }
diff --git a/.happy/terraform/modules/lambda-container/variables.tf b/.happy/terraform/modules/lambda-container/variables.tf
@@ -26,6 +26,11 @@ variable log_retention_in_days {
   default = null
 }
 
+variable maximum_retry_attempts {
+  type    = number
+  default = 2
+}
+
 variable function_description {
   type        = string
   description = "Description for lambda function."
@@ -72,6 +77,12 @@ variable memory_size {
   default     = 128
 }
 
+variable ephemeral_storage_size {
+  type        = number
+  description = "Amount of ephemeral storage (/tmp) in MB your Lambda Function can use at runtime."
+  default     = 512
+}
+
 variable allowed_triggers {
   description = "Map of allowed triggers to create Lambda permissions"
   type        = map(any)
@@ -94,4 +105,16 @@ variable cmd {
 variable description {
   type    = string
   default = ""
+}
+
+variable create_async_event_config {
+  type        = bool
+  description = "Controls whether async event configuration for Lambda Function/Alias should be created"
+  default     = false
+}
+
+variable destination_on_failure {
+  type        = string
+  description = "Amazon Resource Name (ARN) of the destination resource for failed asynchronous invocations"
+  default     = null
 }
diff --git a/backend/api/app.py b/backend/api/app.py
@@ -1,4 +1,5 @@
 import os
+
 from apig_wsgi import make_lambda_handler
 from werkzeug.middleware.dispatcher import DispatcherMiddleware
 from flask import Flask, Response, jsonify, render_template