Adding Module and Example for ECS cluster monitoring with ecs_observer (

#211) * Adding Module and Example for ECS cluster monitoring with ecs_observer * Adding Module and Example for ECS cluster monitoring with ecs_observer * Incorporating PR comments * Restructuring Examples and modules folder for ECS, Added content in main Readme * Fixing path as per PR comments * Parameterzing the config files, incorporated PR review comments * Adding condition for AMP WS and fixing AMP endpoint * Adding Document for ECS Monitoring and parameterized some variables * Added sample dashboard * Adding Document for ECS Monitoring and parameterized some variables * Fixing failures detected by pre-commit * Fixing failures detected by pre-commit * Fixing failures detected by pre-commit * Pre-commit fixes * Fixing failures detected by pre-commit * Fixing failures detected by pre-commit * Pre-commit * Fixing HIGH security alerts detected by pre-commit * Fixing HIGH security alerts detected by pre-commit * Fixing HIGH security alerts detected by pre-commit, 31stOct * Add links after merge * 2ndNov - Added condiotnal creation for Grafana WS and module versions for AMG, AMP --------- Co-authored-by: Rodrigue Koffi <[email protected]>
aws-observability · Nov 2, 2023 · 70405b9 · 70405b9
1 parent 0b42935
commit 70405b9
Show file tree

Hide file tree

Showing 17 changed files with 863 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -127,6 +127,18 @@ module "eks_monitoring" {
 }
 ```
 
+#### Amazon ECS monitoring
+ECS cluster with VPC and EC2 can be created using the example [here](./examples/ecs_cluster_with_vpc)
+
+```hcl
+module "ecs_monitoring" {
+  source = "github.com/aws-observability/terraform-aws-observability-accelerator//modules/ecs-monitoring"
+
+  aws_ecs_cluster_name  = module.ecs_cluster.cluster_name
+  task_role_arn           = module.ecs_cluster.task_exec_iam_role_arn
+  execution_role_arn      = module.ecs_cluster.task_exec_iam_role_arn
+}
+```
 Grafana Dashboards
 
 <img width="2056" alt="image" src="https://user-images.githubusercontent.com/10175027/199110753-9bc7a9b7-1b45-4598-89d3-32980154080e.png">

diff --git a/docs/ecs/ecs-monitoring-on-ec2.md b/docs/ecs/ecs-monitoring-on-ec2.md
@@ -0,0 +1,62 @@
+# Amazon ECS on EC2 cluster monitoring
+
+This example demonstrates how to monitor your Amazon Elastic Container Service on EC2
+(Amazon ECS) cluster with the Observability Accelerator's ECS monitoring module
+
+The module collects Prometheus metrics from tasks running on ECS and sends it to Prometheus using AWS Distro for OpenTelemetry Collector (ADOT).
+You can either run the collector as a sidecar or deploy the collector as its own ECS service for entire cluster.
+ECS tasks with Prometheus endpoints are discovered using extension
+[ecsobserver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/observer/ecsobserver/README.md).
+(Unlike EKS, there is no builtin discovery for ECS inside prometheus)
+
+Additionally, you can optionally collect custom Prometheus metrics from your applications running
+on your ECS cluster.
+
+## Prerequisites
+
+!!! note
+    Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites) before proceeding.
+
+## Available Samples for various Worklods
+Make sure to update your exisitng Application Task Definitions based on the workload type :-
+
+#### 1. [Java/JMX workload for ECS Clusters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Sample-Workloads-ECS-javajmx.html)
+#### 2. [NGINX workload for Amazon ECS clusters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-nginx-ecs.html)
+#### 3. [App Mesh workload](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Sample-Workloads-ECS-appmesh.html)
+
+## Setup
+
+#### 1. Add the ECS Monitoring Module to your exisitng ECS CLuster
+
+```
+module "ecs_monitoring" {
+  source                = "../../modules/ecs-monitoring"
+  aws_ecs_cluster_name  = module.ecs_cluster.cluster_name
+  task_role_arn           = module.ecs_cluster.task_exec_iam_role_arn
+  execution_role_arn      = module.ecs_cluster.task_exec_iam_role_arn
+
+  depends_on = [
+    module.ecs_cluster
+  ]
+}
+```
+
+## Deploy
+
+Simply run this command to deploy the example
+
+```bash
+terraform apply
+```
+
+## Visualization
+![image](https://github.com/ruchimo/terraform-aws-observability-accelerator/assets/106240341/006c387e-92e8-45c8-ae2e-825900990741)
+
+
+## Cleanup
+
+To clean up your environment, destroy the Terraform example by running
+
+```sh
+terraform destroy
+```
diff --git a/docs/helpers/ecs-cluster-with-vpc.md b/docs/helpers/ecs-cluster-with-vpc.md
@@ -0,0 +1,45 @@
+# Example Amazon ECS Cluster with VPC
+This example deploys an AWS ECS Cluster with VPC and also add the ECS Monitoring module
+
+## Prerequisites
+
+!!! note
+    Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites) before proceeding.
+
+## Setup
+#### 1. Download sources and initialize Terraform¶
+
+```
+git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
+cd terraform-aws-observability-accelerator/examples/ecs-cluster-with-vpc
+terraform init
+```
+
+#### 2. AWS Region¶
+Specify the AWS Region where the resources will be deployed:
+
+```
+export TF_VAR_aws_region=xxx
+```
+
+#### 3. Terraform Plan to validate the changes/updates
+
+```
+terraform plan
+```
+
+## Deploy
+
+Simply run this command to deploy the example
+
+```bash
+terraform apply
+```
+
+## Cleanup
+
+To clean up your environment, destroy the Terraform example by running
+
+```sh
+terraform destroy
+```
diff --git a/examples/ecs-cluster-with-vpc/README.md b/examples/ecs-cluster-with-vpc/README.md
@@ -0,0 +1,67 @@
+# ECS Cluster w/ EC2 Autoscaling
+
+Configuration in this directory creates:
+
+- ECS cluster using EC2 autoscaling groups
+- Autoscaling groups with IAM instance profile to be used by ECS cluster
+- Example ECS service that utilizes
+  - Mounts a host volume into the container definition
+  - Load balancer target group attachment
+  - Security group for access to the example service
+
+## Usage
+
+To run this example you need to execute:
+
+```bash
+$ terraform init
+$ terraform plan
+$ terraform apply
+```
+
+Note that this example may create resources which will incur monetary charges on your AWS bill. Run `terraform destroy` when you no longer need these resources.
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.55 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.55 |
+
+## Modules
+
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_alb_sg"></a> [alb\_sg](#module\_alb\_sg) | terraform-aws-modules/security-group/aws | ~> 5.0 |
+| <a name="module_autoscaling"></a> [autoscaling](#module\_autoscaling) | terraform-aws-modules/autoscaling/aws | ~> 6.5 |
+| <a name="module_autoscaling_sg"></a> [autoscaling\_sg](#module\_autoscaling\_sg) | terraform-aws-modules/security-group/aws | ~> 5.0 |
+| <a name="module_ecs_cluster"></a> [ecs\_cluster](#module\_ecs\_cluster) | terraform-aws-modules/ecs/aws | 5.2.2 |
+| <a name="module_ecs_monitoring"></a> [ecs\_monitoring](#module\_ecs\_monitoring) | ../../modules/ecs-monitoring | n/a |
+| <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
+| [aws_ssm_parameter.ecs_optimized_ami](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ssm_parameter) | data source |
+
+## Inputs
+
+No inputs.
+
+## Outputs
+
+No outputs.
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+
+## License
+
+Apache-2.0 Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-aws-ecs/blob/master/LICENSE).