Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Module and Example for ECS cluster monitoring with ecs_observer #211

Merged
merged 38 commits into from
Nov 2, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
727674c
Adding Module and Example for ECS cluster monitoring with ecs_observer
ruchimo Aug 9, 2023
1900fc8
Adding Module and Example for ECS cluster monitoring with ecs_observer
ruchimo Aug 9, 2023
930721a
Merge branch 'main' into main
bonclay7 Aug 9, 2023
c1ac8b8
Incorporating PR comments
ruchimo Aug 10, 2023
52c14f4
Merge branch 'main' of github.com:ruchimo/terraform-aws-observability…
ruchimo Aug 10, 2023
5968b45
Merge branch 'main' into main
ruchimo Aug 10, 2023
e2e7af7
Restructuring Examples and modules folder for ECS, Added content in m…
ruchimo Aug 10, 2023
3be9980
Merge branch 'main' of github.com:ruchimo/terraform-aws-observability…
ruchimo Aug 10, 2023
0ef4294
Fixing path as per PR comments
ruchimo Aug 14, 2023
6071dbd
Parameterzing the config files, incorporated PR review comments
ruchimo Aug 14, 2023
63b1db9
Merge branch 'main' into main
bonclay7 Aug 17, 2023
3ca49bd
Adding condition for AMP WS and fixing AMP endpoint
ruchimo Aug 24, 2023
54a2a28
Merge branch 'main' into main
bonclay7 Aug 25, 2023
ac1058d
Adding Document for ECS Monitoring and parameterized some variables
ruchimo Sep 1, 2023
23c073a
Added sample dashboard
ruchimo Sep 1, 2023
c1d8303
Adding Document for ECS Monitoring and parameterized some variables
ruchimo Sep 1, 2023
47be776
Merge branch 'main' of github.com:ruchimo/terraform-aws-observability…
ruchimo Sep 1, 2023
c272480
Merge branch 'main' into main
ruchimo Sep 1, 2023
b2ae876
Fixing failures detected by pre-commit
ruchimo Sep 4, 2023
298c7d5
Merge branch 'main' of github.com:ruchimo/terraform-aws-observability…
ruchimo Sep 4, 2023
a94d212
Merge branch 'main' into main
ruchimo Sep 4, 2023
6170820
Fixing failures detected by pre-commit
ruchimo Sep 4, 2023
f591aff
Merge branch 'main' of github.com:ruchimo/terraform-aws-observability…
ruchimo Sep 4, 2023
2343c10
Merge branch 'main' into main
ruchimo Sep 5, 2023
47364af
Merge branch 'main' into main
bonclay7 Sep 21, 2023
bf318db
Merge branch 'main' into main
bonclay7 Oct 25, 2023
86ebf4e
Fixing failures detected by pre-commit
ruchimo Oct 27, 2023
70f7e44
Pre-commit fixes
bonclay7 Oct 29, 2023
bdcdc0d
Fixing failures detected by pre-commit
ruchimo Oct 30, 2023
289a01d
Merge branch 'main' of github.com:ruchimo/terraform-aws-observability…
ruchimo Oct 30, 2023
90ffa83
Fixing failures detected by pre-commit
ruchimo Oct 30, 2023
4440d0c
Pre-commit
bonclay7 Oct 30, 2023
0f126e7
Fixing HIGH security alerts detected by pre-commit
ruchimo Oct 30, 2023
d4d7c8e
Merge branch 'main' of github.com:ruchimo/terraform-aws-observability…
ruchimo Oct 30, 2023
a1c1821
Fixing HIGH security alerts detected by pre-commit
ruchimo Oct 31, 2023
59355d6
Fixing HIGH security alerts detected by pre-commit, 31stOct
ruchimo Oct 31, 2023
8919e6c
Add links after merge
bonclay7 Nov 2, 2023
382987b
2ndNov - Added condiotnal creation for Grafana WS and module versions…
ruchimo Nov 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,18 @@ module "eks_monitoring" {
}
```

#### Amazon ECS monitoring
ECS cluster with VPC and EC2 can be created using the example [here](./examples/ecs_cluster_with_vpc)

```hcl
module "ecs_monitoring" {
source = "github.com/aws-observability/terraform-aws-observability-accelerator//modules/ecs-monitoring"

aws_ecs_cluster_name = module.ecs_cluster.cluster_name
task_role_arn = module.ecs_cluster.task_exec_iam_role_arn
execution_role_arn = module.ecs_cluster.task_exec_iam_role_arn
}
```
Grafana Dashboards

<img width="2056" alt="image" src="https://user-images.githubusercontent.com/10175027/199110753-9bc7a9b7-1b45-4598-89d3-32980154080e.png">
Expand Down
62 changes: 62 additions & 0 deletions docs/ecs/ecs-monitoring-on-ec2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Amazon ECS on EC2 cluster monitoring

This example demonstrates how to monitor your Amazon Elastic Container Service on EC2
(Amazon ECS) cluster with the Observability Accelerator's ECS monitoring module

The module collects Prometheus metrics from tasks running on ECS and sends it to Prometheus using AWS Distro for OpenTelemetry Collector (ADOT).
You can either run the collector as a sidecar or deploy the collector as its own ECS service for entire cluster.
ECS tasks with Prometheus endpoints are discovered using extension
[ecsobserver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/observer/ecsobserver/README.md).
(Unlike EKS, there is no builtin discovery for ECS inside prometheus)

Additionally, you can optionally collect custom Prometheus metrics from your applications running
on your ECS cluster.

## Prerequisites

!!! note
Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites) before proceeding.

## Available Samples for various Worklods
Make sure to update your exisitng Application Task Definitions based on the workload type :-

#### 1. [Java/JMX workload for ECS Clusters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Sample-Workloads-ECS-javajmx.html)
#### 2. [NGINX workload for Amazon ECS clusters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-nginx-ecs.html)
#### 3. [App Mesh workload](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Sample-Workloads-ECS-appmesh.html)

## Setup

#### 1. Add the ECS Monitoring Module to your exisitng ECS CLuster

```
module "ecs_monitoring" {
source = "../../modules/ecs-monitoring"
aws_ecs_cluster_name = module.ecs_cluster.cluster_name
task_role_arn = module.ecs_cluster.task_exec_iam_role_arn
execution_role_arn = module.ecs_cluster.task_exec_iam_role_arn

depends_on = [
module.ecs_cluster
]
}
```

## Deploy

Simply run this command to deploy the example

```bash
terraform apply
```

## Visualization
![image](https://github.com/ruchimo/terraform-aws-observability-accelerator/assets/106240341/006c387e-92e8-45c8-ae2e-825900990741)


## Cleanup

To clean up your environment, destroy the Terraform example by running

```sh
terraform destroy
```
45 changes: 45 additions & 0 deletions docs/helpers/ecs-cluster-with-vpc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Example Amazon ECS Cluster with VPC
This example deploys an AWS ECS Cluster with VPC and also add the ECS Monitoring module

## Prerequisites

!!! note
Make sure to complete the [prerequisites section](https://aws-observability.github.io/terraform-aws-observability-accelerator/concepts/#prerequisites) before proceeding.

## Setup
#### 1. Download sources and initialize Terraform¶

```
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
cd terraform-aws-observability-accelerator/examples/ecs-cluster-with-vpc
terraform init
```

#### 2. AWS Region¶
Specify the AWS Region where the resources will be deployed:

```
export TF_VAR_aws_region=xxx
```

#### 3. Terraform Plan to validate the changes/updates

```
terraform plan
```

## Deploy

Simply run this command to deploy the example

```bash
terraform apply
```

## Cleanup

To clean up your environment, destroy the Terraform example by running

```sh
terraform destroy
```
67 changes: 67 additions & 0 deletions examples/ecs-cluster-with-vpc/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# ECS Cluster w/ EC2 Autoscaling

Configuration in this directory creates:

- ECS cluster using EC2 autoscaling groups
- Autoscaling groups with IAM instance profile to be used by ECS cluster
- Example ECS service that utilizes
- Mounts a host volume into the container definition
- Load balancer target group attachment
- Security group for access to the example service

## Usage

To run this example you need to execute:

```bash
$ terraform init
$ terraform plan
$ terraform apply
```

Note that this example may create resources which will incur monetary charges on your AWS bill. Run `terraform destroy` when you no longer need these resources.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.55 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.55 |

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_alb_sg"></a> [alb\_sg](#module\_alb\_sg) | terraform-aws-modules/security-group/aws | ~> 5.0 |
| <a name="module_autoscaling"></a> [autoscaling](#module\_autoscaling) | terraform-aws-modules/autoscaling/aws | ~> 6.5 |
| <a name="module_autoscaling_sg"></a> [autoscaling\_sg](#module\_autoscaling\_sg) | terraform-aws-modules/security-group/aws | ~> 5.0 |
| <a name="module_ecs_cluster"></a> [ecs\_cluster](#module\_ecs\_cluster) | terraform-aws-modules/ecs/aws | 5.2.2 |
| <a name="module_ecs_monitoring"></a> [ecs\_monitoring](#module\_ecs\_monitoring) | ../../modules/ecs-monitoring | n/a |
| <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |

## Resources

| Name | Type |
|------|------|
| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
| [aws_ssm_parameter.ecs_optimized_ami](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ssm_parameter) | data source |

## Inputs

No inputs.

## Outputs

No outputs.
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

## License

Apache-2.0 Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-aws-ecs/blob/master/LICENSE).
Loading
Loading