Skip to content

Latest commit

 

History

History
130 lines (89 loc) · 6.33 KB

metrics_to_cloudwatch_amp.md

File metadata and controls

130 lines (89 loc) · 6.33 KB

Offload EKS cluster (on EC2) metrics to Amazon CloudWatch and/or Amazon Managed Prometheus(AMP)

Install Chart

The helm chart provides flexibility to offload mertrics into target platform(CloudWatch, AMP) of your choice. Here are the possible deployment options -

  • Offload metrics to Amazon Managed Prometheus(AMP)
  • Offload metrics to Amazon CloudWatch
  • Offload metrics to both Amazon Managed Prometheus(AMP) and Amazon CloudWatch

The above-mentioned deployment modes are explained in-detail in the following sections.

Default Mode (Offload metrics to only Amazon Managed Prometheus)

By default, the helm chart is configured to offload metrics data to the Amazon Managed Prometheus(AMP) workspace only. You need update the values.yaml file to add the Remote Write Endpoint URL and the region for your AMP workspace. By default, the region is set to us-west-2. By default, only the prometheus receiver and prometheusremotewrite exporter is included to be able to send metrics to AMP.

helm install \
  [RELEASE_NAME] [REPO_NAME]/adot-exporter-for-eks-on-ec2 \
  --set clusterName=[CLUSTER_NAME] --set awsRegion=[AWS_REGION]

CLUSTER_NAME and AWS_REGION must be specified with your own EKS cluster and the region. You can find these values by executing following command.

$ kubectl config current-context

[IAM_User_Name]@[CLUSTER_NAME].[AWS_REGION].eksctl.io

To verify the installation is successful, you can execute the following command.

$ kubectl get pods --all-namespaces

NAMESPACE         NAME                              READY      STATUS         RESTARTS        AGE
amazon-metrics    adot-collector-daemonset-jlhvs    1/1        Running        0               5s
amazon-metrics    adot-collector-daemonset-ph7xs    1/1        Running        0               5s

If you see these two running pods, two for ADOT Collector as DaemonSets within the specified namespaces, they are successfully deployed.

Offload metrics to Amazon CloudWatch

To offload metrics to CloudWatch, you need to disable the AMP pipeline and enable the CloudWatch pipeline. You can either update the values.yaml to change the values of the following variable exporters and then run the helm install command as above. Only include the awscontainerinsightreceiver receiver and awsemf exporter to send metrics to CloudWatch Container Insights.

helm install \
  [RELEASE_NAME] [REPO_NAME]/adot-exporter-for-eks-on-ec2 \
  --set clusterName=[CLUSTER_NAME] --set awsRegion=[AWS_REGION] \
  --set adotCollector.daemonSet.service.metrics.receivers={awscontainerinsightreceiver}
  --set adotCollector.daemonSet.service.metrics.exporters={awsemf}

To verify the installation is successful, you can execute the following command.

$ kubectl get pods --all-namespaces

NAMESPACE           NAME                              READY      STATUS         RESTARTS        AGE
amazon-metrics      adot-collector-daemonset-jlhvs    1/1        Running        0               5s
amazon-metrics      adot-collector-daemonset-ph7xs    1/1        Running        0               5s
amazon-cloudwatch   fluent-bit-16rrn                  1/1        Running        0               5s
amazon-cloudwatch   fluent-bit-ksj6f                  1/1        Running        0               5s

If you see these four running pods, two for ADOT Collector and two for Fluent Bit as DaemonSets within the specified namespaces, they are successfully deployed.

Offload metrics to both CloudWatch and Amazon Managed Prometheus(AMP)

To send metrics to CloudWatch and AMP, you need to enable the CloudWatch pipeline. You can either update the values.yaml to change the values of the following variable exporters and then run the helm install command as above. Include both prometheus, awscontainerinsightreceiver as receivers, awsemf,prometheusremotewrite exporter to send metrics to both CloudWatch and AMP.

helm install \
  [RELEASE_NAME] [REPO_NAME]/adot-exporter-for-eks-on-ec2 \
  --set clusterName=[CLUSTER_NAME] --set awsRegion=[AWS_REGION] \
  --set adotCollector.daemonSet.service.metrics.receivers={prometheus,awscontainerinsightreceiver}
  --set adotCollector.daemonSet.service.metrics.exporters={prometheusremotewrite,awsemf}

To verify the installation is successful, you can execute the following command.

$ kubectl get pods --all-namespaces

NAMESPACE           NAME                              READY      STATUS         RESTARTS        AGE
amazon-metrics      adot-collector-daemonset-jlhvs    1/1        Running        0               5s
amazon-metrics      adot-collector-daemonset-ph7xs    1/1        Running        0               5s
amazon-cloudwatch   fluent-bit-16rrn                  1/1        Running        0               5s
amazon-cloudwatch   fluent-bit-ksj6f                  1/1        Running        0               5s

If you see these four running pods, two for ADOT Collector and two for Fluent Bit as DaemonSets within the specified namespaces, they are successfully deployed.

Verify the Helm chart works as expected

Run chart validation test and lint from MakeFile.

$ cd adot-exporter-for-eks-on-ec2
$ make install-tools # required initially
$ make all           # to run chart validation test and lint

Verify the metrics are sent to Amazon CloudWatch

  • Open Amazon CloudWatch console
  • Select "Logs → Log groups" on the left navigation bar.
  • Check if following four log groups exist (performance log group will take longer than others).
/aws/containerinsights/[CLUSTER_NAME]/application
/aws/containerinsights/[CLUSTER_NAME]/dataplane
/aws/containerinsights/[CLUSTER_NAME]/host
/aws/containerinsights/[CLUSTER_NAME]/performance

alt text

Verify the metrics are sent to Amazon Managed Service for Prometheus(AMP)

Connect to Amazon Managed Service for Grafana to visualize metrics stored in AMP. Please see the following document to setup the Grafana environment.

alt text