CloudWatch Agent with Prometheus in K8s takes 7-8 minutes to start

[annikulin]

I've been following the AWS Guide to setup CloudWatch agent with Prometheus support to collects metrics in Kubernetes cluster. It works as expected, but initial time for the Pod to start takes 7-8 minutes. Based on the logs there is a huge delay during init phase:

2021/08/09 12:32:57 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2021/08/09 12:32:57 Reading json config file path: /etc/cwagentconfig/..2021_08_09_12_32_53.309779470/cwagentconfig.json ...
2021/08/09 12:32:57 Find symbolic link /etc/cwagentconfig/..data 
2021/08/09 12:32:57 Find symbolic link /etc/cwagentconfig/cwagentconfig.json 
2021/08/09 12:32:57 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
Valid Json input schema.
2021/08/09 12:32:57 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2021/08/09 12:32:58 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/08/09 12:32:59 W! retry [1/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/08/09 12:33:00 W! retry [2/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/08/09 12:33:00 I! access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
 
2021/08/09 12:41:01 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml 
2021-08-09T12:41:01Z I! Starting AmazonCloudWatchAgent 1.247348.0
2021-08-09T12:41:01Z I! Loaded inputs: prometheus_scraper
2021-08-09T12:41:01Z I! Loaded aggregators: 
2021-08-09T12:41:01Z I! Loaded processors: emfProcessor

Could you please help to investigate why it takes so much time and if it could be improved? Any help is appreciated.

[annikulin]

The issue was resolved when we updated the EC2 instance to support IMDSv2. Previously, the CloudWatch agent had to use IMDSv1, and it was causing a delay in startup time.