Telegraf unable to write metrics to CloudWatch with IMDSv1 disabled

[anilkun]

Hi Team,

I have an EC2 instance where AWS IMDSV1 is disabled and AWS IMDSV2 is enabled for security reasons. The instance is attached with IAM role with enough privileges. When I deploy telegraf with cloudwatch output plugin, telegraf is unable to write metrics to CloudWatch because it cannot find aws credentials or config file. I tried with providing role_arn and profile but the issue persists.

Relevant telegraf.conf:

[[outputs.cloudwatch]]
region = "us-west-2"
namespace = "test"
role_arn = ""

System info:

Ubuntu 18.04

Steps to reproduce:

1. Disable AWS IMDSV1 on an EC2 instance
2. Configure telegraf cloudwatch output plugin
3. Attach IAM role to EC2 instance
4. Start telegraf

Actual behavior:

systemctl status telegraf
● telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB
   Loaded: loaded (/lib/systemd/system/telegraf.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-04-19 03:52:07 UTC; 1min 54s ago
     Docs: https://github.com/influxdata/telegraf
 Main PID: 12607 (telegraf)
    Tasks: 11 (limit: 4451)
   CGroup: /system.slice/telegraf.service
           └─12607 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

Apr 19 03:52:07 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:52:07Z D! [agent] Initializing plugins
Apr 19 03:52:07 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:52:07Z D! [agent] Connecting outputs
Apr 19 03:52:07 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:52:07Z D! [agent] Attempting connection to [outputs.cloudwatch]
Apr 19 03:52:07 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:52:07Z D! [agent] Successfully connected to outputs.cloudwatch
Apr 19 03:52:07 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:52:07Z D! [agent] Starting service inputs
Apr 19 03:54:00 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:54:00Z E! CloudWatch: Unable to write to CloudWatch : NoCredentialProviders: no valid providers in chain. Deprecated.
Apr 19 03:54:00 ip-xxx-xx-x-xxx telegraf[12607]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Apr 19 03:54:00 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:54:00Z D! [outputs.cloudwatch] Buffer fullness: 6 / 10000 metrics
Apr 19 03:54:00 ip-xxx-xx-x-xxx telegraf[12607]: 2020-04-19T03:54:00Z E! [agent] Error writing to outputs.cloudwatch: NoCredentialProviders: no valid providers in chain. Deprecated.
Apr 19 03:54:00 ip-xxx-xx-x-xxx telegraf[12607]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors