Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

outputs.http plugin not working with Amazon Credentials Using IAM Role specified in role_arn Setting #12992

Closed
glen-neff opened this issue Mar 30, 2023 · 4 comments
Closed

outputs.http plugin not working with Amazon Credentials Using IAM Role specified in role_arn Setting #12992

glen-neff opened this issue Mar 30, 2023 · 4 comments
Labels
bug unexpected problem or unintended behavior

Comments

@glen-neff
Copy link
Contributor

Relevant telegraf.conf

[[outputs.http]]
   region = "us-east-1"
   url = "https://4afootc.execute-api.us-east-1.amazonaws.com/test0/"
   timeout = "5s"
   method = "POST"
   role_arn = "arn:aws:iam::7171foo89:role/TelegrafInstance"
   access_key = "ASIA2N7fooXLAVEG"
   secret_key = "DLiC83oUGxsPfooTwGqAHirV"
   token = "Ibfoo3JpZ2luX2VjEI3//////////wEaCXVzLWVhc3QtMSJHMEUCIFnhSNz07TjkQ+FDnryZFRhqo7MogLTGuV5SO5TreHtRAiEAsc4uRcofvssgU9otoN33SNOtRAA/725JPkLK5WltHscqvAUIZRAAGgw3MTcxOTYzNjIyODkiDNLOF3XXYHftFX6VViqZBd80RMiX67rSlTnbagAw0klSJM3HmJP0/jiWRQ4hpM5Ln7wBhNF9VMbuPw/nbfC9H7TfRwANWPsJz23iwgw84wPvnB0SBQNSMfke8Ai3wO/Om174vgoA+Usji6M+8p12e2HF7DGDWDMuNOUjgq+OxtNuSz6tRq/J12+JBtK9WW/a379EhikH9XoYIxYnVngUyKnos1Z0fltUOFAv1WriFqDc7pJ63bvQS2a39DWDndUWdyiA9CF/tltKcAjIZFIeYlCCoaXKyGcdZXMeB9lOi1lLdqDeLV1xTblSV7yoDMpOJtdolY8AqGhT0GZLrS4u0sx7WId5ynDD5rzCW08ougPAeEi9FqVyyWtCIjjXoDCSiMFXTlBfookBF1hyXmx9enSFg5t5d5f1/m2M6EZ5z+v6ncHFA0fnhqsV+ECuk5I4wnmaXt21ba6INYjxcdzDv2Mphg0/qKvzoXz4AuNptY+czkVlGpJIW9ok7i1vtoY7fJW1fGF1EoFytpEyiWrSPs7y1ZSd0Rksz+oYOayQb2c2uWp6ig9NJfhY54nQjVRXTNzn3priJ70d5xgwCku0Cx7Bvc68UGDYVxAeBA4Z28ImZ2w8FEInTyQRXNwq0a3bFGfz7D99kzu3FaG6/D1OWmTgF26Jn4e0WPD73JWS+hFATNKvxqUKtsM1JFCCHMp1duG0fdVJfO7DNE7lS4qrRxGaSwSwEF/YNJFQDJMK/LEnPYTIKR6mOhf8PgRCB11gsAcBXPHtGbdjmkEASqPZCUI3GVi5II39sNTwX9ny2Bd8a1yJ1/ZVOWgQzfR6L+BvESnAbwEmVnGA+mv2ut5UzsmQq2ppaIixq1x/5Y8CId1/yq6mZ7Fm00bHWu+ItLgbO7poDXWCSVMJO5kqEGOrEBdJ9wyxUkDQGwb+Yh0DHxTv6AGPvh0ze9I3ZEtH2I5AjuybmHrIYO28/yDG5guX9WGmn77x/bjqw6XM+7hW+Z0swmoKYGlZ3y4imSWDxvEnflSQakfPEzAXz70pw918YhXGYsQTk6wV6Cs+9OX2OmU1RRJfsQ+Td6R7MJn9TKIYQ/ZHWXiv1+V60PLG1WmXDbwJLKHPFeP0ScdoyhvdeE/F+JL6NZewFdEcKRuxPOhN1F"

Logs from Telegraf

2023-03-30T14:48:15Z I! Starting Telegraf 1.26.0
2023-03-30T14:48:15Z I! Available plugins: 235 inputs, 9 aggregators, 27 processors, 22 parsers, 57 outputs, 2 secret-stores
2023-03-30T14:48:15Z I! Loaded inputs: ping
2023-03-30T14:48:15Z I! Loaded aggregators:
2023-03-30T14:48:15Z I! Loaded processors:
2023-03-30T14:48:15Z I! Loaded secretstores:
2023-03-30T14:48:15Z I! Loaded outputs: http
2023-03-30T14:48:15Z I! Tags enabled: host=test501-telegraf site=us-east-1
2023-03-30T14:48:15Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"test501-telegraf", Flush Interval:15s
2023-03-30T14:48:15Z D! [agent] Initializing plugins
2023-03-30T14:48:15Z D! [agent] Connecting outputs
2023-03-30T14:48:15Z D! [agent] Attempting connection to [outputs.http]
2023-03-30T14:48:15Z D! [agent] Successfully connected to outputs.http
2023-03-30T14:48:15Z D! [agent] Starting service inputs
2023-03-30T14:48:30Z D! [outputs.http] Buffer fullness: 0 / 20000 metrics
2023-03-30T14:48:45Z D! [outputs.http] Buffer fullness: 0 / 20000 metrics
2023-03-30T14:49:00Z D! [outputs.http] Buffer fullness: 10 / 20000 metrics
2023-03-30T14:49:00Z E! [agent] Error writing to outputs.http: when writing to [https://4afootc.execute-api.us-east-1.amazonaws.com/test0/] received status code: 403. body: {"message":"Missing Authentication Token"}
2023-03-30T14:49:15Z D! [outputs.http] Buffer fullness: 10 / 20000 metrics
2023-03-30T14:49:15Z E! [agent] Error writing to outputs.http: when writing to [https://4a6footc.execute-api.us-east-1.amazonaws.com/test0/] received status code: 403. body: {"message":"Missing Authentication Token"}

System info

Telegraf 1.26.0 | Debian 11 (bullseye) | Linux telegraf 5.10.0-21-cloud-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Docker

No response

Steps to reproduce

  1. Start Telegraf.
  2. Let it collect data from an inputs.ping test config.
  3. See it log the error: status code: 403. body: {"message":"Missing Authentication Token"}

Expected behavior

Our configuration is:

Telegraf -> AWS API Gateway -> AWS Python Lambda

If this was working, I would expect to see it logged that outputs.http successfully POSTed data to the configured URL and be able to verify receipt of the JSON payload in the CloudWatch logs associated with the AWS Lambda.

FYI, I have seen this work without the API Gateway configured to require IAM Role authentication, with just the access_key, secret_key, and token configured for outputs.http.

I should also mention, in our environment, we do have Telegraf successfully using both outputs.cloudwatch and outputs.timestream working with only the role_arn specified.

Actual behavior

Telegraf logs: status code: 403. body: {"message":"Missing Authentication Token"}

Additional info

The AWS API Gateway that I must point outputs.http towards requires HTTPS - it doesn't even allow for temporary configuration of simple HTTP testing/sniffing purposes.

In a sloppy attempt at being able to sniff the traffic to see what was being sent, I changed the outputs.http URL to an HTTP target where I ran tcpdump. What I discovered was that Telegraf did not appear to be attempting to send any authentication headers:

From tcpdump on the HTTP server:
12:41:13.181258 IP (tos 0x0, ttl 43, id 60206, offset 0, flags [DF], proto TCP (6), length 60)
ec2-52-229.compute-1.amazonaws.com.46784 > foobar.example.com.http: Flags [S], cksum 0x7f34 (correct), seq 3789377456, win 62727, options [mss 1460,sackOK,TS val 980443550 ecr 0,nop,wscale 6], length 0
12:41:13.181318 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
foobar.example.com.http > ec2-52-229.compute-1.amazonaws.com.46784: Flags [S.], cksum 0x2fa0 (correct), seq 125446129, ack 3789377457, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3018285368 ecr 980443550], length 0
12:41:13.194684 IP (tos 0x0, ttl 43, id 60207, offset 0, flags [DF], proto TCP (6), length 52)
ec2-52-229.compute-1.amazonaws.com.46784 > foobar.example.com.http: Flags [.], cksum 0x5a89 (correct), seq 1, ack 1, win 981, options [nop,nop,TS val 980443563 ecr 3018285368], length 0
12:41:13.194902 IP (tos 0x0, ttl 43, id 60208, offset 0, flags [DF], proto TCP (6), length 1500)
ec2-52-229.compute-1.amazonaws.com.46784 > foobar.example.com.http: Flags [.], cksum 0xd0c8 (correct), seq 1:1449, ack 1, win 981, options [nop,nop,TS val 980443563 ecr 3018285368], length 1448: HTTP, length: 1448
POST / HTTP/1.1
Host: www.example.com
User-Agent: Telegraf/1.26.0 Go/1.20.2
Content-Length: 2741
Content-Type: application/json
Accept-Encoding: gzip

    ping,host=test501-telegraf,site=us-east-1,url=208.67.220.222 packets_transmitted=1i,packets_received=1i,ttl=53i,percent_packet_loss=0,minimum_response_ms=0.87931,result_code=0i,average_response_ms=0.87931,maximum_response_ms=0.87931,standard_deviation_ms=0 1680194460000000000
    ping,host=test501-telegraf,site=us-east-1,url=1.1.1.1 percent_packet_loss=0,standard_deviation_ms=0,packets_transmitted=1i,packets_received=1i,minimum_response_ms=0.809671,average_response_ms=0.809671,maximum_response_ms=0.809671,result_code=0i,ttl=52i 1680194460000000000
    ping,host=test501-telegraf,site=us-east-1,url=9.9.9.9 minimum_response_ms=0.767647,maximum_response_ms=0.767647,ttl=53i,percent_packet_loss=0,packets_received=1i,average_response_ms=0.767647,standard_deviation_ms=0,result_code=0i,packets_transmitted=1i 1680194460000000000
    ping,host=test501-telegraf,site=us-east-1,url=208.67.220.220 packets_transmitted=1i,percent_packet_loss=0,minimum_response_ms=0.777673,maximum_response_ms=0.777673,standard_deviation_ms=0,result_code=0i,packets_received=1i,ttl=53i,average_response_ms=0.777673 1680194460000000000
    ping,host=test501-telegraf,site=us-east-1,url=4.2.2.2 result_code=0i,packets_received=1i,average_response_ms=1.602754,packets_transmitted=1i,ttl=52i,percent_packet_loss=0,minimum_respo[!http]
@glen-neff glen-neff added the bug unexpected problem or unintended behavior label Mar 30, 2023
@powersj
Copy link
Contributor

powersj commented Mar 30, 2023

Looking at the output, we do not even try to set up an AWS config, unless aws_service is set. Can you try setting that please?

@powersj powersj added the waiting for response waiting for response from contributor label Mar 30, 2023
@glen-neff
Copy link
Contributor Author

Looking at the output, we do not even try to set up an AWS config, unless aws_service is set. Can you try setting that please?

I'm sorry, could you elaborate on that? Unless aws_service is set where?

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Mar 30, 2023
@glen-neff
Copy link
Contributor Author

Okay, so I tried just sticking aws_service = "yes" under the outputs.http section and it changed the error. To an actual STS IAM Role error. . . stand by while I troubleshoot this further.

@powersj powersj added the waiting for response waiting for response from contributor label Mar 30, 2023
@glen-neff
Copy link
Contributor Author

I ended up having to specify aws_service = "execute-api" to match the AWS API Gateway I'm uploading too.

I'll submit a small documentation PR to hopefully save the next person this grief.

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Mar 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@powersj @glen-neff