AWS Cost Explorer example

[yesoreyeram]

The following example shows how to get the AWS Cost details in Grafana using Infinity Datasource

For initial setup of the datasource, follow the same instructions given in #501 except couple of changes as we are going to use https://ce.us-east-1.amazonaws.com for the cost explorer queries

• For service, choose ce
• For region, choose us-east-1
• For the allowed hosts, add https://ce.us-east-1.amazonaws.com
• Ensure the IAM user have enough permissions as specified here.

Config Editor

Query Editor

For example, we are going to use AWS GetCostAndUsage API.

The main thing we notice is the headers specified in the query. We have added X-Amz-Target AWSInsightsIndexService.GetCostAndUsage and Content-Type application/x-amz-json-1.1 headers to ensure that the entire flow is working as expected.

We are also going to use POST method with some payload as described in the API document. For adding payload/headers, click the OPTIONS next to the URL.

Actual Query JSON is given below for reference

{
    "refId": "A",
    "datasource": {
      "type": "yesoreyeram-infinity-datasource",
      "uid": "xxxxxx"
    },
    "type": "json",
    "source": "url",
    "format": "table",
    "parser": "backend"
    "url": "https://ce.us-east-1.amazonaws.com",
    "url_options": {
      "method": "POST",
      "data": "{\n\t\t\"TimePeriod\" : { \n      \"Start\" : \"${__from:date:YYYY-MM-DD}\", \n      \"End\" : \"${__to:date:YYYY-MM-DD}\" \n    },\n\t\t\"Granularity\" : \"MONTHLY\",\n\t\t\"Metrics\" : [\"BlendedCost\" ]\n}",
      "body_type": "raw",
      "body_content_type": "application/json",
      "headers": [
        {
          "key": "X-Amz-Target",
          "value": "AWSInsightsIndexService.GetCostAndUsage"
        },
        {
          "key": "Content-Type",
          "value": "application/x-amz-json-1.1"
        }
      ]
    },
    "root_selector": "ResultsByTime",
    "columns": [
      {
        "text": "",
        "selector": "TimePeriod.Start",
        "type": "timestamp",
        "timestampFormat": "2006-01-02"
      },
      {
        "text": "",
        "selector": "Total.BlendedCost.Amount",
        "type": "number"
      }
    ]
 }

body content

{
    "TimePeriod" : { 
      "Start" : "${__from:date:YYYY-MM-DD}", 
      "End" : "${__to:date:YYYY-MM-DD}" 
    },
   "Granularity" : "MONTHLY",
   "Metrics" : [ "BlendedCost" ]
}

raw response without root selector and fields

[jamescarignan]

I'm trying to run this query as shown, but receive "error getting data frame. 400 Bad Request". I've made the same API call successfully in Postman, so I'm not sure what the problem is. Here's the query JSON:

[ { "schema": { "refId": "A", "meta": { "typeVersion": [ 0, 0 ], "custom": { "data": null, "duration": 121720577, "error": "400 Bad Request", "query": { "alias": "", "columns": [], "computed_columns": [], "csv_options": { "columns": "", "comment": "", "delimiter": "", "relax_column_count": false, "skip_empty_lines": false, "skip_lines_with_error": false }, "data": "", "dataOverrides": null, "expression": "", "filterExpression": "", "filters": [], "format": "table", "global_query_id": "", "groq": "", "json_options": { "columnar": false, "root_is_not_array": false }, "parser": "backend", "query_mode": "", "refId": "A", "root_selector": "", "seriesCount": 0, "source": "url", "sqlite_query": "", "summarizeBy": "", "summarizeExpression": "", "type": "json", "uql": "", "url": "https://ce.us-east-1.amazonaws.com", "url_options": { "body_content_type": "application/json", "body_form": null, "body_graphql_query": "", "body_graphql_variables": "", "body_type": "raw", "data": "{\n \"TimePeriod\" : { \n \"Start\" : \"2024-01-01\", \n \"End\" : \"2024-01-31\" \n },\n \"Granularity\" : \"MONTHLY\",\n \"Metrics\" : [ \"BlendedCost\" ]\n}", "headers": [ { "key": "X-Amz-Target", "value": "AWSInsightsIndexService.GetCostAndUsage" }, { "key": "Content-Type", "value": "application/x-amz-json-1.1" } ], "method": "POST", "params": null } }, "responseCodeFromServer": 400 }, "executedQueryString": "###############\n## URL\n###############\n\nhttps://ce.us-east-1.amazonaws.com\n\n###############\n## Curl Command\n###############\n\ncurl -X 'POST' -d '{\n \"TimePeriod\" : { \n \"Start\" : \"2024-01-01\", \n \"End\" : \"2024-01-31\" \n },\n \"Granularity\" : \"MONTHLY\",\n \"Metrics\" : [ \"BlendedCost\" ]\n}' -H 'Accept: application/json;q=0.9,text/plain' -H 'Content-Type: xxxxxxxx' -H 'X-Amz-Target: xxxxxxxx' 'https://ce.us-east-1.amazonaws.com'\n###############\n> Authentication steps not included for AWS authentication" }, "name": "A", "fields": [] }, "data": { "values": [] } } ]

I should also mention that I can successfully implement the ListMetrics example using the monitoring API.

Any idea what I'm doing wrong here?

[jeremygiberson]

I was experiencing the same issue. It turned out that the cost explorer APIs are very sensitive to the request parameters. TimePeriod start and end dates are particularly finicky. You need to make sure you're actually sending in a valid time range (for the service and data available) don't use future dates, use start date earlier than end date, etc.

The issue I had ultimately was due to the TimePeriod start/end not being different dates. I was using a query relative time and timeshift options to change the desired query range from what was currently selected on the dashboard date picker. However the $__from and $__to don't respect the query option overrides (see #855). I ended up using $__timeFrom and $__timeTo to make sure the datasource used the correct date values.

I recommend starting with a hard coded date range in the past with a daily granularity and a single metric unblended cost. See if you can get that working, then start tweaking it to get what you want.