Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CloudWatch: High resolution is not work #20436

Closed
mtanda opened this issue Nov 18, 2019 · 5 comments · Fixed by #20440
Closed

CloudWatch: High resolution is not work #20436

mtanda opened this issue Nov 18, 2019 · 5 comments · Fixed by #20440

Comments

@mtanda
Copy link
Collaborator

mtanda commented Nov 18, 2019

What happened:
High resolution query cause error.
The error message is too many datapoint requested in query A. Please try to reduce the time range.

What you expected to happen:
Get success response.

How to reproduce it (as minimally and precisely as possible):

  • Check high resolution
  • Set time range to 30 days
  • Select 12 or more dimensions (the expected response size should exceed 100,800 data points)
    • If test with AWS/EC2 CPUUtilization (the metrics resolution is 5 min), 100800 / ((60 / 5) * 24 * 30) = 11.6, select 12 InstanceId

Anything else we need to know?:
Original implementation is #10925 .

Environment:

  • Grafana version: v6.5.0 beta-1
  • Data source type & version: CloudWatch
  • OS Grafana is installed on: Mac
  • User OS & Browser: Firefox
  • Grafana plugins:
  • Others:
@hugohaggmark
Copy link
Contributor

Thanks @mtanda for reporting this!

@sunker sunker added this to the 6.5.0-stable milestone Nov 18, 2019
@marefr marefr modified the milestones: 6.5.0-stable, 6.5.0-beta2 Nov 18, 2019
@sunker
Copy link
Contributor

sunker commented Nov 18, 2019

Thanks for reporting this @mtanda.

I'm afraid I don't see how we could support HighRes now that we use GMD instead of GMS. The reason is that even though there's plenty of queries in the panel, we only make one call to GetMetricData API. That call uses the same startTime and endTime for all queries, so really there's no way to define HighRes on a per query level anymore. Or do you have any solution to this @mtanda? That is something that does not include multiple GMD calls.

So the error message that you've received is expected - a workaround is to reduce the timespan. I suggest I remove the HighRes toggle from the GUI and also describe this as a breaking change in the changelog.

@sunker
Copy link
Contributor

sunker commented Nov 18, 2019

@mtanda I'm also available on grafanas public slack if you'd prefer to discuss this over there.

@mtanda
Copy link
Collaborator Author

mtanda commented Nov 19, 2019

I send a partial fix of this issue.
#20459

If the query doesn't include expression, datasource can handle high resolution.

@mtanda
Copy link
Collaborator Author

mtanda commented Nov 19, 2019

The hardest limitation of CloudWatch API is, NextToken is not returned when query has expression.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDataResult.html

The status of the returned data. Complete indicates that all data points in the requested time range were returned. PartialData means that an incomplete set of data points were returned. You can use the NextToken value that was returned and repeat your request to get more data points. NextToken is not returned if you are performing a math expression. InternalError indicates that an error occurred. Retry your request using NextToken, if present.

New GMD implementation utilize SEARCH() expression for expanding template variable.
So, I think only solution is splitting original query to multiple SEARCH() expression query...

@sunker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants