Migrate to AWS SDK v2

[anas-aso]

Signed-off-by: Anas [email protected]

Fixes #172

@brian-brazil can I have your review please ?

[brian-brazil]

From a cursory review this looks okay, however I expect the devil is in the details. What sort of manual testing have you done on this?

[anas-aso]

@brian-brazil I have addressed your reviews. Can you please have a 2nd look ?

From a cursory review this looks okay, however I expect the devil is in the details. What sort of manual testing have you done on this?

I run the latest released version and this PR locally, fetch the metrics (wget) and compare the output (I did it multiple times). The result is always like this :

2a3,6
> # HELP cloudwatch_requests_total API requests made to CloudWatch
> # TYPE cloudwatch_requests_total counter
> cloudwatch_requests_total{action="getMetricStatistics",namespace="AWS/SQS",} 5117.0
> cloudwatch_requests_total{action="listMetrics",namespace="AWS/SQS",} 119.0
637c641
< cloudwatch_exporter_scrape_duration_seconds 25.683923456
---
> cloudwatch_exporter_scrape_duration_seconds 25.514181182
641,644d644
< # HELP cloudwatch_requests_total API requests made to CloudWatch
< # TYPE cloudwatch_requests_total counter
< cloudwatch_requests_total{action="getMetricStatistics",namespace="AWS/SQS",} 27692.0
< cloudwatch_requests_total{action="listMetrics",namespace="AWS/SQS",} 644.0

I am also running a local Prometheus and it's scrapping both version of the exporter :

Not perfect, but should give a rough idea about the returned results by both versions of the exporter.

PS: I am importing SQS metrics only.

If you have a better idea for manual testing I can give it a try (if it doesn't require too much time).

[brian-brazil]

Try all the config options basically, the tag ones in particular as that's a separate code path. Something is likely to break with a change like this, but we should try to minimise the chances of it.

[anas-aso]

@brian-brazil I have tried all the config options and all seems fine. It was time consuming but I think it was worth it since I found something that I missed before ecd822f

Regarding the tagging feature, here is the results of the tests I did:

Test 1: Without tag filtering

Using the config below:

region: eu-central-1
metrics:
  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: RequestCount
    aws_dimensions: [LoadBalancer]
    aws_statistics: [Sum]

I got the following results

# HELP cloudwatch_requests_total API requests made to CloudWatch
# TYPE cloudwatch_requests_total counter
cloudwatch_requests_total{action="listMetrics",namespace="AWS/ApplicationELB",} 4.0
cloudwatch_requests_total{action="getMetricStatistics",namespace="AWS/ApplicationELB",} 33.0
# HELP tagging_api_requests_total API requests made to the Resource Groups Tagging API
# TYPE tagging_api_requests_total counter
# HELP aws_applicationelb_request_count_sum CloudWatch metric AWS/ApplicationELB RequestCount Dimensions: [LoadBalancer] Statistic: Sum Unit: Count
# TYPE aws_applicationelb_request_count_sum gauge
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 14.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 6.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 2.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 4.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 3.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 4.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 4.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 0.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 2.0 1604676480000
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 2.0 1604676480000
# HELP aws_resource_info AWS information available for resource
# TYPE aws_resource_info gauge
# HELP cloudwatch_exporter_scrape_duration_seconds Time this CloudWatch scrape took, in seconds.
# TYPE cloudwatch_exporter_scrape_duration_seconds gauge
cloudwatch_exporter_scrape_duration_seconds 0.377298838
# HELP cloudwatch_exporter_scrape_error Non-zero if this scrape failed.
# TYPE cloudwatch_exporter_scrape_error gauge
cloudwatch_exporter_scrape_error 0.0

Test 2: With tag filtering

Using the config below:

region: eu-central-1
metrics:
  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: RequestCount
    aws_dimensions: [LoadBalancer]
    aws_statistics: [Sum]
    aws_tag_select:
      tag_selections:
        kubernetes.io/ingress-name: ["REDACTED"]
      resource_type_selection: "elasticloadbalancing:loadbalancer"
      resource_id_dimension: LoadBalancer

I got the following results

# HELP cloudwatch_requests_total API requests made to CloudWatch
# TYPE cloudwatch_requests_total counter
cloudwatch_requests_total{action="listMetrics",namespace="AWS/ApplicationELB",} 10.0
cloudwatch_requests_total{action="getMetricStatistics",namespace="AWS/ApplicationELB",} 64.0
# HELP tagging_api_requests_total API requests made to the Resource Groups Tagging API
# TYPE tagging_api_requests_total counter
tagging_api_requests_total{action="getResources",resource_type="elasticloadbalancing:loadbalancer",} 4.0
# HELP aws_applicationelb_request_count_sum CloudWatch metric AWS/ApplicationELB RequestCount Dimensions: [LoadBalancer] Statistic: Sum Unit: Count
# TYPE aws_applicationelb_request_count_sum gauge
aws_applicationelb_request_count_sum{job="aws_applicationelb",instance="",load_balancer="app/REDACTED",} 6.0 1604676480000
# HELP aws_resource_info AWS information available for resource
# TYPE aws_resource_info gauge
aws_resource_info{job="aws_applicationelb",instance="",arn="arn:aws:elasticloadbalancing:eu-central-1:REDACTED:loadbalancer/app/REDACTED",load_balancer="app/REDACTED",tag_ingress_k8s_aws_cluster="REDACTED",tag_ingress_k8s_aws_resource="LoadBalancer",tag_ingress_k8s_aws_stack="REDACTED",tag_kubernetes_io_cluster_REDACTED="REDACTED",tag_kubernetes_io_ingress_name="REDACTED",tag_kubernetes_io_namespace="REDACTED",} 1.0
# HELP cloudwatch_exporter_scrape_duration_seconds Time this CloudWatch scrape took, in seconds.
# TYPE cloudwatch_exporter_scrape_duration_seconds gauge
cloudwatch_exporter_scrape_duration_seconds 0.138822136
# HELP cloudwatch_exporter_scrape_error Non-zero if this scrape failed.
# TYPE cloudwatch_exporter_scrape_error gauge
cloudwatch_exporter_scrape_error 0.0

[anas-aso]

@brian-brazil is there anything else needed for this PR ?

[brian-brazil]

I haven't had a chance to test this out myself yet.

[brian-brazil]

Thanks!