[MetricBeat] AWS module support wildcard format dimension value to reduce the metrics count

[kwinstonix]

1. Describe the enhancement:

AWS CloudWatch metrics have many dimensions, for example, In AWS/NetworkELB namespace, UnHealthyHostCount has the dimensions:

• TargetGroup, LoadBalancer
• TargetGroup, LoadBalancer, AvailableZone

When we query CloudWatch metrics with GetMetricData API, there are two metric.

        {
            "Namespace": "AWS/NetworkELB", 
            "Dimensions": [
                {
                    "Name": "TargetGroup", 
                    "Value": "targetgroup/some-name/222222"
                }, 
                {
                    "Name": "AvailabilityZone", 
                    "Value": "ap-northeast-1a"
                }, 
                {
                    "Name": "LoadBalancer", 
                    "Value": "net/some-lb-name/1111111"
                }
            ], 
            "MetricName": "UnHealthyHostCount"
        },

                {
            "Namespace": "AWS/NetworkELB", 
            "Dimensions": [
                {
                    "Name": "TargetGroup", 
                    "Value": "targetgroup/some-name/222222"
                }, 
                {
                    "Name": "LoadBalancer", 
                    "Value": "net/some-lb-name/1111111"
                }
            ], 
            "MetricName": "UnHealthyHostCount"
        }, 

---

If metricbeat could limit dimension names scope, after compared to ListMetrics API result, just one metric is retrived from AWS. so this can reduce AWS GetMetircData API cost and metricbeat events amount.

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/NetworkELB
      statistic: ["Maximum"]
      name: ["UnHealthyHostCount"]
     # demo param
      dimension_names: ["LoadBalancer", "TargetGroup"]
      tags.resource_type_filter: elasticloadbalancing

metricbeat will only make one metric query:

        {
            "Namespace": "AWS/NetworkELB", 
            "Dimensions": [
                {
                    "Name": "TargetGroup", 
                    "Value": "targetgroup/some-name/222222"
                }, 
                {
                    "Name": "LoadBalancer", 
                    "Value": "net/some-lb-name/1111111"
                }
            ], 
            "MetricName": "UnHealthyHostCount"
        }, 

2. Describe a specific use case for the enhancement or feature:

• reduce AWS GetMetricData cost
• reduce metricbeat events amount

3. reference

• aws doc:

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html

• inspired by:

https://github.com/prometheus/cloudwatch_exporter#cost

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

Thanks for creating this issue! Right now in CloudWatch metricset, we have a config parameter called dimensions: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-cloudwatch.html#_metricset_specific_configuration_notes
But this dimensions requires key-value pairs. So user has to specify a value for the specific dimension name. For example:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/NetworkELB
      statistic: ["Maximum"]
      name: ["UnHealthyHostCount"]
      dimensions: 
        - name: LoadBalancer
          value: "net/some-lb-name/1111111"
        - name: TargetGroup
          value: "targetgroup/some-name/222222"
      tags.resource_type_filter: elasticloadbalancing

Maybe we should add support for using wildcard * in dimension values to indicate all/any value for a specific dimension name:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/NetworkELB
      statistic: ["Maximum"]
      name: ["UnHealthyHostCount"]
      dimensions: 
        - name: LoadBalancer
          value: *
        - name: TargetGroup
          value: *
      tags.resource_type_filter: elasticloadbalancing

With value: *, cloudwatch metricset should behave the same as the demo param you gave.

[kwinstonix]

Yes , you are right, dimensions value: * can figure out this, wildcard value is more flexible . I read master version document and found dimensions ， but I forgot it when I wrote the issue.

thanks for your reply.

[kwinstonix]

Maybe we should add support for using wildcard * in dimension values to indicate all/any value for a specific dimension name

Will this improvement be added ? 😁

[kaiyan-sheng]

@kwinstonix Yes, sounds like a good enhancement to have! I will keep this issue open to track the work for enabling * in dimension value. Thank you!

[kwinstonix]

Is there any progress on this issue ? 🤔

[kwinstonix]

@kaiyan-sheng , I have created a PR to implement this feature, could you help me review it?

[kaiyan-sheng]

@kwinstonix Thank you and will do!!