AWS Metricbeat Module

[kaiyan-sheng]

This is a meta issue to track the work on adding AWS metricbeat module including varies metricsets.
This checklist is intended for Devs which create or update a module to make sure modules are consistent.

• EC2 metricset (Add Metricbeat AWS EC2 Module #9257)
• SQS metricset ([Metricbeat] Adding AWS SQS Metricset #10053)
• RDS metricset ([Metricbeat] Add AWS RDS Metricset #10054)
• S3 metricset ([Metricbeat] Add AWS S3 Metricset #10055)
• EBS metricset ([Metricbeat] Adding AWS EBS Metricset #11699)
• ELB metricset ([Metricbeat] Adding AWS ELB Metricset #11701)
• Cloudwatch metricset ([Metricbeat] Adding AWS Cloudwatch Metricset #11734)

[kaiyan-sheng]

One enhancement issue: #11820

[kaiyan-sheng]

Adding tags support in AWS module: #12263

[Giaco9]

Hi @kaiyan-sheng, I don't know if this is the right place to ask questions, but I'm very interested in the AWS module especially for EC2 metrics, this is a really cool feature! My question is: Is there any way to collect only selected metrics and get them only for a subset of my ec2 instances running in a region? Would be great filter them by tags or name. Should I open an issue for that?

Thanks!

[kaiyan-sheng]

Hi @Giaco9, thanks for reaching out! If you only want to collect metrics from a specific region, then you can use

- module: aws
  period: 300s
  metricsets:
    - ec2
  regions:
    - us-east-1

If you want to specify both ec2 instances and regions, then you can try cloudwatch metricset:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization", "DiskWriteOps"]
      tags.resource_type_filter: ec2:intance
      dimensions:
        - name: InstanceId
          value: i-0686946e22cf9494a

For ec2 metricset, tags will be collected by default so if you have tags for the instances you are interested, you can also collect all metrics and then filter by tags in Kibana for visualization.

[kaiyan-sheng]

@Giaco9 Filtering by tags/names to decide what are the metrics should be collected (before collecting all metrics) is a feature we don't have right now. Please correct me if I understood wrong 😄 I created a new issue for this #13145, please feel free to comment on it. Thanks!