Cherry-pick #16892 to 7.x: [Metricbeat] Separate vpc metricset to vpn, transitgateway and natgateway metricsets

CHANGELOG.next.asciidoc

@@ -324,6 +324,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Release ActiveMQ module as GA. {issue}17047[17047] {pull}17049[17049]
 - Add support for CouchDB v2 {issue}16352[16352] {pull}16455[16455]
 - Release Zookeeper/connection module as GA. {issue}14281[14281] {pull}17043[17043]
+- Replace vpc metricset into vpn, transitgateway and natgateway metricsets. {pull}16892[16892]
 
 *Packetbeat*
 

metricbeat/docs/fields.asciidoc

@@ -1972,6 +1972,12 @@ type: integer
 `lambda` contains the metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS Lambda.
 
 
+[float]
+=== natgateway
+
+`natgateway` contains the metrics from Cloudwatch to track usage of NAT gateway related resources.
+
+
 [float]
 === rds
 
@@ -3082,16 +3088,22 @@ type: keyword
 
 --
 
+[float]
+=== transitgateway
+
+`transitgateway` contains the metrics from Cloudwatch to track usage of transit gateway related resources.
+
+
 [float]
 === usage
 
 `usage` contains the metrics from Cloudwatch to track usage of some AWS resources.
 
 
 [float]
-=== vpc
+=== vpn
 
-`vpc` contains the metrics from Cloudwatch to track usage of VPC related resources.
+`vpn` contains the metrics from Cloudwatch to track usage of VPN related resources.
 
 
 [[exported-fields-azure]]

metricbeat/docs/modules/aws.asciidoc

@@ -35,8 +35,8 @@ image::./images/metricbeat-aws-overview.png[]
 == Metricsets
 
 Currently, we have `billing`, `cloudwatch`, `dynamodb`, `ebs`, `ec2`, `elb`,
-`lambda`, `rds`, `s3_daily_storage`, `s3_request`, `sns`, `sqs` and `usage`
-metricset in `aws` module.
+`lambda`, `natgateway`, `rds`, `s3_daily_storage`, `s3_request`, `sns`, `sqs`,
+`transitgateway`, `usage` and `vpn` metricset in `aws` module.
 
 Collecting `tags` for `ec2`, `rds`, `cloudwatch`, and metricset created based on
 `cloudwatch` using light module is supported.
@@ -124,6 +124,13 @@ The lambda metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-aws-lambda-overview.png[]
 
+[float]
+=== `natgateway`
+CloudWatch collects information from NAT gateways and creates readable, near real-time metrics.
+This metricset enables users to collect these metrics from CloudWatch to monitor and troubleshoot
+their NAT gateway. NAT gateway metric data is provided at 1-minute intervals and therefore,
+`period` for `natgateway` metricset is recommended to be `1m` or multiples of `1m`.
+
 [float]
 === `rds`
 `period` for `rds` metricset is recommended to be `60s` or multiples of `60s` because Amazon RDS sends metrics and
@@ -162,6 +169,13 @@ The sqs metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-aws-sqs-overview.png[]
 
+[float]
+=== `transitgateway`
+Amazon VPC reports metrics to CloudWatch only when requests are flowing through
+the transit gateway. If there are requests flowing through the transit gateway,
+Amazon VPC measures and sends its metrics in 60-second intervals. `period` for
+`transitgateway` metricset is recommended to be `1m` or multiples of `1m`.
+
 [float]
 === `usage`
 CloudWatch collects metrics that track the usage of some AWS resources. These
@@ -174,6 +188,12 @@ The usage metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-aws-usage-overview.png[]
 
+[float]
+=== `vpn`
+CloudWatch collects and processes raw data from the VPN service into readable, near
+real-time metrics for users to better understand the performance of their web
+applications and services.
+
 [float]
 == AWS API requests count per metricset
 This session is to document what are the AWS API called made by each metricset
@@ -305,7 +325,12 @@ metricbeat.modules:
     - key: "dept"
       value: "eng"
   metricsets:
+    - elb
+    - natgateway
     - rds
+    - transitgateway
+    - usage
+    - vpn
 ----
 
 [float]
@@ -327,6 +352,8 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-aws-lambda,lambda>>
 
+* <<metricbeat-metricset-aws-natgateway,natgateway>>
+
 * <<metricbeat-metricset-aws-rds,rds>>
 
 * <<metricbeat-metricset-aws-s3_daily_storage,s3_daily_storage>>
@@ -337,9 +364,11 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-aws-sqs,sqs>>
 
+* <<metricbeat-metricset-aws-transitgateway,transitgateway>>
+
 * <<metricbeat-metricset-aws-usage,usage>>
 
-* <<metricbeat-metricset-aws-vpc,vpc>>
+* <<metricbeat-metricset-aws-vpn,vpn>>
 
 include::aws/billing.asciidoc[]
 
@@ -355,6 +384,8 @@ include::aws/elb.asciidoc[]
 
 include::aws/lambda.asciidoc[]
 
+include::aws/natgateway.asciidoc[]
+
 include::aws/rds.asciidoc[]
 
 include::aws/s3_daily_storage.asciidoc[]
@@ -365,7 +396,9 @@ include::aws/sns.asciidoc[]
 
 include::aws/sqs.asciidoc[]
 
+include::aws/transitgateway.asciidoc[]
+
 include::aws/usage.asciidoc[]
 
-include::aws/vpc.asciidoc[]
+include::aws/vpn.asciidoc[]
 

metricbeat/docs/modules/aws/natgateway.asciidoc

@@ -0,0 +1,24 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-aws-natgateway]]
+=== aws natgateway metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/aws/natgateway/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-aws,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/aws/natgateway/_meta/data.json[]
+----

metricbeat/docs/modules/aws/transitgateway.asciidoc

@@ -0,0 +1,24 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-aws-transitgateway]]
+=== aws transitgateway metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/aws/transitgateway/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-aws,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/aws/transitgateway/_meta/data.json[]
+----

metricbeat/docs/modules/aws/vpc.asciidoc → metricbeat/docs/modules/aws/vpn.asciidoc

@@ -2,12 +2,12 @@
 This file is generated! See scripts/mage/docs_collector.go
 ////
 
-[[metricbeat-metricset-aws-vpc]]
-=== aws vpc metricset
+[[metricbeat-metricset-aws-vpn]]
+=== aws vpn metricset
 
 beta[]
 
-include::../../../../x-pack/metricbeat/module/aws/vpc/_meta/docs.asciidoc[]
+include::../../../../x-pack/metricbeat/module/aws/vpn/_meta/docs.asciidoc[]
 
 This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
 
@@ -20,5 +20,5 @@ Here is an example document generated by this metricset:
 
 [source,json]
 ----
-include::../../../../x-pack/metricbeat/module/aws/vpc/_meta/data.json[]
+include::../../../../x-pack/metricbeat/module/aws/vpn/_meta/data.json[]
 ----

metricbeat/docs/modules_list.asciidoc

@@ -16,20 +16,22 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-module-appsearch,App Search>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .1+| .1+|  |<<metricbeat-metricset-appsearch-stats,stats>> beta[]  
 |<<metricbeat-module-aws,aws>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
-.14+| .14+|  |<<metricbeat-metricset-aws-billing,billing>> beta[]  
+.16+| .16+|  |<<metricbeat-metricset-aws-billing,billing>> beta[]  
 |<<metricbeat-metricset-aws-cloudwatch,cloudwatch>>   
 |<<metricbeat-metricset-aws-dynamodb,dynamodb>> beta[]  
 |<<metricbeat-metricset-aws-ebs,ebs>>   
 |<<metricbeat-metricset-aws-ec2,ec2>>   
 |<<metricbeat-metricset-aws-elb,elb>>   
 |<<metricbeat-metricset-aws-lambda,lambda>> beta[]  
+|<<metricbeat-metricset-aws-natgateway,natgateway>> beta[]  
 |<<metricbeat-metricset-aws-rds,rds>>   
 |<<metricbeat-metricset-aws-s3_daily_storage,s3_daily_storage>>   
 |<<metricbeat-metricset-aws-s3_request,s3_request>>   
 |<<metricbeat-metricset-aws-sns,sns>> beta[]  
 |<<metricbeat-metricset-aws-sqs,sqs>>   
+|<<metricbeat-metricset-aws-transitgateway,transitgateway>> beta[]  
 |<<metricbeat-metricset-aws-usage,usage>> beta[]  
-|<<metricbeat-metricset-aws-vpc,vpc>> beta[]  
+|<<metricbeat-metricset-aws-vpn,vpn>> beta[]  
 |<<metricbeat-module-azure,azure>>  beta[]   |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .8+| .8+|  |<<metricbeat-metricset-azure-compute_vm,compute_vm>> beta[]  
 |<<metricbeat-metricset-azure-compute_vm_scaleset,compute_vm_scaleset>> beta[]  

x-pack/metricbeat/metricbeat.reference.yml

@@ -229,7 +229,12 @@ metricbeat.modules:
     - key: "dept"
       value: "eng"
   metricsets:
+    - elb
+    - natgateway
     - rds
+    - transitgateway
+    - usage
+    - vpn
 
 #-------------------------------- Azure Module --------------------------------
 - module: azure

x-pack/metricbeat/module/aws/_meta/config.reference.yml

@@ -45,4 +45,9 @@
     - key: "dept"
       value: "eng"
   metricsets:
+    - elb
+    - natgateway
     - rds
+    - transitgateway
+    - usage
+    - vpn

x-pack/metricbeat/module/aws/_meta/config.yml

@@ -2,7 +2,11 @@
   period: 1m
   metricsets:
     - elb
+    - natgateway
+    - rds
+    - transitgateway
     - usage
+    - vpn
 - module: aws
   period: 5m
   metricsets:

x-pack/metricbeat/module/aws/_meta/docs.asciidoc

@@ -27,8 +27,8 @@ image::./images/metricbeat-aws-overview.png[]
 == Metricsets
 
 Currently, we have `billing`, `cloudwatch`, `dynamodb`, `ebs`, `ec2`, `elb`,
-`lambda`, `rds`, `s3_daily_storage`, `s3_request`, `sns`, `sqs` and `usage`
-metricset in `aws` module.
+`lambda`, `natgateway`, `rds`, `s3_daily_storage`, `s3_request`, `sns`, `sqs`,
+`transitgateway`, `usage` and `vpn` metricset in `aws` module.
 
 Collecting `tags` for `ec2`, `rds`, `cloudwatch`, and metricset created based on
 `cloudwatch` using light module is supported.
@@ -116,6 +116,13 @@ The lambda metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-aws-lambda-overview.png[]
 
+[float]
+=== `natgateway`
+CloudWatch collects information from NAT gateways and creates readable, near real-time metrics.
+This metricset enables users to collect these metrics from CloudWatch to monitor and troubleshoot
+their NAT gateway. NAT gateway metric data is provided at 1-minute intervals and therefore,
+`period` for `natgateway` metricset is recommended to be `1m` or multiples of `1m`.
+
 [float]
 === `rds`
 `period` for `rds` metricset is recommended to be `60s` or multiples of `60s` because Amazon RDS sends metrics and
@@ -154,6 +161,13 @@ The sqs metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-aws-sqs-overview.png[]
 
+[float]
+=== `transitgateway`
+Amazon VPC reports metrics to CloudWatch only when requests are flowing through
+the transit gateway. If there are requests flowing through the transit gateway,
+Amazon VPC measures and sends its metrics in 60-second intervals. `period` for
+`transitgateway` metricset is recommended to be `1m` or multiples of `1m`.
+
 [float]
 === `usage`
 CloudWatch collects metrics that track the usage of some AWS resources. These
@@ -166,6 +180,12 @@ The usage metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-aws-usage-overview.png[]
 
+[float]
+=== `vpn`
+CloudWatch collects and processes raw data from the VPN service into readable, near
+real-time metrics for users to better understand the performance of their web
+applications and services.
+
 [float]
 == AWS API requests count per metricset
 This session is to document what are the AWS API called made by each metricset