diff --git a/packages/aws/_dev/build/docs/README.md b/packages/aws/_dev/build/docs/README.md index 0ffb2480638..cbd86320411 100644 --- a/packages/aws/_dev/build/docs/README.md +++ b/packages/aws/_dev/build/docs/README.md @@ -17,7 +17,7 @@ This may include every user request that CloudFront receives, every action taken by an AWS user or role, and more. **Metrics** give you insight into the state of your AWS services. -his may include understanding where you're spending the most and why, the volume of storage you're using, +This may include understanding where you're spending the most and why, the volume of storage you're using, CPU utilization of your instances, and more. For a complete list of all AWS services and the data streams available for each, see [Reference](#reference). @@ -188,12 +188,26 @@ Use the AWS integration to connect to your AWS account and collect data from mul When you configure the integration, you can collect data from as many AWS services as you'd like. If you only need to collect data from one AWS service, consider using the individual integration -(for example, to only collect billing metrics, you can use the -**AWS CloudFront** integration). +(for example, to only collect monitoring metrics for EC2, you can configure only the **AWS EC2** integration). For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide. +## Debug +### Latency causes missing metrics +Some AWS services send monitoring metrics to CloudWatch with a latency to process larger than the integration collection +period. This will cause data points missing or none get collected by the agent. In this case, please specify a +latency parameter so collection start time and end time will be shifted by the given latency amount. + +In order to check how much the latency is, you can log into the AWS CloudWatch portal. Wait till a new point to +show up in AWS CloudWatch and record the current timestamp. Compare the timestamp of this latest data point with the +current timestamp to see what's the difference. This difference can be used as latency. + +For example, the screenshot below is taken at `2023-05-09 22:30 UTC` and the timestamp for the last data point is +`2023-05-09 22:15 UTC`. This means there is a 15min delay between the current time and CloudWatch. With this information, +we should add a `latency` configuration for `15m` when adding the integration. +![alt text](../img/metricbeat-aws-cloudwatch-latency.png "CloudWatch Last Data Point Timestamp") + ## Reference Below is an overview of the type of data you can collect from each AWS service. diff --git a/packages/aws/changelog.yml b/packages/aws/changelog.yml index e95ff983f73..e2f6e888065 100644 --- a/packages/aws/changelog.yml +++ b/packages/aws/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "1.43.2" + changes: + - description: Add documentation for latency parameter + type: enhancement + link: https://github.com/elastic/integrations/pull/6231 - version: "1.43.1" changes: - description: Add tags_filter and include_linked_accounts config parameter in missing metric data streams. diff --git a/packages/aws/data_stream/cloudwatch_metrics/manifest.yml b/packages/aws/data_stream/cloudwatch_metrics/manifest.yml index 0ded668b37a..9e45991ba68 100644 --- a/packages/aws/data_stream/cloudwatch_metrics/manifest.yml +++ b/packages/aws/data_stream/cloudwatch_metrics/manifest.yml @@ -37,6 +37,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: metrics type: yaml title: Metrics diff --git a/packages/aws/data_stream/dynamodb/manifest.yml b/packages/aws/data_stream/dynamodb/manifest.yml index d4ccb42311a..a7a42255248 100644 --- a/packages/aws/data_stream/dynamodb/manifest.yml +++ b/packages/aws/data_stream/dynamodb/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/ebs/manifest.yml b/packages/aws/data_stream/ebs/manifest.yml index 13a61103039..fbd538cbb2b 100644 --- a/packages/aws/data_stream/ebs/manifest.yml +++ b/packages/aws/data_stream/ebs/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/ec2_metrics/manifest.yml b/packages/aws/data_stream/ec2_metrics/manifest.yml index d712ff1332d..1e4ca21c1c3 100644 --- a/packages/aws/data_stream/ec2_metrics/manifest.yml +++ b/packages/aws/data_stream/ec2_metrics/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/ecs_metrics/manifest.yml b/packages/aws/data_stream/ecs_metrics/manifest.yml index 7b7901dc178..6f9d7abb130 100644 --- a/packages/aws/data_stream/ecs_metrics/manifest.yml +++ b/packages/aws/data_stream/ecs_metrics/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/elb_metrics/manifest.yml b/packages/aws/data_stream/elb_metrics/manifest.yml index 1ae9320a723..e7663476fb9 100644 --- a/packages/aws/data_stream/elb_metrics/manifest.yml +++ b/packages/aws/data_stream/elb_metrics/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/firewall_metrics/manifest.yml b/packages/aws/data_stream/firewall_metrics/manifest.yml index 9251d30578a..5f42d45c232 100644 --- a/packages/aws/data_stream/firewall_metrics/manifest.yml +++ b/packages/aws/data_stream/firewall_metrics/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/kinesis/manifest.yml b/packages/aws/data_stream/kinesis/manifest.yml index d399a90cf2f..66cd668eb8b 100644 --- a/packages/aws/data_stream/kinesis/manifest.yml +++ b/packages/aws/data_stream/kinesis/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/lambda/manifest.yml b/packages/aws/data_stream/lambda/manifest.yml index f21e65e615c..d95a9f33212 100644 --- a/packages/aws/data_stream/lambda/manifest.yml +++ b/packages/aws/data_stream/lambda/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/natgateway/manifest.yml b/packages/aws/data_stream/natgateway/manifest.yml index 8f799ecba9e..f0de4c9c433 100644 --- a/packages/aws/data_stream/natgateway/manifest.yml +++ b/packages/aws/data_stream/natgateway/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/rds/manifest.yml b/packages/aws/data_stream/rds/manifest.yml index b85e63d299c..17539f8739c 100644 --- a/packages/aws/data_stream/rds/manifest.yml +++ b/packages/aws/data_stream/rds/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/redshift/manifest.yml b/packages/aws/data_stream/redshift/manifest.yml index abf33113ab5..0fa3dc6692c 100644 --- a/packages/aws/data_stream/redshift/manifest.yml +++ b/packages/aws/data_stream/redshift/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/s3_daily_storage/manifest.yml b/packages/aws/data_stream/s3_daily_storage/manifest.yml index 5ce6af8adc8..4768bbf65ca 100644 --- a/packages/aws/data_stream/s3_daily_storage/manifest.yml +++ b/packages/aws/data_stream/s3_daily_storage/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/s3_request/manifest.yml b/packages/aws/data_stream/s3_request/manifest.yml index 07e5a3684c4..1070acd093d 100644 --- a/packages/aws/data_stream/s3_request/manifest.yml +++ b/packages/aws/data_stream/s3_request/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/s3_storage_lens/manifest.yml b/packages/aws/data_stream/s3_storage_lens/manifest.yml index 3f5a2f092ef..4c2f334d91f 100644 --- a/packages/aws/data_stream/s3_storage_lens/manifest.yml +++ b/packages/aws/data_stream/s3_storage_lens/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/sns/manifest.yml b/packages/aws/data_stream/sns/manifest.yml index 60d257f8064..bc9fc5e12cd 100644 --- a/packages/aws/data_stream/sns/manifest.yml +++ b/packages/aws/data_stream/sns/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/data_stream/sqs/manifest.yml b/packages/aws/data_stream/sqs/manifest.yml index 2619857c87c..663f235623b 100644 --- a/packages/aws/data_stream/sqs/manifest.yml +++ b/packages/aws/data_stream/sqs/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/transitgateway/manifest.yml b/packages/aws/data_stream/transitgateway/manifest.yml index 217b4de594e..f5fec6b5374 100644 --- a/packages/aws/data_stream/transitgateway/manifest.yml +++ b/packages/aws/data_stream/transitgateway/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/usage/manifest.yml b/packages/aws/data_stream/usage/manifest.yml index 9e0706cdb9b..ed871e37a93 100644 --- a/packages/aws/data_stream/usage/manifest.yml +++ b/packages/aws/data_stream/usage/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: include_linked_accounts type: bool title: Include Linked Accounts diff --git a/packages/aws/data_stream/vpn/manifest.yml b/packages/aws/data_stream/vpn/manifest.yml index 30f5e749601..fdc5d1e7ff7 100644 --- a/packages/aws/data_stream/vpn/manifest.yml +++ b/packages/aws/data_stream/vpn/manifest.yml @@ -28,6 +28,7 @@ streams: multi: false required: false show_user: false + description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`. - name: tags_filter type: yaml title: Tags Filter diff --git a/packages/aws/docs/README.md b/packages/aws/docs/README.md index 19e170ae084..859b4c827e3 100644 --- a/packages/aws/docs/README.md +++ b/packages/aws/docs/README.md @@ -17,7 +17,7 @@ This may include every user request that CloudFront receives, every action taken by an AWS user or role, and more. **Metrics** give you insight into the state of your AWS services. -his may include understanding where you're spending the most and why, the volume of storage you're using, +This may include understanding where you're spending the most and why, the volume of storage you're using, CPU utilization of your instances, and more. For a complete list of all AWS services and the data streams available for each, see [Reference](#reference). @@ -188,12 +188,26 @@ Use the AWS integration to connect to your AWS account and collect data from mul When you configure the integration, you can collect data from as many AWS services as you'd like. If you only need to collect data from one AWS service, consider using the individual integration -(for example, to only collect billing metrics, you can use the -**AWS CloudFront** integration). +(for example, to only collect monitoring metrics for EC2, you can configure only the **AWS EC2** integration). For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +## Debug +### Latency causes missing metrics +Some AWS services send monitoring metrics to CloudWatch with a latency to process larger than the integration collection +period. This will cause data points missing or none get collected by the agent. In this case, please specify a +latency parameter so collection start time and end time will be shifted by the given latency amount. + +In order to check how much the latency is, you can log into the AWS CloudWatch portal. Wait till a new point to +show up in AWS CloudWatch and record the current timestamp. Compare the timestamp of this latest data point with the +current timestamp to see what's the difference. This difference can be used as latency. + +For example, the screenshot below is taken at `2023-05-09 22:30 UTC` and the timestamp for the last data point is +`2023-05-09 22:15 UTC`. This means there is a 15min delay between the current time and CloudWatch. With this information, +we should add a `latency` configuration for `15m` when adding the integration. +![alt text](../img/metricbeat-aws-cloudwatch-latency.png "CloudWatch Last Data Point Timestamp") + ## Reference Below is an overview of the type of data you can collect from each AWS service. diff --git a/packages/aws/img/metricbeat-aws-cloudwatch-latency.png b/packages/aws/img/metricbeat-aws-cloudwatch-latency.png new file mode 100644 index 00000000000..65ce8897a07 Binary files /dev/null and b/packages/aws/img/metricbeat-aws-cloudwatch-latency.png differ diff --git a/packages/aws/manifest.yml b/packages/aws/manifest.yml index 740c9eac69a..2fe2ea375ee 100644 --- a/packages/aws/manifest.yml +++ b/packages/aws/manifest.yml @@ -1,7 +1,7 @@ format_version: 1.0.0 name: aws title: AWS -version: 1.43.1 +version: 1.43.2 license: basic description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent. type: integration