Skip to content

Commit

Permalink
[AWS] add latency documentation (#6231)
Browse files Browse the repository at this point in the history
  • Loading branch information
kaiyan-sheng authored Jun 12, 2023
1 parent f568024 commit 8c68f89
Show file tree
Hide file tree
Showing 25 changed files with 60 additions and 7 deletions.
20 changes: 17 additions & 3 deletions packages/aws/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This may include every user request that CloudFront receives, every action taken
by an AWS user or role, and more.

**Metrics** give you insight into the state of your AWS services.
his may include understanding where you're spending the most and why, the volume of storage you're using,
This may include understanding where you're spending the most and why, the volume of storage you're using,
CPU utilization of your instances, and more.

For a complete list of all AWS services and the data streams available for each, see [Reference](#reference).
Expand Down Expand Up @@ -188,12 +188,26 @@ Use the AWS integration to connect to your AWS account and collect data from mul
When you configure the integration, you can collect data from as many AWS services as you'd like.

If you only need to collect data from one AWS service, consider using the individual integration
(for example, to only collect billing metrics, you can use the
**AWS CloudFront** integration).
(for example, to only collect monitoring metrics for EC2, you can configure only the **AWS EC2** integration).

For step-by-step instructions on how to set up an integration, see the
{{ url "getting-started-observability" "Getting started" }} guide.

## Debug
### Latency causes missing metrics
Some AWS services send monitoring metrics to CloudWatch with a latency to process larger than the integration collection
period. This will cause data points missing or none get collected by the agent. In this case, please specify a
latency parameter so collection start time and end time will be shifted by the given latency amount.

In order to check how much the latency is, you can log into the AWS CloudWatch portal. Wait till a new point to
show up in AWS CloudWatch and record the current timestamp. Compare the timestamp of this latest data point with the
current timestamp to see what's the difference. This difference can be used as latency.

For example, the screenshot below is taken at `2023-05-09 22:30 UTC` and the timestamp for the last data point is
`2023-05-09 22:15 UTC`. This means there is a 15min delay between the current time and CloudWatch. With this information,
we should add a `latency` configuration for `15m` when adding the integration.
![alt text](../img/metricbeat-aws-cloudwatch-latency.png "CloudWatch Last Data Point Timestamp")

## Reference

Below is an overview of the type of data you can collect from each AWS service.
Expand Down
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.43.2"
changes:
- description: Add documentation for latency parameter
type: enhancement
link: https://github.com/elastic/integrations/pull/6231
- version: "1.43.1"
changes:
- description: Add tags_filter and include_linked_accounts config parameter in missing metric data streams.
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/cloudwatch_metrics/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: metrics
type: yaml
title: Metrics
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/dynamodb/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/ebs/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/ec2_metrics/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/ecs_metrics/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/elb_metrics/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/firewall_metrics/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/kinesis/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/lambda/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/natgateway/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/rds/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/redshift/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/s3_daily_storage/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/s3_request/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/s3_storage_lens/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/sns/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/sqs/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/transitgateway/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/usage/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: include_linked_accounts
type: bool
title: Include Linked Accounts
Expand Down
1 change: 1 addition & 0 deletions packages/aws/data_stream/vpn/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ streams:
multi: false
required: false
show_user: false
description: To address latency issues between certain AWS services and CloudWatch, specify a latency parameter to adjust the collection start time and end time in Metricbeat such as `15m`.
- name: tags_filter
type: yaml
title: Tags Filter
Expand Down
20 changes: 17 additions & 3 deletions packages/aws/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This may include every user request that CloudFront receives, every action taken
by an AWS user or role, and more.

**Metrics** give you insight into the state of your AWS services.
his may include understanding where you're spending the most and why, the volume of storage you're using,
This may include understanding where you're spending the most and why, the volume of storage you're using,
CPU utilization of your instances, and more.

For a complete list of all AWS services and the data streams available for each, see [Reference](#reference).
Expand Down Expand Up @@ -188,12 +188,26 @@ Use the AWS integration to connect to your AWS account and collect data from mul
When you configure the integration, you can collect data from as many AWS services as you'd like.

If you only need to collect data from one AWS service, consider using the individual integration
(for example, to only collect billing metrics, you can use the
**AWS CloudFront** integration).
(for example, to only collect monitoring metrics for EC2, you can configure only the **AWS EC2** integration).

For step-by-step instructions on how to set up an integration, see the
[Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.

## Debug
### Latency causes missing metrics
Some AWS services send monitoring metrics to CloudWatch with a latency to process larger than the integration collection
period. This will cause data points missing or none get collected by the agent. In this case, please specify a
latency parameter so collection start time and end time will be shifted by the given latency amount.

In order to check how much the latency is, you can log into the AWS CloudWatch portal. Wait till a new point to
show up in AWS CloudWatch and record the current timestamp. Compare the timestamp of this latest data point with the
current timestamp to see what's the difference. This difference can be used as latency.

For example, the screenshot below is taken at `2023-05-09 22:30 UTC` and the timestamp for the last data point is
`2023-05-09 22:15 UTC`. This means there is a 15min delay between the current time and CloudWatch. With this information,
we should add a `latency` configuration for `15m` when adding the integration.
![alt text](../img/metricbeat-aws-cloudwatch-latency.png "CloudWatch Last Data Point Timestamp")

## Reference

Below is an overview of the type of data you can collect from each AWS service.
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion packages/aws/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 1.0.0
name: aws
title: AWS
version: 1.43.1
version: 1.43.2
license: basic
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
type: integration
Expand Down

0 comments on commit 8c68f89

Please sign in to comment.