From fe8795c145d590b773beb21daf7f36ea5ee869b1 Mon Sep 17 00:00:00 2001 From: Maurizio Branca Date: Wed, 12 Apr 2023 19:40:16 +0200 Subject: [PATCH] [AWS] Add `number_of_workers` and `latency` to all CloudWatch Logs based integrations (#5794) * Add number_of_workers and latency All the CloudWatch Logs based integration should have these options available to them. The `number_of_workers` is essential to increase the number of workers when users decide to use `log_group_name_prefix`. The `latency` can be required on the busiest log groups to deal with potential latency. * Add advanced options docs to all CloudWatch logs --- packages/aws/_dev/build/docs/cloudtrail.md | 18 +++++++++++++++++ packages/aws/_dev/build/docs/cloudwatch.md | 20 ++++++++++++++----- packages/aws/_dev/build/docs/ec2.md | 18 +++++++++++++++++ packages/aws/_dev/build/docs/elb.md | 20 ++++++++++++++++++- packages/aws/_dev/build/docs/firewall.md | 18 +++++++++++++++++ packages/aws/_dev/build/docs/route53.md | 18 +++++++++++++++++ packages/aws/_dev/build/docs/vpcflow.md | 18 +++++++++++++++++ packages/aws/_dev/build/docs/waf.md | 18 +++++++++++++++++ packages/aws/changelog.yml | 7 ++++++- .../agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ .../aws/data_stream/cloudtrail/manifest.yml | 13 ++++++++++++ .../agent/stream/aws-cloudwatch.yml.hbs | 3 +++ .../data_stream/cloudwatch_logs/manifest.yml | 6 ++++++ .../agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ .../aws/data_stream/ec2_logs/manifest.yml | 13 ++++++++++++ .../agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ .../aws/data_stream/elb_logs/manifest.yml | 13 ++++++++++++ .../agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ .../data_stream/firewall_logs/manifest.yml | 13 ++++++++++++ .../agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ .../route53_public_logs/manifest.yml | 13 ++++++++++++ .../agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ .../route53_resolver_logs/manifest.yml | 13 ++++++++++++ .../agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ packages/aws/data_stream/vpcflow/manifest.yml | 13 ++++++++++++ .../waf/agent/stream/aws-cloudwatch.yml.hbs | 7 +++++++ packages/aws/data_stream/waf/manifest.yml | 13 ++++++++++++ packages/aws/docs/cloudtrail.md | 18 +++++++++++++++++ packages/aws/docs/cloudwatch.md | 19 ++++++++++++++---- packages/aws/docs/ec2.md | 18 +++++++++++++++++ packages/aws/docs/elb.md | 19 ++++++++++++++++++ packages/aws/docs/firewall.md | 18 +++++++++++++++++ packages/aws/docs/route53.md | 18 +++++++++++++++++ packages/aws/docs/vpcflow.md | 18 +++++++++++++++++ packages/aws/docs/waf.md | 18 +++++++++++++++++ packages/aws/manifest.yml | 2 +- .../agent/stream/aws-cloudwatch.yml.hbs | 3 +++ .../aws_logs/data_stream/generic/manifest.yml | 8 +++++++- 38 files changed, 470 insertions(+), 13 deletions(-) diff --git a/packages/aws/_dev/build/docs/cloudtrail.md b/packages/aws/_dev/build/docs/cloudtrail.md index 8b1d6bc9e91..c0d82eeaeac 100644 --- a/packages/aws/_dev/build/docs/cloudtrail.md +++ b/packages/aws/_dev/build/docs/cloudtrail.md @@ -40,6 +40,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `cloudtrail` data stream collects AWS CloudTrail logs. CloudTrail monitors events like diff --git a/packages/aws/_dev/build/docs/cloudwatch.md b/packages/aws/_dev/build/docs/cloudwatch.md index 9f074e793fd..0b208f94f4f 100644 --- a/packages/aws/_dev/build/docs/cloudwatch.md +++ b/packages/aws/_dev/build/docs/cloudwatch.md @@ -40,13 +40,23 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide. -### Advanced +### Advanced options -#### Latency +#### CloudWatch -Log events on the busies log groups may require a longer time before they are available to CloudWatch Logs. +The CloudWatch logs input has several advanced options to fit specific use cases. -The CloudWatch integration offers the `latency` setting to cope with this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. ## Logs reference @@ -61,4 +71,4 @@ CloudWatch logs to monitor, store, and access log files from different sources. {{event "cloudwatch_metrics"}} -{{fields "cloudwatch_metrics"}} \ No newline at end of file +{{fields "cloudwatch_metrics"}} diff --git a/packages/aws/_dev/build/docs/ec2.md b/packages/aws/_dev/build/docs/ec2.md index c479a0fceb4..e20c7183b85 100644 --- a/packages/aws/_dev/build/docs/ec2.md +++ b/packages/aws/_dev/build/docs/ec2.md @@ -40,6 +40,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `ec2` data stream supports both EC2 logs stored in AWS CloudWatch and EC2 logs stored in Amazon S3. diff --git a/packages/aws/_dev/build/docs/elb.md b/packages/aws/_dev/build/docs/elb.md index c5b0dfba9bb..9247db52ddf 100644 --- a/packages/aws/_dev/build/docs/elb.md +++ b/packages/aws/_dev/build/docs/elb.md @@ -47,6 +47,24 @@ For an application load balancer, see [enable access log for application load ba For a network load balancer, see [enable access log for network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html). +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `elb` dataset collects logs from AWS ELBs. @@ -59,4 +77,4 @@ The `elb` dataset collects logs from AWS ELBs. {{event "elb_metrics"}} -{{fields "elb_metrics"}} \ No newline at end of file +{{fields "elb_metrics"}} diff --git a/packages/aws/_dev/build/docs/firewall.md b/packages/aws/_dev/build/docs/firewall.md index 97fdebc10ad..8dd7803a16e 100644 --- a/packages/aws/_dev/build/docs/firewall.md +++ b/packages/aws/_dev/build/docs/firewall.md @@ -40,6 +40,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `firewall_logs` dataset collects AWS Network Firewall logs. Users can use these logs to diff --git a/packages/aws/_dev/build/docs/route53.md b/packages/aws/_dev/build/docs/route53.md index 2102188251e..82c9dd86cb6 100644 --- a/packages/aws/_dev/build/docs/route53.md +++ b/packages/aws/_dev/build/docs/route53.md @@ -37,6 +37,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference ### Public Hosted Zone logs diff --git a/packages/aws/_dev/build/docs/vpcflow.md b/packages/aws/_dev/build/docs/vpcflow.md index f5ff68a896f..f27dcd3df26 100644 --- a/packages/aws/_dev/build/docs/vpcflow.md +++ b/packages/aws/_dev/build/docs/vpcflow.md @@ -58,6 +58,24 @@ This integration supports various plain text VPC flow log formats: ${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr} ${region} ${az-id} ${sublocation-type} ${sublocation-id} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction} ${traffic-path} ``` +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference > Note: The Parquet format is not supported. diff --git a/packages/aws/_dev/build/docs/waf.md b/packages/aws/_dev/build/docs/waf.md index fb02f4e2eb2..0956f3c5fe1 100644 --- a/packages/aws/_dev/build/docs/waf.md +++ b/packages/aws/_dev/build/docs/waf.md @@ -41,6 +41,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `waf` dataset is specifically for WAF logs. Export logs from Kinesis Data Firehose to Amazon S3 bucket which has SQS notification setup already. diff --git a/packages/aws/changelog.yml b/packages/aws/changelog.yml index 419ab35c3d1..4439c7b166c 100644 --- a/packages/aws/changelog.yml +++ b/packages/aws/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "1.33.3" + changes: + - description: Add number_of_workers and latency to all CloudWatch Logs based integrations. + type: enhancement + link: https://github.com/elastic/integrations/pull/5794 - version: "1.33.2" changes: - description: Add missing permissions in the AWS Billing integration documentation. @@ -12,7 +17,7 @@ - version: "1.33.0" changes: - description: Add latency configuration option on the CloudWatch Logs integration. - type: bugfix + type: enhancement link: https://github.com/elastic/integrations/pull/5777 - version: "1.32.2" changes: diff --git a/packages/aws/data_stream/cloudtrail/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/cloudtrail/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/cloudtrail/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/cloudtrail/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/cloudtrail/manifest.yml b/packages/aws/data_stream/cloudtrail/manifest.yml index 6e46e9bd5ff..a73797e3621 100644 --- a/packages/aws/data_stream/cloudtrail/manifest.yml +++ b/packages/aws/data_stream/cloudtrail/manifest.yml @@ -273,6 +273,19 @@ streams: show_user: false default: 200ms description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account. + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/cloudwatch_logs/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/cloudwatch_logs/agent/stream/aws-cloudwatch.yml.hbs index cef7b419126..9588557d158 100644 --- a/packages/aws/data_stream/cloudwatch_logs/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/cloudwatch_logs/agent/stream/aws-cloudwatch.yml.hbs @@ -56,6 +56,9 @@ api_sleep: {{ api_sleep }} {{#if latency }} latency: {{ latency }} {{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} diff --git a/packages/aws/data_stream/cloudwatch_logs/manifest.yml b/packages/aws/data_stream/cloudwatch_logs/manifest.yml index a5fac6073f3..04bd2936e3a 100644 --- a/packages/aws/data_stream/cloudwatch_logs/manifest.yml +++ b/packages/aws/data_stream/cloudwatch_logs/manifest.yml @@ -156,6 +156,12 @@ streams: required: false show_user: false description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching the `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/ec2_logs/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/ec2_logs/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/ec2_logs/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/ec2_logs/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/ec2_logs/manifest.yml b/packages/aws/data_stream/ec2_logs/manifest.yml index cac4ec31247..8f1bb76d097 100644 --- a/packages/aws/data_stream/ec2_logs/manifest.yml +++ b/packages/aws/data_stream/ec2_logs/manifest.yml @@ -148,6 +148,19 @@ streams: show_user: false default: 200ms description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account. + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/elb_logs/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/elb_logs/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/elb_logs/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/elb_logs/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/elb_logs/manifest.yml b/packages/aws/data_stream/elb_logs/manifest.yml index 44a57a93da2..17dbabac05e 100644 --- a/packages/aws/data_stream/elb_logs/manifest.yml +++ b/packages/aws/data_stream/elb_logs/manifest.yml @@ -148,6 +148,19 @@ streams: show_user: false default: 200ms description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account. + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/firewall_logs/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/firewall_logs/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/firewall_logs/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/firewall_logs/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/firewall_logs/manifest.yml b/packages/aws/data_stream/firewall_logs/manifest.yml index f55a83e7a65..8c1334a94b4 100644 --- a/packages/aws/data_stream/firewall_logs/manifest.yml +++ b/packages/aws/data_stream/firewall_logs/manifest.yml @@ -148,6 +148,19 @@ streams: show_user: false default: 200ms description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account. + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/route53_public_logs/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/route53_public_logs/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/route53_public_logs/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/route53_public_logs/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/route53_public_logs/manifest.yml b/packages/aws/data_stream/route53_public_logs/manifest.yml index eeca62fe833..84bdcfc83b2 100644 --- a/packages/aws/data_stream/route53_public_logs/manifest.yml +++ b/packages/aws/data_stream/route53_public_logs/manifest.yml @@ -78,6 +78,19 @@ streams: multi: false show_user: false required: false + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/route53_resolver_logs/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/route53_resolver_logs/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/route53_resolver_logs/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/route53_resolver_logs/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/route53_resolver_logs/manifest.yml b/packages/aws/data_stream/route53_resolver_logs/manifest.yml index f55fedda446..294a6b204fd 100644 --- a/packages/aws/data_stream/route53_resolver_logs/manifest.yml +++ b/packages/aws/data_stream/route53_resolver_logs/manifest.yml @@ -78,6 +78,19 @@ streams: multi: false show_user: false required: false + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/vpcflow/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/vpcflow/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/vpcflow/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/vpcflow/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/vpcflow/manifest.yml b/packages/aws/data_stream/vpcflow/manifest.yml index d0fa62804bd..73a6daa68bd 100644 --- a/packages/aws/data_stream/vpcflow/manifest.yml +++ b/packages/aws/data_stream/vpcflow/manifest.yml @@ -148,6 +148,19 @@ streams: show_user: false default: 200ms description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account. + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/data_stream/waf/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws/data_stream/waf/agent/stream/aws-cloudwatch.yml.hbs index 9d9f44a8ddd..934f97a44c3 100644 --- a/packages/aws/data_stream/waf/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws/data_stream/waf/agent/stream/aws-cloudwatch.yml.hbs @@ -50,6 +50,13 @@ scan_frequency: {{ scan_frequency }} api_sleep: {{ api_sleep }} {{/if}} +{{#if latency }} +latency: {{ latency }} +{{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} + {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} {{/if}} diff --git a/packages/aws/data_stream/waf/manifest.yml b/packages/aws/data_stream/waf/manifest.yml index c8188ded6bc..892bfb078a8 100644 --- a/packages/aws/data_stream/waf/manifest.yml +++ b/packages/aws/data_stream/waf/manifest.yml @@ -148,6 +148,19 @@ streams: show_user: false default: 200ms description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account. + - name: latency + type: text + title: Latency + multi: false + required: false + show_user: false + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags diff --git a/packages/aws/docs/cloudtrail.md b/packages/aws/docs/cloudtrail.md index 0c9fd22d1cf..ea7dceb9762 100644 --- a/packages/aws/docs/cloudtrail.md +++ b/packages/aws/docs/cloudtrail.md @@ -40,6 +40,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `cloudtrail` data stream collects AWS CloudTrail logs. CloudTrail monitors events like diff --git a/packages/aws/docs/cloudwatch.md b/packages/aws/docs/cloudwatch.md index a5567d3c599..99061902bb3 100644 --- a/packages/aws/docs/cloudwatch.md +++ b/packages/aws/docs/cloudwatch.md @@ -40,13 +40,23 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. -### Advanced +### Advanced options -#### Latency +#### CloudWatch -Log events on the busies log groups may require a longer time before they are available to CloudWatch Logs. +The CloudWatch logs input has several advanced options to fit specific use cases. -The CloudWatch integration offers the `latency` setting to cope with this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. ## Logs reference @@ -240,3 +250,4 @@ An example event for `cloudwatch` looks as following: | host.os.version | Operating system version as a raw string. | keyword | | host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | + diff --git a/packages/aws/docs/ec2.md b/packages/aws/docs/ec2.md index fcb0479fb42..db7a8825323 100644 --- a/packages/aws/docs/ec2.md +++ b/packages/aws/docs/ec2.md @@ -40,6 +40,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `ec2` data stream supports both EC2 logs stored in AWS CloudWatch and EC2 logs stored in Amazon S3. diff --git a/packages/aws/docs/elb.md b/packages/aws/docs/elb.md index f5a1fd28bbd..6a23c645af5 100644 --- a/packages/aws/docs/elb.md +++ b/packages/aws/docs/elb.md @@ -47,6 +47,24 @@ For an application load balancer, see [enable access log for application load ba For a network load balancer, see [enable access log for network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html). +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `elb` dataset collects logs from AWS ELBs. @@ -472,3 +490,4 @@ An example event for `elb` looks as following: | host.os.version | Operating system version as a raw string. | keyword | | host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | + diff --git a/packages/aws/docs/firewall.md b/packages/aws/docs/firewall.md index 01383fe3c6b..b76d0c62acc 100644 --- a/packages/aws/docs/firewall.md +++ b/packages/aws/docs/firewall.md @@ -40,6 +40,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `firewall_logs` dataset collects AWS Network Firewall logs. Users can use these logs to diff --git a/packages/aws/docs/route53.md b/packages/aws/docs/route53.md index aa3036de7b6..b40dacc6900 100644 --- a/packages/aws/docs/route53.md +++ b/packages/aws/docs/route53.md @@ -37,6 +37,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference ### Public Hosted Zone logs diff --git a/packages/aws/docs/vpcflow.md b/packages/aws/docs/vpcflow.md index 07984db0507..feb31adfe8e 100644 --- a/packages/aws/docs/vpcflow.md +++ b/packages/aws/docs/vpcflow.md @@ -58,6 +58,24 @@ This integration supports various plain text VPC flow log formats: ${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr} ${region} ${az-id} ${sublocation-type} ${sublocation-id} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction} ${traffic-path} ``` +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference > Note: The Parquet format is not supported. diff --git a/packages/aws/docs/waf.md b/packages/aws/docs/waf.md index 50e38143da6..0b01caef0c0 100644 --- a/packages/aws/docs/waf.md +++ b/packages/aws/docs/waf.md @@ -41,6 +41,24 @@ When you configure the AWS integration, you can collect data from as many AWS se For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +### Advanced options + +#### CloudWatch + +The CloudWatch logs input has several advanced options to fit specific use cases. + +##### Latency + +AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent. + +The CloudWatch integration offers the `latency` setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago. + +##### Number of workers + +If you are collecting log events from multiple log groups using `log_group_name_prefix`, you should review the value of the `number_of_workers`. + +The `number_of_workers` setting defines the number of workers assigned to reading from log groups. Each log group matching the `log_group_name_prefix` requires a worker to keep log ingestion as close to real-time as possible. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. + ## Logs reference The `waf` dataset is specifically for WAF logs. Export logs from Kinesis Data Firehose to Amazon S3 bucket which has SQS notification setup already. diff --git a/packages/aws/manifest.yml b/packages/aws/manifest.yml index bdd83696985..6acdd7263f6 100644 --- a/packages/aws/manifest.yml +++ b/packages/aws/manifest.yml @@ -1,7 +1,7 @@ format_version: 1.0.0 name: aws title: AWS -version: 1.33.2 +version: 1.33.3 license: basic description: Collect logs and metrics from Amazon Web Services with Elastic Agent. type: integration diff --git a/packages/aws_logs/data_stream/generic/agent/stream/aws-cloudwatch.yml.hbs b/packages/aws_logs/data_stream/generic/agent/stream/aws-cloudwatch.yml.hbs index db2cf4f64ae..c141d81c552 100644 --- a/packages/aws_logs/data_stream/generic/agent/stream/aws-cloudwatch.yml.hbs +++ b/packages/aws_logs/data_stream/generic/agent/stream/aws-cloudwatch.yml.hbs @@ -62,6 +62,9 @@ api_sleep: {{ api_sleep }} {{#if latency }} latency: {{ latency }} {{/if}} +{{#if number_of_workers }} +number_of_workers: {{ number_of_workers }} +{{/if}} {{#if credential_profile_name}} credential_profile_name: {{credential_profile_name}} diff --git a/packages/aws_logs/data_stream/generic/manifest.yml b/packages/aws_logs/data_stream/generic/manifest.yml index b3f2efe0a7a..78248648f0f 100644 --- a/packages/aws_logs/data_stream/generic/manifest.yml +++ b/packages/aws_logs/data_stream/generic/manifest.yml @@ -94,7 +94,13 @@ streams: multi: false required: false show_user: false - description: Some AWS services send logs to CloudWatch with a latency to process larger than aws-cloudwatch input scan_frequency. This case, please use this latency parameter so collection start time and end time will be shifted by the given latency amount. + description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago." + - name: number_of_workers + type: integer + title: Number of workers + required: false + show_user: false + description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`. - name: tags type: text title: Tags