Add awsfargate module to collect container logs from Amazon ECS on Fa…

…rgate (#25041)

CHANGELOG.next.asciidoc

@@ -839,6 +839,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add `fail_on_template_error` option for httpjson input. {pull}24784[24784]
 - Change `okta.target` to `flattened` field type.  {issue}24354[24354] {pull}24636[24636]
 - Added `http.request.id` to `nginx/ingress_controller` and `elasticsearch/audit`. {pull}24994[24994]
+- Add `awsfargate` module to collect container logs from Amazon ECS on Fargate. {pull}25041[25041]
 - New module `cyberarkpas` for CyberArk Privileged Access Security audit logs. {pull}24803[24803]
 
 *Heartbeat*

filebeat/docs/fields.asciidoc

@@ -17,6 +17,7 @@ grouped in the following categories:
 * <<exported-fields-auditd>>
 * <<exported-fields-aws>>
 * <<exported-fields-aws-cloudwatch>>
+* <<exported-fields-awsfargate>>
 * <<exported-fields-azure>>
 * <<exported-fields-barracuda>>
 * <<exported-fields-beat-common>>
@@ -2318,6 +2319,26 @@ type: keyword
 
 --
 
+[[exported-fields-awsfargate]]
+== AWS Fargate fields
+
+Module for collecting container logs from Amazon ECS Fargate.
+
+
+
+[float]
+=== awsfargate
+
+Fields from Amazon ECS Fargate logs.
+
+
+
+[float]
+=== log
+
+Fields for Amazon Fargate container logs.
+
+
 [[exported-fields-azure]]
 == Azure fields
 

filebeat/docs/modules/awsfargate.asciidoc

@@ -0,0 +1,139 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-awsfargate]]
+[role="xpack"]
+
+:libbeat-xpack-dir: ../../../x-pack/libbeat
+
+:modulename: awsfargate
+:has-dashboards: false
+
+== AWS Fargate module
+
+beta[]
+
+This module can be used to collect container logs from Amazon ECS on Fargate.
+It uses filebeat `awscloudwatch` input to get log files from one or more log
+streams in AWS CloudWatch. Logs from all containers in Fargate launch type tasks
+can be sent to CloudWatch by adding the `awslogs` log driver under `logConfiguration`
+section in the task definition. For example, `logConfiguration` can be added into
+the task definition by adding this section into the `containerDefinitions`:
+
+[source,json]
+----
+{
+   "logDriver":"awslogs",
+   "options":{
+      "awslogs-group":"awslogs-wordpress",
+      "awslogs-region":"us-west-2",
+      "awslogs-stream-prefix":"awslogs-example"
+   }
+}
+----
+
+The `awsfargate` module requires AWS credentials configuration in order to make AWS API calls.
+Users can either use `access_key_id`, `secret_access_key` and/or
+`session_token`, or use `role_arn` AWS IAM role, or use shared AWS credentials file.
+
+Please see <<awsfargate-credentials,AWS credentials options>> for more details.
+
+[float]
+=== Module configuration
+
+Example config:
+
+[source,yaml]
+----
+- module: aws
+  fargate:
+    enabled: true
+    var.credential_profile_name: test-filebeat
+    var.log_group_arn: arn:aws:logs:us-east-1:1234567890:log-group:/ecs/test-log-group:*
+----
+
+*`var.log_group_arn`*::
+
+ARN of the log group to collect logs from.
+
+*`var.log_group_name`*::
+
+Name of the log group to collect logs from. Note: region_name is required when
+log_group_name is given.
+
+*`var.region_name`*::
+
+Region that the specified log group belongs to.
+
+*`var.log_streams`*::
+
+A list of strings of log streams names that Filebeat collect log events from.
+
+*`var.log_stream_prefix`*::
+
+A string to filter the results to include only log events from log streams
+that have names starting with this prefix.
+
+*`var.start_position`*::
+
+`start_position` allows user to specify if this input should read log files from
+the `beginning` or from the `end`.
+
+* `beginning`: reads from the beginning of the log group (default).
+* `end`: read only new messages from current time minus `scan_frequency` going forward
+
+*`var.scan_frequency`*::
+
+This config parameter sets how often Filebeat checks for new log events from the
+specified log group. Default `scan_frequency` is 1 minute, which means Filebeat
+will sleep for 1 minute before querying for new logs again.
+
+*`var.api_timeout`*::
+
+The maximum duration of AWS API can take. If it exceeds the timeout, AWS API
+will be interrupted. The default AWS API timeout for a message is 120 seconds.
+The minimum is 0 seconds.
+
+*`var.api_sleep`*::
+
+This is used to sleep between AWS `FilterLogEvents` API calls inside the same
+collection period. `FilterLogEvents` API has a quota of 5 transactions per
+second (TPS)/account/Region. By default, `api_sleep` is 200 ms. This value should
+only be adjusted when there are multiple Filebeats or multiple Filebeat inputs
+collecting logs from the same region and AWS account.
+
+*`var.shared_credential_file`*::
+
+Filename of AWS credential file.
+
+*`var.credential_profile_name`*::
+
+AWS credential profile name.
+
+*`var.access_key_id`*::
+First part of access key.
+
+*`var.secret_access_key`*::
+Second part of access key.
+
+*`var.session_token`*::
+Required when using temporary security credentials.
+
+*`var.role_arn`*::
+AWS IAM Role to assume.
+
+*`var.endpoint`*::
+
+The custom endpoint used to access AWS APIs.
+
+[id="awsfargate-credentials"]
+include::{libbeat-xpack-dir}/docs/aws-credentials-config.asciidoc[]
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-awsfargate,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -7,6 +7,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-apache>>
   * <<filebeat-module-auditd>>
   * <<filebeat-module-aws>>
+  * <<filebeat-module-awsfargate>>
   * <<filebeat-module-azure>>
   * <<filebeat-module-barracuda>>
   * <<filebeat-module-bluecoat>>
@@ -80,6 +81,7 @@ include::modules/activemq.asciidoc[]
 include::modules/apache.asciidoc[]
 include::modules/auditd.asciidoc[]
 include::modules/aws.asciidoc[]
+include::modules/awsfargate.asciidoc[]
 include::modules/azure.asciidoc[]
 include::modules/barracuda.asciidoc[]
 include::modules/bluecoat.asciidoc[]

filebeat/tests/system/test_modules.py

@@ -274,7 +274,8 @@ def clean_keys(obj):
         "threatintel.abuseurl",
         "threatintel.abusemalware",
         "threatintel.anomali",
-        "snyk.vulnerabilities"
+        "snyk.vulnerabilities",
+        "awsfargate.log"
     }
     # dataset + log file pairs for which @timestamp is kept as an exception from above
     remove_timestamp_exception = {

x-pack/filebeat/docs/inputs/input-aws-cloudwatch.asciidoc

@@ -43,10 +43,12 @@ The `aws-cloudwatch` input supports the following configuration options plus the
 ==== `log_group_arn`
 ARN of the log group to collect logs from.
 
+[float]
 ==== `log_group_name`
 Name of the log group to collect logs from. Note: region_name is required when
 log_group_name is given.
 
+[float]
 ==== `region_name`
 Region that the specified log group belongs to.
 

x-pack/filebeat/filebeat.reference.yml

@@ -348,6 +348,121 @@ filebeat.modules:
     # The maximum number of messages to return from SQS. Valid values: 1 to 10.
     #var.max_number_of_messages: 5
 
+- module: aws
+  fargate:
+    enabled: false
+
+    # Filename of AWS credential file
+    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
+    # "%UserProfile%\.aws\credentials" is used on Windows
+    #var.shared_credential_file: /etc/filebeat/aws_credentials
+
+    # Profile name for aws credential
+    # If not set the default profile is used
+    #var.credential_profile_name: fb-aws
+
+    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
+    #var.access_key_id: access_key_id
+    #var.secret_access_key: secret_access_key
+    #var.session_token: session_token
+
+    # The duration that the received messages are hidden from ReceiveMessage request
+    # Default to be 300s
+    #var.visibility_timeout: 300s
+
+    # Maximum duration before AWS API request will be interrupted
+    # Default to be 120s
+    #var.api_timeout: 120s
+
+    # Custom endpoint used to access AWS APIs
+    #var.endpoint: amazonaws.com
+
+    # AWS IAM Role to assume
+    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
+
+    # ARN of the log group to collect logs from
+    #var.log_group_arn: arn:aws:logs:us-east-1:123456789012:log-group:test:*
+
+    # Name of the log group to collect logs from
+    #var.log_group_name: /ecs/test-log-group
+
+    # Region that the specified log group belongs to
+    #var.region_name: us-east-1
+
+    # A list of strings of log streams names that Filebeat collect log events from
+    #var.log_streams: ["/ecs/test-log-group/test-log-stream"]
+
+    # A string to filter the results to include only log events from log streams that have names starting with this prefix
+    #var.log_stream_prefix: /ecs/test-log-group/
+
+    # Specify if Filebeat should read log files from the beginning or from the end
+    # Default start_position is beginning
+    #var.start_position: beginning
+
+    # How often Filebeat checks for new log events from the specified log group
+    # Default scan_frequency is 1 minute
+    #var.scan_frequency: 1m
+
+    # Time used to sleep between AWS FilterLogEvents API calls inside the same collection period
+    # Default api_sleep is 200 ms
+    #var.api_sleep: 200ms
+
+#----------------------------- AWS Fargate Module -----------------------------
+- module: awsfargate
+  log:
+    enabled: false
+
+    # Filename of AWS credential file
+    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
+    # "%UserProfile%\.aws\credentials" is used on Windows
+    #var.shared_credential_file: /etc/filebeat/aws_credentials
+
+    # Profile name for aws credential
+    # If not set the default profile is used
+    #var.credential_profile_name: fb-aws
+
+    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
+    #var.access_key_id: access_key_id
+    #var.secret_access_key: secret_access_key
+    #var.session_token: session_token
+
+    # Maximum duration before AWS API request will be interrupted
+    # Default to be 120s
+    #var.api_timeout: 120s
+
+    # Custom endpoint used to access AWS APIs
+    #var.endpoint: amazonaws.com
+
+    # AWS IAM Role to assume
+    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
+
+    # ARN of the log group to collect logs from
+    #var.log_group_arn: arn:aws:logs:us-east-1:123456789012:log-group:test:*
+
+    # Name of the log group to collect logs from
+    #var.log_group_name: /ecs/test-log-group
+
+    # Region that the specified log group belongs to
+    #var.region_name: us-east-1
+
+    # A list of strings of log streams names that Filebeat collect log events from
+    #var.log_streams: ["/ecs/test-log-group/test-log-stream"]
+
+    # A string to filter the results to include only log events from log streams that have names starting with this prefix
+    #var.log_stream_prefix: /ecs/test-log-group/
+
+    # Specify if Filebeat should read log files from the beginning or from the end
+    # Default start_position is beginning
+    #var.start_position: beginning
+
+    # How often Filebeat checks for new log events from the specified log group
+    # Default scan_frequency is 1 minute
+    #var.scan_frequency: 1m
+
+    # Time used to sleep between AWS FilterLogEvents API calls inside the same collection period
+    # Default api_sleep is 200 ms
+    #var.api_sleep: 200ms
+
 #-------------------------------- Azure Module --------------------------------
 - module: azure
   # All logs