[Filebeat] Add module for AWS S3 server access log (elastic#13170)

* Initial commit for adding s3 server access log module
* Add basic dashboard for s3access fileset

CHANGELOG.next.asciidoc

@@ -258,6 +258,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update module pipelines to enrich events with autonomous system fields. {pull}13036[13036]
 - Add module for ingesting IBM MQ logs. {pull}8782[8782]
 - Add S3 input to retrieve logs from AWS S3 buckets. {pull}12640[12640] {issue}12582[12582]
+- Add aws module s3access metricset. {pull}13170[13170] {issue}12880[12880]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -49,6 +49,7 @@ grouped in the following categories:
 * <<exported-fields-process>>
 * <<exported-fields-rabbitmq>>
 * <<exported-fields-redis>>
+* <<exported-fields-s3>>
 * <<exported-fields-santa>>
 * <<exported-fields-suricata>>
 * <<exported-fields-system>>
@@ -988,33 +989,250 @@ alias to: destination.address
 --
 
 [[exported-fields-aws]]
-== aws fields
+== AWS fields
 
-AWS fields from s3 input.
+Module for handling logs from AWS.
 
 
 
 [float]
-=== s3
+=== aws
 
-S3 related information.
+Fields from AWS logs.
 
 
 
-*`s3.bucket_name`*::
+[float]
+=== s3access
+
+Fields for AWS S3 server access logs.
+
+
+
+*`aws.s3access.bucket_owner`*::
 +
 --
-Name of the S3 bucket that this log retrieved from.
+The canonical user ID of the owner of the source bucket.
 
 
 type: keyword
 
 --
 
-*`s3.object_key`*::
+*`aws.s3access.bucket`*::
 +
 --
-Name of the S3 object that this log retrieved from.
+The name of the bucket that the request was processed against.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.remote_ip`*::
++
+--
+The apparent internet address of the requester.
+
+
+type: ip
+
+--
+
+*`aws.s3access.requester`*::
++
+--
+The canonical user ID of the requester, or a - for unauthenticated requests.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.request_id`*::
++
+--
+A string generated by Amazon S3 to uniquely identify each request.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.operation`*::
++
+--
+The operation listed here is declared as SOAP.operation, REST.HTTP_method.resource_type, WEBSITE.HTTP_method.resource_type, or BATCH.DELETE.OBJECT.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.key`*::
++
+--
+The "key" part of the request, URL encoded, or "-" if the operation does not take a key parameter.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.request_uri`*::
++
+--
+The Request-URI part of the HTTP request message.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.http_status`*::
++
+--
+The numeric HTTP status code of the response.
+
+
+type: long
+
+--
+
+*`aws.s3access.error_code`*::
++
+--
+The Amazon S3 Error Code, or "-" if no error occurred.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.bytes_sent`*::
++
+--
+The number of response bytes sent, excluding HTTP protocol overhead, or "-" if zero.
+
+
+type: long
+
+--
+
+*`aws.s3access.object_size`*::
++
+--
+The total size of the object in question.
+
+
+type: long
+
+--
+
+*`aws.s3access.total_time`*::
++
+--
+The number of milliseconds the request was in flight from the server's perspective.
+
+
+type: long
+
+--
+
+*`aws.s3access.turn_around_time`*::
++
+--
+The number of milliseconds that Amazon S3 spent processing your request.
+
+
+type: long
+
+--
+
+*`aws.s3access.referrer`*::
++
+--
+The value of the HTTP Referrer header, if present.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.user_agent`*::
++
+--
+The value of the HTTP User-Agent header.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.version_id`*::
++
+--
+The version ID in the request, or "-" if the operation does not take a versionId parameter.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.host_id`*::
++
+--
+The x-amz-id-2 or Amazon S3 extended request ID.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.signature_version`*::
++
+--
+The signature version, SigV2 or SigV4, that was used to authenticate the request or a - for unauthenticated requests.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.cipher_suite`*::
++
+--
+The Secure Sockets Layer (SSL) cipher that was negotiated for HTTPS request or a - for HTTP.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.authentication_type`*::
++
+--
+The type of request authentication used, AuthHeader for authentication headers, QueryString for query string (pre-signed URL) or a - for unauthenticated requests.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.host_header`*::
++
+--
+The endpoint used to connect to Amazon S3.
+
+
+type: keyword
+
+--
+
+*`aws.s3access.tls_version`*::
++
+--
+The Transport Layer Security (TLS) version negotiated by the client.
 
 
 type: keyword
@@ -12943,6 +13161,33 @@ type: keyword
 The arguments with which the command was called.
 
 
+type: keyword
+
+--
+
+[[exported-fields-s3]]
+== s3 fields
+
+S3 fields from s3 input.
+
+
+
+*`bucket_name`*::
++
+--
+Name of the S3 bucket that this log retrieved from.
+
+
+type: keyword
+
+--
+
+*`object_key`*::
++
+--
+Name of the S3 object that this log retrieved from.
+
+
 type: keyword
 
 --

filebeat/docs/modules/aws.asciidoc

@@ -0,0 +1,56 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-aws]]
+[role="xpack"]
+
+:modulename: aws
+:has-dashboards: true
+
+== AWS module
+
+beta[]
+
+This is a module for aws logs. It uses filebeat s3 input to get log files from
+AWS S3 buckets with SQS notification. This module supports reading s3 server
+access logs with `s3access` fileset. Server access logging provides detailed
+records for the requests that are made to a bucket.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard for `s3access` fileset:
+
+[role="screenshot"]
+image::./images/filebeat-aws-s3access-overview.png[]
+
+[float]
+==== `s3access` fileset settings
+
+Example config:
+
+[source,yaml]
+----
+- module: aws
+  s3access:
+    enabled: true
+    var.queue_url: https://sqs.us-west-1.amazonaws.com/123/queue-name
+    var.credential_profile_name: fb-aws
+----
+
+*`var.queue_url`*::
+
+AWS SQS queue url.
+
+*`var.credential_profile_name`*::
+
+AWS credential profile name.
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-aws,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -5,6 +5,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-modules-overview>>
   * <<filebeat-module-apache>>
   * <<filebeat-module-auditd>>
+  * <<filebeat-module-aws>>
   * <<filebeat-module-cisco>>
   * <<filebeat-module-coredns>>
   * <<filebeat-module-elasticsearch>>
@@ -41,6 +42,7 @@ This file is generated! See scripts/docs_collector.py
 include::modules-overview.asciidoc[]
 include::modules/apache.asciidoc[]
 include::modules/auditd.asciidoc[]
+include::modules/aws.asciidoc[]
 include::modules/cisco.asciidoc[]
 include::modules/coredns.asciidoc[]
 include::modules/elasticsearch.asciidoc[]

x-pack/filebeat/filebeat.reference.yml

@@ -76,6 +76,17 @@ filebeat.modules:
     # can be added under this section.
     #input:
 
+#--------------------------------- AWS Module ---------------------------------
+- module: aws
+  s3access:
+    enabled: true
+
+    # AWS SQS queue url
+    #var.queue_url: sqs_queue_url
+
+    # Profile name for aws credential
+    #var.credential_profile_name: fb-aws
+
 #-------------------------------- Cisco Module --------------------------------
 - module: cisco
   asa: