sophosxg-module initial release (elastic#19591)

What does this PR do?

This PR Introduces the SophosXG filebeat module. Focusing currently on SophosXG Firewall, but should include other Sophos products as separate PR's later on.
Why is it important?

Adding more supported products to the filebeat portfolio.

Co-authored-by: StefanSa <[email protected]>
Co-authored-by: P1llus <[email protected]>

filebeat/docs/modules/sophosxg.asciidoc

@@ -0,0 +1,159 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-sophosxg]]
+[role="xpack"]
+
+:modulename: sophosxg
+:has-dashboards: false
+
+== SophosXG module
+
+This is a module for SophosXG SFOS logs sent in the syslog format.
+
+To configure a remote syslog destination, please reference the https://community.sophos.com/kb/en-us/123184[SophosXG/SFOS Documentation].
+
+The syslog format choosen should be `Default`.
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+This module has been tested against SFOS version 17.5.x and 18.0.x. 
+Versions above this are expected to work but have not been tested.
+
+include::../include/configuring-intro.asciidoc[]
+
+:fileset_ex: firewall
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `firewall` fileset settings
+
+[source,yaml]
+----
+- module: sophosxg
+  firewall:
+    enabled: true
+    var.input: udp
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: 9005
+    var.host_name: firewall.localgroup.local
+----
+
+include::../include/var-paths.asciidoc[]
+
+*`var.input`*::
+
+The input to use, can be either the value `tcp`, `udp` or `file`.
+
+*`var.syslog_host`*::
+
+The interface to listen to all syslog traffic. Defaults to localhost.
+Set to 0.0.0.0 to bind to all available interfaces.
+
+*`var.syslog_port`*::
+
+The port to listen for syslog traffic. Defaults to 9005.
+
+*`var.host_name`*::
+
+Host name / Observer name, since SophosXG does not provide this in the syslog file.
+Default to `firewall.localgroup.local`
+
+
+[float]
+==== SophosXG ECS fields
+
+This is a list of FortiOS fields that are mapped to ECS.
+
+[options="header"]
+|==============================================================
+| SophosXG Fields            | ECS Fields                     |
+| application                | network.protocol               |
+| classification             | rule.category                  |
+| device_id                  | observer.serial_number         |
+| domainname                 | url.domain                     |
+| dst_host                   | destination.address            |
+| dst_int                    | observer.egress.interface.name |
+| dstzonetype                | observer.egress.zone           |
+| dst_ip                     | destination.ip                 |
+| destinationip              | destination.ip                 |
+| dst_mac                    | destination.mac                |
+| dstname                    | destination.address            |
+| dst_port                   | destination.port               |
+| dst_domainname             | url.domain                     |
+| duration                   | event.duration                 |
+| filename                   | file.name                      |
+| filetype                   | file.extension                 |
+| file_size                  | file.size                      |
+| file_path                  | file.directory                 |
+| fw_rule_id                 | rule.id                        |
+| from_email_address         | source.user.email              |
+| httpstatus                 | http.response.status_code      |
+| in_interface               | observer.ingress.interface.name|
+| log_id                     | event.code                     |
+| log_subtype                | event.action                   |
+| message                    | message                        |
+| method                     | http.request.method            |
+| policy_type                | rule.ruleset                   |
+| protocol                   | network.transport              |
+| recv_bytes                 | destination.bytes              |
+| recv_pkts                  | destination.packets            |
+| referer                    | http.request.referrer          |
+| sent_bytes                 | source.bytes                   |
+| sent_pkts                  | source.packets                 |
+| sha1sum                    | file.hash.sha1                 |
+| srczonetype                | observer.ingress.zone          |
+| src_ip                     | source.ip                      |
+| src_domainname             | url.domain                     |
+| sourceip                   | source.ip                      |
+| src_mac                    | source.mac                     |
+| src_port                   | source.port                    |
+| status_code                | http.response.status_code      |
+| time_zone                  | event.timezone                 |
+| to_email_address           | destination.user.email         |
+| tran_dst_ip                | destination.nat.ip             |
+| tran_dst_port              | destination.nat.port           |
+| tran_src_ip                | source.nat.ip                  |
+| tran_src_port              | source.nat.port                |
+| url                        | url.original                   |
+| user_agent                 | user_agent.original            |
+| useragent                  | user_agent.original            |
+| user_gp                    | source.user.group              |
+| user_name                  | source.user.name               |
+| ws_protocol                | http.version                   |
+|==============================================================
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+[float]
+[[dynamic-script-compilations]]
+=== Dynamic Script Compilations
+
+The `sophosxg` filesets are based on Elasticsearch ingest pipelines and
+make extensive use of script processors and painless conditions. This can cause
+the pipelines to fail loading the first time the module is used, due to exceeding
+the maximum script compilation limits. It is recommended to tune the following
+parameters on your Elasticsearch cluster:
+
+- {ref}/circuit-breaker.html#script-compilation-circuit-breaker[script.max_compilations_rate]:
+  Increase to at least `100/5m`.
+
+- {ref}/modules-scripting-using.html#modules-scripting-using-caching[script.cache.max_size]:
+  Increase to at least `300` if using both filesets or other script-heavy modules.
+
+:modulename!:
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-sophosxg,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -55,6 +55,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-redis>>
   * <<filebeat-module-santa>>
   * <<filebeat-module-sonicwall>>
+  * <<filebeat-module-sophosxg>>
   * <<filebeat-module-squid>>
   * <<filebeat-module-suricata>>
   * <<filebeat-module-system>>
@@ -120,6 +121,7 @@ include::modules/rapid7.asciidoc[]
 include::modules/redis.asciidoc[]
 include::modules/santa.asciidoc[]
 include::modules/sonicwall.asciidoc[]
+include::modules/sophosxg.asciidoc[]
 include::modules/squid.asciidoc[]
 include::modules/suricata.asciidoc[]
 include::modules/system.asciidoc[]

x-pack/filebeat/filebeat.reference.yml

@@ -1335,6 +1335,30 @@ filebeat.modules:
     # "+02:00" for GMT+02:00
     # var.tz_offset: local
 
+#------------------------------- Sophosxg Module -------------------------------
+- module: sophosxg
+  firewall:
+    enabled: true
+
+    # Set which input to use between tcp, udp (default) or file.
+    #var.input: udp
+
+    # The interface to listen to syslog traffic. Defaults to
+    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
+    #var.syslog_host: localhost
+
+    # The port to listen for syslog traffic. Defaults to 9004.
+    #var.syslog_port: 9005
+
+    # firewall default hostanme 
+    #var.default_host_name: firewall.localgroup.local
+
+    # known firewalls
+    #var.known_devices:
+    #   "device1_serialnumber": "a.host.local"
+    #   "device2_serialnumber": "b.host.local"
+
+
 #-------------------------------- Squid Module --------------------------------
 - module: squid
   log:

x-pack/filebeat/module/sophosxg/_meta/config.yml

@@ -0,0 +1,22 @@
+- module: sophosxg
+  firewall:
+    enabled: true
+
+    # Set which input to use between tcp, udp (default) or file.
+    #var.input: udp
+
+    # The interface to listen to syslog traffic. Defaults to
+    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
+    #var.syslog_host: localhost
+
+    # The port to listen for syslog traffic. Defaults to 9004.
+    #var.syslog_port: 9005
+
+    # firewall default hostanme 
+    #var.default_host_name: firewall.localgroup.local
+
+    # known firewalls
+    #var.known_devices:
+    #   "device1_serialnumber": "a.host.local"
+    #   "device2_serialnumber": "b.host.local"
+

x-pack/filebeat/module/sophosxg/_meta/docs.asciidoc

@@ -0,0 +1,146 @@
+[role="xpack"]
+
+:modulename: sophosxg
+:has-dashboards: false
+
+== SophosXG module
+
+This is a module for SophosXG SFOS logs sent in the syslog format.
+
+To configure a remote syslog destination, please reference the https://community.sophos.com/kb/en-us/123184[SophosXG/SFOS Documentation].
+
+The syslog format choosen should be `Default`.
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+This module has been tested against SFOS version 17.5.x and 18.0.x. 
+Versions above this are expected to work but have not been tested.
+
+include::../include/configuring-intro.asciidoc[]
+
+:fileset_ex: firewall
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `firewall` fileset settings
+
+[source,yaml]
+----
+- module: sophosxg
+  firewall:
+    enabled: true
+    var.input: udp
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: 9005
+    var.host_name: firewall.localgroup.local
+----
+
+include::../include/var-paths.asciidoc[]
+
+*`var.input`*::
+
+The input to use, can be either the value `tcp`, `udp` or `file`.
+
+*`var.syslog_host`*::
+
+The interface to listen to all syslog traffic. Defaults to localhost.
+Set to 0.0.0.0 to bind to all available interfaces.
+
+*`var.syslog_port`*::
+
+The port to listen for syslog traffic. Defaults to 9005.
+
+*`var.host_name`*::
+
+Host name / Observer name, since SophosXG does not provide this in the syslog file.
+Default to `firewall.localgroup.local`
+
+
+[float]
+==== SophosXG ECS fields
+
+This is a list of FortiOS fields that are mapped to ECS.
+
+[options="header"]
+|==============================================================
+| SophosXG Fields            | ECS Fields                     |
+| application                | network.protocol               |
+| classification             | rule.category                  |
+| device_id                  | observer.serial_number         |
+| domainname                 | url.domain                     |
+| dst_host                   | destination.address            |
+| dst_int                    | observer.egress.interface.name |
+| dstzonetype                | observer.egress.zone           |
+| dst_ip                     | destination.ip                 |
+| destinationip              | destination.ip                 |
+| dst_mac                    | destination.mac                |
+| dstname                    | destination.address            |
+| dst_port                   | destination.port               |
+| dst_domainname             | url.domain                     |
+| duration                   | event.duration                 |
+| filename                   | file.name                      |
+| filetype                   | file.extension                 |
+| file_size                  | file.size                      |
+| file_path                  | file.directory                 |
+| fw_rule_id                 | rule.id                        |
+| from_email_address         | source.user.email              |
+| httpstatus                 | http.response.status_code      |
+| in_interface               | observer.ingress.interface.name|
+| log_id                     | event.code                     |
+| log_subtype                | event.action                   |
+| message                    | message                        |
+| method                     | http.request.method            |
+| policy_type                | rule.ruleset                   |
+| protocol                   | network.transport              |
+| recv_bytes                 | destination.bytes              |
+| recv_pkts                  | destination.packets            |
+| referer                    | http.request.referrer          |
+| sent_bytes                 | source.bytes                   |
+| sent_pkts                  | source.packets                 |
+| sha1sum                    | file.hash.sha1                 |
+| srczonetype                | observer.ingress.zone          |
+| src_ip                     | source.ip                      |
+| src_domainname             | url.domain                     |
+| sourceip                   | source.ip                      |
+| src_mac                    | source.mac                     |
+| src_port                   | source.port                    |
+| status_code                | http.response.status_code      |
+| time_zone                  | event.timezone                 |
+| to_email_address           | destination.user.email         |
+| tran_dst_ip                | destination.nat.ip             |
+| tran_dst_port              | destination.nat.port           |
+| tran_src_ip                | source.nat.ip                  |
+| tran_src_port              | source.nat.port                |
+| url                        | url.original                   |
+| user_agent                 | user_agent.original            |
+| useragent                  | user_agent.original            |
+| user_gp                    | source.user.group              |
+| user_name                  | source.user.name               |
+| ws_protocol                | http.version                   |
+|==============================================================
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+[float]
+[[dynamic-script-compilations]]
+=== Dynamic Script Compilations
+
+The `sophosxg` filesets are based on Elasticsearch ingest pipelines and
+make extensive use of script processors and painless conditions. This can cause
+the pipelines to fail loading the first time the module is used, due to exceeding
+the maximum script compilation limits. It is recommended to tune the following
+parameters on your Elasticsearch cluster:
+
+- {ref}/circuit-breaker.html#script-compilation-circuit-breaker[script.max_compilations_rate]:
+  Increase to at least `100/5m`.
+
+- {ref}/modules-scripting-using.html#modules-scripting-using-caching[script.cache.max_size]:
+  Increase to at least `300` if using both filesets or other script-heavy modules.
+
+:modulename!:

x-pack/filebeat/module/sophosxg/_meta/fields.yml

@@ -0,0 +1,9 @@
+- key: sophosxg
+  title: "sophosxg"
+  description: >
+    sophosxg Module
+  fields:
+    - name: sophosxg
+      type: group
+      description: >
+      fields: