Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Allow host.* fields to be disabled in Zeek module (elastic#19113)
Browse files Browse the repository at this point in the history
If `forwarded` as configured as a tag (e.g. `var.tags: [forwarded]`) for the Zeek module then Filebeat will not add `host` fields to events. This is for use cases where Zeek is analyzing forwarded data (like from a network tap or mirror port).

Relates: elastic#13920
(cherry picked from commit 018b17c)
andrewkroh committed Jun 18, 2020

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
1 parent 49836d5 commit 49cc2fe
Showing 38 changed files with 78 additions and 38 deletions.
3 changes: 3 additions & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
@@ -47,6 +47,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
`forwarded` from the list. {issue}13920[13920]
* Cisco {pull}18753[18753]
* Checkpoint {pull}18754[18754]
* Netflow {pull}19087[19087]
* Zeek {pull}19113[19113] (`forwarded` tag is not included by default)
* Suricata {pull}19107[19107] (`forwarded` tag is not included by default)
- Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. {issue}18154[18154] {pull}18359[18359]
- With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta)
will no longer send the `host` field that contains information about the host Filebeat is
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/connection/config/connection.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/dce_rpc/config/dce_rpc.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/dhcp/config/dhcp.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/dnp3/config/dnp3.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/dns/config/dns.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

processors:
- rename:
3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/dpd/config/dpd.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/files/config/files.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/ftp/config/ftp.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

5 changes: 3 additions & 2 deletions x-pack/filebeat/module/zeek/http/config/http.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

@@ -62,7 +63,7 @@ processors:

- from: "zeek.http.password"
to: "url.password"

- from: "zeek.http.user_agent"
to: "user_agent.original"

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/intel/config/intel.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

processors:
- rename:
3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/irc/config/irc.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/kerberos/config/kerberos.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/modbus/config/modbus.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/mysql/config/mysql.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/notice/config/notice.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/ntlm/config/ntlm.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/ocsp/config/ocsp.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/pe/config/pe.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/radius/config/radius.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/rdp/config/rdp.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/rfb/config/rfb.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/sip/config/sip.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/smb_cmd/config/smb_cmd.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/smb_files/config/smb_files.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/smtp/config/smtp.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/snmp/config/snmp.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/socks/config/socks.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/ssh/config/ssh.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/ssl/config/ssl.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/stats/config/stats.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/syslog/config/syslog.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/traceroute/config/traceroute.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/tunnel/config/tunnel.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/weird/config/weird.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

3 changes: 2 additions & 1 deletion x-pack/filebeat/module/zeek/x509/config/x509.yml
Original file line number Diff line number Diff line change
@@ -4,7 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

json.keys_under_root: false

0 comments on commit 49cc2fe

Please sign in to comment.