Merge remote-tracking branch 'upstream/master' into feature/support-a…

…ws-on-file-changes * upstream/master: Fix discovery of Nomad allocations (elastic#28700) Add null (`\u0000`) as a valid line terminator (elastic#28998) Remove `logging.files.suffix` option and always use datetime suffixes (elastic#28927) x-pack/filebeat/module: add note for default var.input (elastic#28324) Fix AccessList & AccessMask processing in security data_stream (elastic#29016) [Metricbeat] Fix wrong mapping on "info" subkey (elastic#28782) ci: daily/weekly jobs (elastic#29050) [mergify] report open backported PRs once a week (elastic#28964)
v1v · Nov 22, 2021 · bb20e3d · bb20e3d
2 parents 8d60b8f + 389da94
commit bb20e3d
Show file tree

Hide file tree

Showing 98 changed files with 470 additions and 817 deletions.
diff --git a/.ci/schedule-daily.groovy b/.ci/schedule-daily.groovy
@@ -21,8 +21,11 @@ pipeline {
     stage('Nighly beats builds') {
       steps {
         runBuild(quietPeriod: 0, job: 'Beats/beats/master')
-        runBuild(quietPeriod: 2000, job: 'Beats/beats/7.16')
-        runBuild(quietPeriod: 4000, job: 'Beats/beats/7.15')
+        // This should be `current_8` bump.getCurrentMinorReleaseFor8
+        runBuild(quietPeriod: 2000, job: 'Beats/beats/8.0')
+        // This should be `current_7` bump.getCurrentMinorReleaseFor7 or
+        // `next_minor_7`  bump.getNextMinorReleaseFor7
+        runBuild(quietPeriod: 4000, job: 'Beats/beats/7.16')
       }
     }
   }

diff --git a/.ci/schedule-weekly.groovy b/.ci/schedule-weekly.groovy
@@ -21,8 +21,11 @@ pipeline {
     stage('Weekly beats builds') {
       steps {
         runBuild(quietPeriod: 0, job: 'Beats/beats/master')
-        runBuild(quietPeriod: 1000, job: 'Beats/beats/7.16')
-        runBuild(quietPeriod: 2000, job: 'Beats/beats/7.15')
+        // This should be `current_8` bump.getCurrentMinorReleaseFor8
+        runBuild(quietPeriod: 1000, job: 'Beats/beats/8.0')
+        // This should be `current_7` bump.getCurrentMinorReleaseFor7 or
+        // `next_minor_7`  bump.getNextMinorReleaseFor7
+        runBuild(quietPeriod: 2000, job: 'Beats/beats/7.16')
       }
     }
   }

diff --git a/.mergify.yml b/.mergify.yml
@@ -101,6 +101,18 @@ pull_request_rules:
         - files~=^\.mergify\.yml$
     actions:
       delete_head_branch:
+  - name: notify the backport has not been merged yet
+    conditions:
+      - -merged
+      - -closed
+      - author=mergify[bot]
+      - "#check-success>0"
+      - schedule=Mon-Mon 06:00-10:00[Europe/Paris]
+      - "#assignee>=1"
+    actions:
+      comment:
+        message: |
+          This pull request has not been merged yet. Could you please review and merge it @{{ assignee | join(', @') }}? 🙏
   - name: notify the backport policy
     conditions:
       - -label~=^backport

diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -27,6 +27,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - add_process_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
 - add_docker_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
 - Index template's default_fields setting is only populated with ECS fields. {pull}28596[28596] {issue}28215[28215]
+- Remove options `logging.files.suffix` and default to datetime endings. {pull}28927[28927]
 
 *Auditbeat*
 
@@ -63,6 +64,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Replace usages of `host.user.*` fields with `user.*` in `cisco`, `microsoft` and `oracle` modules. {pull}28620[28620]
 - Remove `docker` input. Please use `filestream` input with `container` parser or `container` input. {pull}28817[28817]
 - Change `threatintel` module to use new `threat.*` ECS fields. {pull}29014[29014]
+- `filestream` and `log` inputs accept null (`\u0000`) as line terminator. {pull}28998[28998]
 
 *Heartbeat*
 
@@ -136,7 +138,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Output errors when Kibana index pattern setup fails. {pull}20121[20121]
 - Fix issue in autodiscover that kept inputs stopped after config updates. {pull}20305[20305]
 - Add service resource in k8s cluster role. {pull}20546[20546]
-- Periodic metrics in logs will now report `libbeat.output.events.active` and `beat.memstats.rss`
+- Periodic metrics in logs will now report `libbeat.output.events.active` and `beat.memstats.rss` as gauges (rather than counters). {pull}22877[22877]
+- Fix discovery of Nomad allocations with multiple events during startup. {pull}28700[28700]
 - Allows disable pod events enrichment with deployment name {pull}28521[28521]
 - Fix `fingerprint` processor to give it access to the `@timestamp` field. {issue}28683[28683]
 - Fix the wrong beat name on monitoring and state endpoint {issue}27755[27755]
@@ -241,6 +244,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Tolerate faults when Windows Event Log session is interrupted {issue}27947[27947] {pull}28191[28191]
 - Add ECS 1.9 new users fields {pull}26509[26509]
 - Don't split hyphenated tokens {pull}28483[28483]
+- Correctly handle AccessMask if it is an integer or list of masks. {pull}29016[29016]
 
 *Functionbeat*
 

diff --git a/auditbeat/auditbeat.reference.yml b/auditbeat/auditbeat.reference.yml
@@ -1443,11 +1443,6 @@ logging.files:
   # file. Defaults to true.
   # rotateonstartup: true
 
-  # Rotated files are either suffixed with a number e.g. auditbeat.1 when
-  # renamed during rotation. Or when set to date, the date is added to
-  # the end of the file. On rotation a new file is created, older files are untouched.
-  #suffix: count
-
 # ============================= X-Pack Monitoring ==============================
 # Auditbeat can export internal metrics to a central Elasticsearch monitoring
 # cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

diff --git a/filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl b/filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl
@@ -94,7 +94,8 @@ filebeat.inputs:
   #max_bytes: 10485760
 
   # Characters which separate the lines. Valid values: auto, line_feed, vertical_tab, form_feed,
-  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator.
+  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator,
+  # null_terminator
   #line_terminator: auto
 
   ### Recursive glob configuration
@@ -348,7 +349,8 @@ filebeat.inputs:
   #message_max_bytes: 10485760
 
   # Characters which separate the lines. Valid values: auto, line_feed, vertical_tab, form_feed,
-  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator.
+  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator,
+  # null_terminator
   #line_terminator: auto
 
   # The ingest pipeline ID associated with this input. If this is set, it

diff --git a/filebeat/docs/modules/barracuda.asciidoc b/filebeat/docs/modules/barracuda.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "barracudawaf" devic
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 
@@ -78,7 +78,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "barracudasf" device
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/bluecoat.asciidoc b/filebeat/docs/modules/bluecoat.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "bluecoatdirector" d
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/cisco.asciidoc b/filebeat/docs/modules/cisco.asciidoc
@@ -305,7 +305,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "cisconxos" device r
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 
@@ -350,7 +350,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "ciscomeraki" device
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/cylance.asciidoc b/filebeat/docs/modules/cylance.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "cylance" device rev
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/f5.asciidoc b/filebeat/docs/modules/f5.asciidoc
@@ -37,7 +37,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "bigipapm" device re
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 
@@ -82,7 +82,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "bigipafm" device re
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/fortinet.asciidoc b/filebeat/docs/modules/fortinet.asciidoc
@@ -85,7 +85,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "forticlientendpoint
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 
@@ -130,7 +130,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "fortinetfortimail"
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 
@@ -175,7 +175,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "fortinetmgr" device
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/imperva.asciidoc b/filebeat/docs/modules/imperva.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "impervawaf" device
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/infoblox.asciidoc b/filebeat/docs/modules/infoblox.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "infobloxnios" devic
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/juniper.asciidoc b/filebeat/docs/modules/juniper.asciidoc
@@ -146,7 +146,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "junosrouter" device
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 
@@ -191,7 +191,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "netscreen" device r
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/microsoft.asciidoc b/filebeat/docs/modules/microsoft.asciidoc
@@ -224,7 +224,7 @@ include::../include/var-paths.asciidoc[]
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/netscout.asciidoc b/filebeat/docs/modules/netscout.asciidoc
@@ -31,7 +31,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "arborpeakflowsp" de
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/proofpoint.asciidoc b/filebeat/docs/modules/proofpoint.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "proofpoint" device
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/radware.asciidoc b/filebeat/docs/modules/radware.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "radwaredp" device r
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/snort.asciidoc b/filebeat/docs/modules/snort.asciidoc
@@ -31,7 +31,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "snort" device revis
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/sonicwall.asciidoc b/filebeat/docs/modules/sonicwall.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "sonicwall" device r
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/sophos.asciidoc b/filebeat/docs/modules/sophos.asciidoc
@@ -156,7 +156,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "astarosg" device re
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/squid.asciidoc b/filebeat/docs/modules/squid.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "squid" device revis
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/docs/modules/tomcat.asciidoc b/filebeat/docs/modules/tomcat.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "apachetomcat" devic
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.paths`*::
 

diff --git a/filebeat/docs/modules/zscaler.asciidoc b/filebeat/docs/modules/zscaler.asciidoc
@@ -33,7 +33,7 @@ NOTE: This was converted from RSA NetWitness log parser XML "zscalernss" device
 
 *`var.input`*::
 
-The input from which messages are read. One of `file`, `tcp` or `udp`.
+The input from which messages are read. One of `file`, `tcp` or `udp`. Defaults to `udp`.
 
 *`var.syslog_host`*::
 

diff --git a/filebeat/filebeat.reference.yml b/filebeat/filebeat.reference.yml
@@ -501,7 +501,8 @@ filebeat.inputs:
   #max_bytes: 10485760
 
   # Characters which separate the lines. Valid values: auto, line_feed, vertical_tab, form_feed,
-  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator.
+  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator,
+  # null_terminator
   #line_terminator: auto
 
   ### Recursive glob configuration
@@ -755,7 +756,8 @@ filebeat.inputs:
   #message_max_bytes: 10485760
 
   # Characters which separate the lines. Valid values: auto, line_feed, vertical_tab, form_feed,
-  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator.
+  # carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator,
+  # null_terminator
   #line_terminator: auto
 
   # The ingest pipeline ID associated with this input. If this is set, it
@@ -2355,11 +2357,6 @@ logging.files:
   # file. Defaults to true.
   # rotateonstartup: true
 
-  # Rotated files are either suffixed with a number e.g. filebeat.1 when
-  # renamed during rotation. Or when set to date, the date is added to
-  # the end of the file. On rotation a new file is created, older files are untouched.
-  #suffix: count
-
 # ============================= X-Pack Monitoring ==============================
 # Filebeat can export internal metrics to a central Elasticsearch monitoring
 # cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

diff --git a/filebeat/tests/system/filebeat.py b/filebeat/tests/system/filebeat.py
@@ -36,7 +36,7 @@ def access_registry(self, name=None, data_path=None):
         return Registry(data_path, name)
 
     def log_access(self, file=None):
-        file = file if file else self.beat_name + ".log"
+        file = file if file else self.beat_name + "-" + self.today + ".ndjson"
         return LogState(os.path.join(self.working_dir, file))
 
     def has_registry(self, name=None, data_path=None):

diff --git a/filebeat/tests/system/test_harvester.py b/filebeat/tests/system/test_harvester.py
@@ -493,11 +493,11 @@ def test_boms(self, fb_encoding, py_encoding, bom):
         filebeat = self.start_beat(output=fb_encoding + ".log")
 
         self.wait_until(
-            lambda: self.output_has(lines=1, output_file="output/" + fb_encoding),
+            lambda: self.output_has(lines=1, output_file="output/" + fb_encoding + "-" + self.today + ".ndjson"),
             max_timeout=10)
 
         # Verify that output does not contain bom
-        output = self.read_output_json(output_file="output/" + fb_encoding)
+        output = self.read_output_json(output_file="output/" + fb_encoding + "-" + self.today + ".ndjson")
         assert output[0]["message"] == message
 
         filebeat.kill_and_wait()