Automatic merge from master to 7.x branch

CHANGELOG-developer.next.asciidoc

@@ -35,5 +35,9 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
   by `make` and `mage`. Example: `export PYTHON_EXE=python2.7`. {pull}11212[11212]
 - Prometheus helper for metricbeat contains now `Namespace` field for `prometheus.MetricsMappings` {pull}11424[11424]
 - Update Jinja2 version to 2.10.1. {pull}11817[11817]
-- Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. {pull}11777[11777]
+- Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. {pull}11777[11777],{pull}12065[12065],{pull}12067[12067],{pull}12160[12160]
 - Update urllib3 version to 1.24.2 {pull}11930[11930]
+- Add libbeat/common/cleanup package. {pull}12134[12134]
+- Only Load minimal template if no fields are provided. {pull}12103[12103]
+- Add new option `IgnoreAllErrors` to `libbeat.common.schema` for skipping fields that failed while converting. {pull}12089[12089]
+- Deprecate setup cmds for `template` and `ilm-policy`. Add new setup cmd for `index-management`. {pull}12132[12132]

CHANGELOG.next.asciidoc

@@ -20,6 +20,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Socket dataset: Exclude localhost by default {pull}11993[11993]
 
 *Filebeat*
+
 - Modify apache/error dataset to follow ECS. {pull}8963[8963]
 - Rename many `traefik.access.*` fields to map to ECS. {pull}9005[9005]
 - Fix parsing of GC entries in elasticsearch server log. {issue}9513[9513] {pull}9810[9810]
@@ -63,19 +64,27 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Not hiding error in case of http failure using elastic fetcher {pull}11604[11604]
 - Relax validation of the X-Pack license UID value. {issue}11640[11640]
 - Fix a parsing error with the X-Pack license check on 32-bit system. {issue}11650[11650]
+- Escape BOM on JsonReader before trying to decode line {pull}11661[11661]
 - Fix ILM policy always being overwritten. {pull}11671[11671]
 - Fix template always being overwritten. {pull}11671[11671]
 - Fix matching of string arrays in contains condition. {pull}11691[11691]
 - Fix formatting for `event.duration`, "human readable" was not working well for this. {pull}11675[11675]
 - Fix initialization of the TCP input logger. {pull}11605[11605]
 - Fix flaky service_integration_windows_test test by introducing a confidence factor and enriching the error message with more service details. {issue}8880[8880] and {issue}7977[7977]
 - Replace wmi queries with win32 api calls as they were consuming CPU resources {issue}3249[3249] and {issue}11840[11840]
+- Fix queue.spool.write.flush.events config type. {pull}12080[12080]
+- Fixed a memory leak when using the add_process_metadata processor under Windows. {pull}12100[12100]
+- Fixed Beat ID being reported by GET / API. {pull}12180[12180]
 
 *Auditbeat*
 
 - Package dataset: dlopen versioned librpm shared objects. {pull}11565[11565]
 - Package dataset: Nullify Librpm's rpmsqEnable. {pull}11628[11628]
 - Package dataset: Log error when Homebrew is not installed. {pull}11667[11667]
+- Process dataset: Fixed a memory leak under Windows. {pull}12100[12100]
+- Login dataset: Fix re-read of utmp files. {pull}12028[12028]
+- Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. {issue}12147[12147] {pull}12168[12168]
+- Fix formatting of config files on macOS and Windows. {pull}12148[12148]
 
 *Filebeat*
 
@@ -87,6 +96,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix `add_docker_metadata` source matching, using `log.file.path` field now. {pull}11577[11577]
 - Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. {pull}11591[11591]
 - Reduce memory usage if long lines are truncated to fit `max_bytes` limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. {pull}11524[11524]
+- Fix memory leak in Filebeat pipeline acker. {pull}12063[12063]
+- Fix goroutine leak caused on initialization failures of log input. {pull}12125[12125]
+- Fix goroutine leak on non-explicit finalization of log input. {pull}12164[12164]
 
 *Heartbeat*
 
@@ -105,11 +117,19 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Change diskio metrics retrieval method (only for Windows) from wmi query to DeviceIOControl function using the IOCTL_DISK_PERFORMANCE control code {pull}11635[11635]
 - Call GetMetricData api per region instead of per instance. {issue}11820[11820] {pull}11882[11882]
 - Update documentation with cloudwatch:ListMetrics permission. {pull}11987[11987]
+- Check permissions in system socket metricset based on capabilities. {pull}12039[12039]
+- Get process information from sockets owned by current user when system socket metricset is run without privileges. {pull}12039[12039]
+- Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. {issue}8264[8264] {pull}12086[12086]
+- Fixed a socket leak in the postgresql module under Windows when SSL is disabled on the server. {pull}11393[11393]
+- Change some field type from scaled_float to long in aws module. {pull}11982[11982]
+- Fixed RabbitMQ `queue` metricset gathering when `consumer_utilisation` is set empty at the metrics source {pull}12089[12089] 
 
 *Packetbeat*
 
 - Prevent duplicate packet loss error messages in HTTP events. {pull}10709[10709]
 - Avoid reporting unknown MongoDB opcodes more than once. {pull}10878[10878]
+- Fixed a memory leak when using process monitoring under Windows. {pull}12100[12100]
+- Improved debug logging efficiency in PGQSL module. {issue}12150[12150]
 
 *Winlogbeat*
 
@@ -134,12 +154,16 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. {pull}NNNN[NNNN]
 - Add `add_observer_metadata` processor. {pull}11394[11394]
 - Add `decode_csv_fields` processor. {pull}11753[11753]
+- Add `convert` processor for converting data types of fields. {issue}8124[8124] {pull}11686[11686]
+- New `extract_array` processor. {pull}11761[11761]
+- Add number of goroutines to reported metrics. {pull}12135[12135]
 
 *Auditbeat*
 
 - Auditd module: Add `event.outcome` and `event.type` for ECS. {pull}11432[11432]
 - Package: Enable suse. {pull}11634[11634]
 - Add support to the system package dataset for the SUSE OS family. {pull}11634[11634]
+- Process: Add file hash of process executable. {pull}11722[11722]
 
 *Filebeat*
 
@@ -157,6 +181,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add Filebeat envoyproxy module. {pull}11700[11700]
 - Add apache2(httpd) log path (`/var/log/httpd`) to make apache2 module work out of the box on Redhat-family OSes. {issue}11887[11887] {pull}11888[11888]
 - Add support to new MongoDB additional diagnostic information {pull}11952[11952]
+- New module `palo_alto` for Palo Alto Networks PAN-OS logs. {pull}11999[11999]
+- Add RabbitMQ module. {pull}12032[12032]
+- Add new `container` input. {pull}12162[12162]
 
 *Heartbeat*
 
@@ -179,6 +206,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add check on object name in the counter path if the instance name is missing {issue}6528[6528] {pull}11878[11878]
 - Add AWS cloudwatch metricset. {pull}11798[11798] {issue}11734[11734]
 - Add `regions` in aws module config to specify target regions for querying cloudwatch metrics. {issue}11932[11932] {pull}11956[11956]
+- Keep `etcd` followers members from reporting `leader` metricset events {pull}12004[12004]
 
 *Packetbeat*
 
@@ -196,6 +224,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Filebeat*
 
+- `docker` input is deprecated in favour `container`. {pull}12162[12162]
+
 *Heartbeat*
 
 *Journalbeat*

NOTICE.txt

@@ -715,15 +715,15 @@ Apache License 2.0
 
 --------------------------------------------------------------------
 Dependency: github.com/elastic/go-sysinfo
-Revision: ab4f04edfc3d6b3864f5f06a068ddab9ad79774f
+Revision: 9a4be54a53be4c48b44d351d52fb425a5e274be5
 License type (autodetected): Apache-2.0
 ./vendor/github.com/elastic/go-sysinfo/LICENSE.txt:
 --------------------------------------------------------------------
 Apache License 2.0
 
 -------NOTICE.txt-----
 Elastic go-sysinfo
-Copyright 2017-2018 Elasticsearch B.V.
+Copyright 2017-2019 Elasticsearch B.V.
 
 This product includes software developed at
 Elasticsearch, B.V. (https://www.elastic.co/).
@@ -765,8 +765,8 @@ Elasticsearch, B.V. (https://www.elastic.co/).
 
 --------------------------------------------------------------------
 Dependency: github.com/elastic/gosigar
-Version: v0.10.1
-Revision: fc57ef8c6efc0b4fdc6d7c623173073a6d3d4736
+Version: v0.10.2
+Revision: 1227b9d6877d126ad640087e44439d70dba2df4f
 License type (autodetected): Apache-2.0
 ./vendor/github.com/elastic/gosigar/LICENSE:
 --------------------------------------------------------------------
@@ -1921,7 +1921,7 @@ Apache License 2.0
 
 --------------------------------------------------------------------
 Dependency: github.com/lib/pq
-Revision: 2704adc878c21e1329f46f6e56a1c387d788ff94
+Revision: 2ff3cb3adc01768e0a552b3a02575a6df38a9bea
 License type (autodetected): MIT
 ./metricbeat/module/postgresql/vendor/github.com/lib/pq/LICENSE.md:
 --------------------------------------------------------------------

auditbeat/docs/fields.asciidoc

@@ -6540,6 +6540,157 @@ type: keyword
 ID uniquely identifying the process. It is computed as a SHA-256 hash of the host ID, PID, and process start time.
 
 
+--
+
+[float]
+== hash fields
+
+Hashes of the executable. The keys are algorithm names and the values are the hex encoded digest values.
+
+
+
+*`process.hash.blake2b_256`*::
++
+--
+type: keyword
+
+BLAKE2b-256 hash of the executable.
+
+--
+
+*`process.hash.blake2b_384`*::
++
+--
+type: keyword
+
+BLAKE2b-384 hash of the executable.
+
+--
+
+*`process.hash.blake2b_512`*::
++
+--
+type: keyword
+
+BLAKE2b-512 hash of the executable.
+
+--
+
+*`process.hash.md5`*::
++
+--
+type: keyword
+
+MD5 hash of the executable.
+
+--
+
+*`process.hash.sha1`*::
++
+--
+type: keyword
+
+SHA1 hash of the executable.
+
+--
+
+*`process.hash.sha224`*::
++
+--
+type: keyword
+
+SHA224 hash of the executable.
+
+--
+
+*`process.hash.sha256`*::
++
+--
+type: keyword
+
+SHA256 hash of the executable.
+
+--
+
+*`process.hash.sha384`*::
++
+--
+type: keyword
+
+SHA384 hash of the executable.
+
+--
+
+*`process.hash.sha3_224`*::
++
+--
+type: keyword
+
+SHA3_224 hash of the executable.
+
+--
+
+*`process.hash.sha3_256`*::
++
+--
+type: keyword
+
+SHA3_256 hash of the executable.
+
+--
+
+*`process.hash.sha3_384`*::
++
+--
+type: keyword
+
+SHA3_384 hash of the executable.
+
+--
+
+*`process.hash.sha3_512`*::
++
+--
+type: keyword
+
+SHA3_512 hash of the executable.
+
+--
+
+*`process.hash.sha512`*::
++
+--
+type: keyword
+
+SHA512 hash of the executable.
+
+--
+
+*`process.hash.sha512_224`*::
++
+--
+type: keyword
+
+SHA512/224 hash of the executable.
+
+--
+
+*`process.hash.sha512_256`*::
++
+--
+type: keyword
+
+SHA512/256 hash of the executable.
+
+--
+
+*`process.hash.xxh64`*::
++
+--
+type: keyword
+
+XX64 hash of the executable.
+
 --
 
 

auditbeat/docs/getting-started.asciidoc

@@ -282,4 +282,4 @@ The dashboards are provided as examples. We recommend that you
 {kibana-ref}/dashboard.html[customize] them to meet your needs.
 
 [role="screenshot"]
-image:./images/auditbeat-file-integrity-dashboard.png[Auditbeat File Integrity Dashboard]
+image::./images/auditbeat-file-integrity-dashboard.png[Auditbeat File Integrity Dashboard]

auditbeat/docs/modules/auditd.asciidoc

@@ -298,5 +298,6 @@ auditbeat.modules:
     #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
     #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
 
+
 ----