Merge remote-tracking branch 'upstream/master' into feature/linting-x…

…pack-when-oss-changes

* upstream/master:
  [DOCS] Add setup content to Kubernetes and Cloud Foundry docs (elastic#23580)
  [CI] Mandatory windows support for all the versions (elastic#23615)
  Add check when retrieving the worker process id using performance counters  (elastic#23647)
  Remove 4912 evtx from testing (elastic#23669)
  Add missing SSL settings (elastic#23632)
  Update X-Pack Packetbeat config (elastic#23666)
  Use hostname check from verify.go to handle patterns in TLS certs (elastic#23661)
  Fix: Dissect Cisco ASA 302013 message usernames (elastic#21196)
  Add FAQ entry for MADV settings in older versions (elastic#23429)
  Sync fixes from Integration Package Testing (elastic#23424)
  [Filebeat] Add Cisco ASA message '302023' parsing (elastic#23092)
  [Elastic Log Driver] Change hosts config flag (elastic#23628)
  Audit and Authentication Policy Change Events (elastic#20684)

.ci/scripts/generate_build_table.py

@@ -31,6 +31,7 @@
                     if "withModule" in doc["stages"][stage]:
                         withModule = doc["stages"][stage]["withModule"]
                     if "when" in doc["stages"][stage]:
-                        when = f"optional"
+                        if "not_changeset_full_match" not in doc["stages"][stage]["when"]:
+                            when = "optional"
                     print("| {} | {} | `{}` | {} | `{}` | {} |".format(
                         module, stage, command, withModule, platforms, when))

CHANGELOG.next.asciidoc

@@ -230,6 +230,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix typo in config docs {pull}23185[23185]
 - Fix `nested` subfield handling in generated Elasticsearch templates. {issue}23178[23178] {pull}23183[23183]
 - Fix CPU usage metrics on VMs with dynamic CPU config {pull}23154[23154]
+- Add FAQ entry for madvdontneed variable {pull}23429[23429]
 - Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete {pull}23419[23419]
 - Fix error loop with runaway CPU use when the Kafka output encounters some connection errors {pull}23484[23484]
 
@@ -269,6 +270,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix integer overflow in S3 offsets when collecting very large files. {pull}22523[22523]
 - Fix various processing errors in the Suricata module. {pull}23236[23236]
 - Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277]
+- CheckPoint Firewall module: Change event.severity JSON data type to a number because the field mapping is a `long`. {pull}23424[23424]
+- Cisco IOS: Change icmp.type/code and igmp.type JSON data types to strings because the fields mappings are `keyword`. {pull}23424[23424]
+- CrowdStrike Falcon: Change JSON field types to match the field mappings. {pull}23424[23424]
+- Fortinet Firewall: Drop `fortinet.firewall.assignip` when the value is "N/A". {pull}23424[23424]
+- Juniper SRX: Change JSON field types to match the field mappings. {pull}23424[23424]
+- Suricata EVE: Convert `suricata.eve.flow_id` to string because the field is a keyword in the mapping. {pull}23424[23424]
+- Zeek DNS: Ignore failures in data type conversions. And change `dns.id` JSON field to a string to match its `keyword` mapping. {pull}23424[23424]
 - Change the `event.created` in Netflow events to be the time the event was created by Filebeat
   to be consistent with ECS. {pull}23094[23094]
 - Update `filestream` reader offset when a line is skipped. {pull}23417[23417]
@@ -487,6 +495,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update config in `windows.yml` file. {issue}23027[23027]{pull}23327[23327]
 - Add stack monitoring section to elasticsearch module documentation {pull}#23286[23286]
 - Fix metric grouping for windows/perfmon module {issue}23489[23489] {pull}23505[23505]
+- Add check for iis/application_pool metricset for nil worker process id values. {issue}23605[23605] {pull}23647[23647]
 
 *Packetbeat*
 
@@ -503,6 +512,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Functionbeat*
 
+*Elastic Logging Plugin*
+- Fix out of date CLI flags on docs. {pull}23628[23628]
+
 
 ==== Added
 
@@ -812,6 +824,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Added support for first_event context in filebeat httpjson input {pull}23437[23437]
 - Add parsing of tcp flags to AWS vpcflow fileset {issue}228020[22820] {pull}23157[23157]
 - Added `alternative_host` option to google pubsub input {pull}23215[23215]
+- Added username parsing from Cisco ASA message 302013. {pull}21196[21196]
 - Added `encode_as` and `decode_as` options to httpjson along with pluggable encoders/decoders {pull}23478[23478]
 - Added `application/x-ndjson` as decode option for httpjson input {pull}23521[23521]
 - Added `application/x-www-form-urlencoded` as encode option for httpjson input {pull}23521[23521]
@@ -978,6 +991,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add additional event categorization for security and sysmon modules. {pull}22988[22988]
 - Add dns.question.subdomain fields for sysmon DNS events. {pull}22999[22999]
 - Add dns.question.top_level_domain fields for sysmon DNS events. {pull}23046[23046]
+- Add Audit and Authentication Polixy Change Events and related.ip information {pull}20684[20684]
 
 *Elastic Log Driver*
 

auditbeat/Jenkinsfile.yml

@@ -72,45 +72,33 @@ stages:
         platforms:             ## override default labels in this specific stage.
             - "windows-2016"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test auditbeat for windows-2016"
-            labels:
-                - "windows-2016"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags
     windows-2012:
         mage: "mage build unitTest"
         platforms:             ## override default labels in this specific stage.
             - "windows-2012-r2"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test auditbeat for windows-2012"
-            labels:
-                - "windows-2012"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags
     windows-10:
         mage: "mage build unitTest"
         platforms:             ## override default labels in this specific stage.
             - "windows-10"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test auditbeat for windows-10"
-            labels:
-                - "windows-10"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags
     windows-8:
         mage: "mage build unitTest"
         platforms:             ## override default labels in this specific stage.
             - "windows-8"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test auditbeat for windows-8"
-            labels:
-                - "windows-8"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags
     #windows-7:  See https://github.com/elastic/beats/issues/19831
     #    mage: "mage build unitTest"
     #    platforms:             ## override default labels in this specific stage.

auditbeat/auditbeat.reference.yml

@@ -527,8 +527,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -660,8 +658,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -867,8 +863,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -1029,8 +1023,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -1331,8 +1323,6 @@ setup.kibana:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -1534,8 +1524,6 @@ logging.files:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative

filebeat/Jenkinsfile.yml

@@ -66,34 +66,25 @@ stages:
         platforms:             ## override default labels in this specific stage.
             - "windows-2016"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test filebeat for windows-2016"
-            labels:
-                - "windows-2016"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags
     windows-10:
         mage: "mage build unitTest"
         platforms:             ## override default labels in this specific stage.
             - "windows-10"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test filebeat for windows-10"
-            labels:
-                - "windows-10"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags
     windows-8:
         mage: "mage build unitTest"
         platforms:             ## override default labels in this specific stage.
             - "windows-8"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test filebeat for windows-8"
-            labels:
-                - "windows-8"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags
     #windows-7:  See https://github.com/elastic/beats/issues/22317
     #    mage: "mage build unitTest"
     #    platforms:             ## override default labels in this specific stage.
@@ -110,9 +101,6 @@ stages:
         platforms:             ## override default labels in this specific stage.
             - "windows-7-32-bit"
         when:                  ## Override the top-level when.
-            comments:
-                - "/test filebeat for windows-7-32"
-            labels:
-                - "windows-7-32"
-            branches: true     ## for all the branches
-            tags: true         ## for all the tags
+            not_changeset_full_match: "^x-pack/.*" ## Disable the stage if ONLY changes for the x-pack
+            branches: true                         ## for all the branches
+            tags: true                             ## for all the tags

filebeat/docs/running-on-kubernetes.asciidoc

@@ -112,6 +112,28 @@ oc patch namespace kube-system -p \
 This command sets the node selector for the project to an empty string. If you
 don't run this command, the default node selector will skip master nodes.
 
+[float]
+==== Load {kib} dashboards
+
+{beatname_uc} comes packaged with various pre-built {kib} dashboards
+that you can use to visualize logs from your Kubernetes environment.
+
+If these dashboards are not already loaded into {kib}, you must <<{beatname_lc}-installation-configuration,install {beatname_uc}>>
+on any system that can connect to the {stack}, and then run the `setup` command to load the dashboards.
+To learn how, see <<load-kibana-dashboards,Load {kib} dashboards>>.
+
+The `setup` command does not load the ingest pipelines used to parse log lines. By default, ingest pipelines
+are set up automatically the first time you run {beatname_uc} and connect to {es}.
+
+[IMPORTANT]
+=======================================
+If you are using a different output other than {es}, such as {ls}, you
+need to:
+
+* <<load-template-manually>>
+* <<load-kibana-dashboards>>
+* <<load-ingest-pipelines>>
+=======================================
 
 [float]
 ==== Deploy

filebeat/filebeat.reference.yml

@@ -1407,8 +1407,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -1540,8 +1538,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -1747,8 +1743,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -1909,8 +1903,6 @@ output.elasticsearch:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -2211,8 +2203,6 @@ setup.kibana:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative
@@ -2414,8 +2404,6 @@ logging.files:
   # * full, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate.
-  # * certificate, which verifies that the provided certificate is signed by a
-  # trusted authority (CA), but does not perform any hostname verification.
   # * strict, which verifies that the provided certificate is signed by a trusted
   # authority (CA) and also verifies that the server's hostname (or IP address)
   # matches the names identified within the certificate. If the Subject Alternative