Fix azure fields names (elastic#14098)

* Fix azure fields names * Add changelog entry * Add claims fields * Add "claims.name" field (dashboards) * Create claims_initiated_by object * REmove dummy pipeline file * Fix condition on category and initiated_by fields (cherry picked from commit 6f9dcd9)
narph · Oct 18, 2019 · db6d568 · db6d568
1 parent 508543c
commit db6d568
Show file tree

Hide file tree

Showing 9 changed files with 205 additions and 16 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -46,6 +46,47 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Filebeat*
 
+- Add support for Cisco syslog format used by their switch. {pull}10760[10760]
+- Cover empty request data, url and version in Apache2 module{pull}10730[10730]
+- Fix registry entries not being cleaned due to race conditions. {pull}10747[10747]
+- Improve detection of file deletion on Windows. {pull}10747[10747]
+- Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. {pull}11591[11591]
+- Reduce memory usage if long lines are truncated to fit `max_bytes` limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. {pull}11524[11524]
+- Fix memory leak in Filebeat pipeline acker. {pull}12063[12063]
+- Fix goroutine leak caused on initialization failures of log input. {pull}12125[12125]
+- Fix goroutine leak on non-explicit finalization of log input. {pull}12164[12164]
+- Skipping unparsable log entries from docker json reader {pull}12268[12268]
+- Parse timezone in PostgreSQL logs as part of the timestamp {pull}12338[12338]
+- Load correct pipelines when system module is configured in modules.d. {pull}12340[12340]
+- Fix timezone offset parsing in system/syslog. {pull}12529[12529]
+- When TLS is configured for the TCP input and a `certificate_authorities` is configured we now default to `required` for the `client_authentication`. {pull}12584[12584]
+- Apply `max_message_size` to incoming message buffer. {pull}11966[11966]
+- Syslog input will now omit the `process` object from events if it is empty. {pull}12700[12700]
+- Fix multiline pattern in Postgres which was too permissive {issue}12078[12078] {pull}13069[13069]
+- Allow path variables to be used in files loaded from modules.d. {issue}13184[13184]
+- Fix filebeat autodiscover fileset hint for container input. {pull}13296[13296]
+- Fix incorrect references to index patterns in AWS and CoreDNS dashboards. {pull}13303[13303]
+- Fix timezone parsing of system module ingest pipelines. {pull}13308[13308]
+- Fix timezone parsing of elasticsearch module ingest pipelines. {pull}13367[13367]
+- Change iis url path grok pattern from URIPATH to NOTSPACE. {issue}12710[12710] {pull}13225[13225] {issue}7951[7951] {pull}13378[13378]
+- Fix timezone parsing of nginx module ingest pipelines. {pull}13369[13369]
+- Fix incorrect field references in envoyproxy dashboard {issue}13420[13420] {pull}13421[13421]
+- Fixed early expiration of templates (Netflow v9 and IPFIX). {pull}13821[13821]
+- Fixed bad handling of sequence numbers when multiple observation domains were exported by a single device (Netflow V9 and IPFIX). {pull}13821[13821]
+- Fix timezone parsing of rabbitmq module ingest pipelines. {pull}13879[13879]
+- Fix conditions and error checking of date processors in ingest pipelines that use `event.timezone` to parse dates. {pull}13883[13883]
+- Fix timezone parsing of Cisco module ingest pipelines. {pull}13893[13893]
+- Fix timezone parsing of logstash module ingest pipelines. {pull}13890[13890]
+- cisco asa and ftd filesets: Fix parsing of message 106001. {issue}13891[13891] {pull}13903[13903]
+- Fix timezone parsing of iptables, mssql and panw module ingest pipelines. {pull}13926[13926]
+- Fix merging of fields specified in global scope with fields specified under an input's scope. {issue}3628[3628] {pull}13909[13909]
+- Fix delay in enforcing close_renamed and close_removed options. {issue}13488[13488] {pull}13907[13907]
+- Fix missing netflow fields in index template. {issue}13768[13768] {pull}13914[13914]
+- Fix cisco module's asa and ftd filesets parsing of domain names where an IP address is expected. {issue}14034[14034]
+- Fixed increased memory usage with large files when multiline pattern does not match. {issue}14068[14068]
+- panw module: Use geo.name instead of geo.country_iso_code for free-form location. {issue}13272[13272]
+- Fix azure fields names. {pull}14098[14098]
+- Fix calculation of `network.bytes` and `network.packets` for bi-directional netflow events. {pull}14111[14111]
 
 *Heartbeat*
 

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -1559,7 +1559,17 @@ type: keyword
 
 --
 
-*`azure.resource.type`*::
+*`azure.resource.provider`*::
++
+--
+Resource type/namespace
+
+
+type: keyword
+
+--
+
+*`azure.resource.namespace`*::
 +
 --
 Resource type/namespace
@@ -1589,21 +1599,78 @@ Fields for Azure activity logs.
 [float]
 === identity
 
-The canonical user ID of the owner of the source bucket.
+Identity
+
+
+
+[float]
+=== claims_initiated_by_user
 
+Claims initiated by user
+
+
+
+*`azure.activitylogs.identity.claims_initiated_by_user.name`*::
++
+--
+Name
+
+
+type: keyword
+
+--
+
+*`azure.activitylogs.identity.claims_initiated_by_user.givenname`*::
++
+--
+Givenname
 
 
+type: keyword
+
+--
+
+*`azure.activitylogs.identity.claims_initiated_by_user.surname`*::
++
+--
+Surname
+
+
+type: keyword
+
+--
+
+*`azure.activitylogs.identity.claims_initiated_by_user.fullname`*::
++
+--
+Fullname
+
+
+type: keyword
+
+--
+
+*`azure.activitylogs.identity.claims_initiated_by_user.schema`*::
++
+--
+Schema
+
+
+type: keyword
+
+--
+
 [float]
 === authorization
 
-Node allocatable pods
+Authorization
 
 
 
 [float]
 === evidence
 
-Node allocatable pods
+Evidence
 
 
 

diff --git a/x-pack/filebeat/module/azure/_meta/fields.yml b/x-pack/filebeat/module/azure/_meta/fields.yml
@@ -33,7 +33,11 @@
               type: keyword
               description: >
                 Resource group
-            - name: type
+            - name: provider
+              type: keyword
+              description: >
+                Resource type/namespace
+            - name: namespace
               type: keyword
               description: >
                 Resource type/namespace

diff --git a/x-pack/filebeat/module/azure/activitylogs/_meta/fields.yml b/x-pack/filebeat/module/azure/activitylogs/_meta/fields.yml
@@ -7,17 +7,42 @@
     - name: identity
       type: group
       description: >
-        The canonical user ID of the owner of the source bucket.
+        Identity
       fields:
+        - name: claims_initiated_by_user
+          type: group
+          description: >
+            Claims initiated by user
+          fields:
+            - name: name
+              type: keyword
+              description: >
+                Name
+            - name: givenname
+              type: keyword
+              description: >
+                Givenname
+            - name: surname
+              type: keyword
+              description: >
+                Surname
+            - name: fullname
+              type: keyword
+              description: >
+                Fullname
+            - name: schema
+              type: keyword
+              description: >
+                Schema
         - name: authorization
           type: group
           description: >
-            Node allocatable pods
+            Authorization
           fields:
             - name: evidence
               type: group
               description: >
-                Node allocatable pods
+                Evidence
               fields:
                 - name: role_assignment_scope
                   type: keyword

diff --git a/x-pack/filebeat/module/azure/activitylogs/ingest/pipeline.json b/x-pack/filebeat/module/azure/activitylogs/ingest/pipeline.json
@@ -20,15 +20,16 @@
           "date": {
               "field": "azure.activitylogs.time",
               "target_field": "@timestamp",
-              "ignore_failure": false,
+              "ignore_failure": true,
               "formats": [
                   "ISO8601"
               ]
           }
       },
       {
           "remove": {
-              "field": ["message", "azure.activitylogs.time"]
+              "field": ["message", "azure.activitylogs.time"],
+              "ignore_missing": true
           }
       },
       {
@@ -79,10 +80,17 @@
       {
           "script": {
               "lang": "painless",
-              "source": "if (ctx.azure.activitylogs.properties.eventCategory == null) { if (ctx.azure.activitylogs.properties.policies != null)  { ctx.event.category = 'Policy';}} else {ctx.event.category = ctx.azure.activitylogs.properties.eventCategory;} if (ctx.event.category == null) {ctx.event.category='Administrative'}",
+              "source": "if (ctx.azure.activitylogs.properties != null && ctx.azure.activitylogs.properties.eventCategory != null) {ctx.eventCategory = ctx.azure.activitylogs.properties.eventCategory} if (ctx.azure.activitylogs.properties != null && ctx.azure.activitylogs.properties.policies != null)  { ctx.eventCategory = 'Policy'}  if (ctx.eventCategory == null) {ctx.eventCategory='Administrative'}",
               "ignore_failure": true
           }
       },
+      {
+          "rename": {
+              "field": "eventCategory",
+              "target_field": "event.category",
+              "ignore_missing": true
+          }
+      },
       {
           "rename": {
               "field": "azure.activitylogs.resultType",
@@ -167,6 +175,50 @@
               "ignore_missing": true
           }
       },
+      {
+          "geoip" : {
+              "field" : "source.ip",
+              "target_field" : "geo",
+              "ignore_missing": true
+          }
+      },
+      {
+          "rename": {
+              "field": "azure.activitylogs.identity.claims.name",
+              "target_field": "azure.activitylogs.identity.claims_initiated_by_user.fullname",
+              "ignore_missing": true
+          }
+      },
+      {
+          "script": {
+              "lang": "painless",
+              "source": "if (ctx.azure.activitylogs.identity.claims['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname'] != null) {  ctx.azure.activitylogs.identity.claims_initiated_by_user.surname = ctx.azure.activitylogs.identity.claims['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname'];}",
+              "ignore_failure": true
+          }
+      },
+      {
+          "script": {
+              "lang": "painless",
+              "source": "if (ctx.azure.activitylogs.identity.claims['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name'] != null) {  ctx.azure.activitylogs.identity.claims_initiated_by_user.name = ctx.azure.activitylogs.identity.claims['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name'];}",
+              "ignore_failure": true
+          }
+      },
+      {
+          "script": {
+              "lang": "painless",
+              "source": "if (ctx.azure.activitylogs.identity.claims['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname'] != null) {  ctx.azure.activitylogs.identity.claims_initiated_by_user.givenname = ctx.azure.activitylogs.identity.claims['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname'];}",
+              "ignore_failure": true
+          }
+      },
+      {
+          "set": {
+              "if" : "ctx.azure.activitylogs.identity!= null && ctx.azure.activitylogs.identity.claims_initiated_by_user != null && ctx.azure.activitylogs.identity.claims_initiated_by_user.name != null",
+              "field": "azure.activitylogs.identity.claims_initiated_by_user.schema",
+              "value": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims",
+              "ignore_failure": true
+          }
+      },
+
       {
           "pipeline": {
               "name": "{< IngestPipeline "azure-shared-pipeline" >}"

diff --git a/x-pack/filebeat/module/azure/activitylogs/test/activity_log_expected.json b/x-pack/filebeat/module/azure/activitylogs/test/activity_log_expected.json
@@ -70,7 +70,7 @@
         "azure" : {
             "subscription_id" : "2a7e2503-d7e2-405a-a84c-c333b9f7cb73",
             "resource" : {
-                "resource_group" : "SA-HEMANT",
+                "group" : "SA-HEMANT",
                 "provider" : "MICROSOFT.EVENTHUB",
                 "namespace" : "AZURELSEVENTS",
                 "id" : "/SUBSCRIPTIONS/2a7e2503-d7e2-405a-a84c-c333b9f7cb73/RESOURCEGROUPS/SA-HEMANT/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/AZURELSEVENTS/AUTHORIZATIONRULES/ROOTMANAGESHAREDACCESSKEY",

diff --git a/x-pack/filebeat/module/azure/auditlogs/ingest/pipeline.json b/x-pack/filebeat/module/azure/auditlogs/ingest/pipeline.json
@@ -16,7 +16,7 @@
           "date": {
               "field": "azure.auditlogs.time",
               "target_field": "@timestamp",
-              "ignore_failure": false,
+              "ignore_failure": true,
               "formats": [
                   "ISO8601"
               ]

diff --git a/x-pack/filebeat/module/azure/azure-shared-pipeline.json b/x-pack/filebeat/module/azure/azure-shared-pipeline.json
@@ -10,7 +10,7 @@
         {
             "grok": {
                 "field": "azure.resource_id",
-                "patterns": ["/SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.resource_group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/NAMESPACES/%{NAMESPACE:azure.resource.namespace}/AUTHORIZATIONRULES/%{RULE:azure.resource.authorization_rule}"],
+                "patterns": ["/SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/NAMESPACES/%{NAMESPACE:azure.resource.namespace}/AUTHORIZATIONRULES/%{RULE:azure.resource.authorization_rule}"],
                 "pattern_definitions" : {
                     "SUBID" : "(\\{){0,1}[0-9a-fA-F]{8}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{12}(\\}){0,1}",
                     "GROUPID" : ".+",
@@ -24,7 +24,7 @@
         {
             "grok": {
                 "field": "azure.resource_id",
-                "patterns": ["/SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.resource_group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}"],
+                "patterns": ["/SUBSCRIPTIONS/%{SUBID:azure.subscription_id}/RESOURCEGROUPS/%{GROUPID:azure.resource.group}/PROVIDERS/%{PROVIDERNAME:azure.resource.provider}/%{NAME:azure.resource.name}"],
                 "pattern_definitions" : {
                     "SUBID" : "(\\{){0,1}[0-9a-fA-F]{8}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{12}(\\}){0,1}",
                     "GROUPID" : ".+",

diff --git a/x-pack/filebeat/module/azure/fields.go b/x-pack/filebeat/module/azure/fields.go