[m365_defender] Add support for Incident data-stream with new Security Graph API

packages/m365_defender/_dev/build/docs/README.md

@@ -1,24 +1,58 @@
 # M365 Defender integration
 
-This integration is for M365 Defender logs, previously known as Threat Protection.
+## Overview
 
-## Configuration
+The [Microsoft 365 Defender](https://learn.microsoft.com/en-us/microsoft-365/security/defender) integration allows you to monitor Incident Logs. Microsoft 365 Defender is a unified pre and post-breach enterprise defense suite that natively coordinates detection, prevention, investigation, and response across endpoints, identities, email, and applications to provide integrated protection against sophisticated attacks.
 
-To configure access for Elastic Agent to communicate with Microsoft 365 Defender you will have to create a new Azure Application registration, this will again return OAuth tokens with access to the Microsoft 365 Defender API.
+Use the Microsoft 365 Defender integration to collect and parse data from the Microsoft Graph Security Beta REST API and Microsoft 365 Defender API. Then visualise that data in Kibana.
 
-The procedure to create an application is found on the below link:
+For example, you could use the data from this integration to consolidate and correlate security alerts from multiple sources. Also, by looking into the alert and incident, a user can take an appropriate action in the Microsoft 365 Defender Portal.
 
-[Create a new Azure Application](https://docs.microsoft.com/en-us/microsoft-365/security/mtp/api-create-app-web?view=o365-worldwide#create-an-app)
+## Data streams
 
-When giving the application the API permissions described in the documentation (Incident.Read.All) it will only grant access to read Incidents from 365 Defender and nothing else in the Azure Domain.
+The Microsoft 365 Defender integration collects logs for two types of events: Incident and Log.
 
-After the application has been created, it should contain 3 values that you need to apply to the module configuration.
+**Incident** in Microsoft 365 Defender is a collection of correlated alert instances and associated metadata that reflects the story of an attack in a tenant. It uses the Microsoft Graph Security Beta REST API to collect data. See Example Schema [here](https://learn.microsoft.com/en-us/graph/api/resources/security-incident?view=graph-rest-beta#properties).
 
-These values are:
+**Log** incidents API allows you to sort through incidents to create an informed cybersecurity response. It exposes a collection of incidents that were flagged in your network, within the time range you specified in your environmental retention policy. The most recent incidents are displayed at the top of the list. Each incident contains an array of related alerts and their related entities. It uses the Microsoft 365 Defender API to collect data. See Example Schema [here](https://learn.microsoft.com/en-us/microsoft-365/security/defender/api-list-incidents?view=o365-worldwide#schema-mapping).
 
-- Client ID
-- Client Secret
-- Tenant ID
+## Requirements
+
+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
+
+This module has used **Microsoft Graph Security Beta REST API** and **Microsoft 365 Defender API**.
+
+## Setup
+
+### To collect data from Microsoft Graph Security Beta REST API, follow the below steps:
+
+1. [Register a new Azure Application](https://learn.microsoft.com/en-us/graph/auth-register-app-v2?view=graph-rest-beta).
+2. Permission required for accessing Incident API would be **SecurityIncident.Read.All**. See more details [here](https://learn.microsoft.com/en-us/graph/auth-v2-service?view=graph-rest-beta)
+3. After the application has been created, it will generate Client ID, Client Secret and Tenant ID values that are required for alert and incident data collection.
+
+### To collect data from Microsoft 365 Defender REST API, follow the below steps:
+
+1. [Register a new Azure Application](https://learn.microsoft.com/en-us/microsoft-365/security/defender/api-create-app-web?view=o365-worldwide#create-an-app).
+2. Permission required for accessing Log API would be **Incident.Read.All**.
+3. After the application has been created, it will generate Client ID, Client Secret and Tenant ID values that are required for log data collection.
+
+## Logs reference
+
+### incident
+
+This is the `incident` dataset.
+
+#### Example
+
+{{event "incident"}}
+
+{{fields "incident"}}
+
+### log
+
+This is the `log` dataset.
+
+#### Example
 
 {{event "log"}}
 

packages/m365_defender/_dev/deploy/docker/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '2.3'
 services:
   m365-defender-http:
-    image: docker.elastic.co/observability/stream:v0.6.1
+    image: docker.elastic.co/observability/stream:v0.8.0
     ports:
       - 8080
     volumes:

packages/m365_defender/_dev/deploy/docker/http-mock-config.yml

@@ -13,6 +13,18 @@ rules:
             - "application/json"
         body: |-
           {"access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Ilg1ZVhrN","token_type": "Bearer","not_before": 1549647431,"expires_in": 3600}
+  - path: /beta/security/incidents
+    methods: [GET]
+    request_headers:
+      Authorization:
+        - "Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Ilg1ZVhrN"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |
+          {"value":[{"@odata.type":"#microsoft.graph.security.incident","id":"2972395","incidentWebUrl":"https://security.microsoft.com/incidents/2972395?tid=12f988bf-16f1-11af-11ab-1d7cd011db47","redirectIncidentId":null,"tenantId":"b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","displayName":"Multi-stage incident involving Initial access & Command and control on multiple endpoints reported by multiple sources","createdDateTime":"2021-08-13T08:43:35.5533333Z","lastUpdateDateTime":"2021-09-30T09:35:45.1133333Z","assignedTo":"[email protected]","classification":"truePositive","determination":"multiStagedAttack","status":"active","severity":"medium","tags":["Demo"],"comments":[{"comment":"Demo incident","createdBy":"[email protected]","createdTime":"2021-09-30T12:07:37.2756993Z"}],"alerts":[{"@odata.type":"#microsoft.graph.security.alert","id":"da637551227677560813_-961444813","providerAlertId":"da637551227677560813_-961444813","incidentId":"28282","status":"new","severity":"low","classification":"unknown","determination":"unknown","serviceSource":"microsoftDefenderForEndpoint","detectionSource":"antivirus","detectorId":"e0da400f-affd-43ef-b1d5-afc2eb6f2756","tenantId":"b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","title":"Suspicious execution of hidden file","description":"A hidden file has been launched. This activity could indicate a compromised host. Attackers often hide files associated with malicious tools to evade file system inspection and defenses.","recommendedActions":"Collect artifacts and determine scope\n�\tReview the machine timeline for suspicious activities that may have occurred before and after the time of the alert, and record additional related artifacts (files, IPs/URLs) \n�\tLook for the presence of relevant artifacts on other systems. Identify commonalities and differences between potentially compromised systems.\n�\tSubmit relevant files for deep analysis and review resulting detailed behavioral information.\n�\tSubmit undetected files to the MMPC malware portal\n\nInitiate containment & mitigation \n�\tContact the user to verify intent and initiate local remediation actions as needed.\n�\tUpdate AV signatures and run a full scan. The scan might reveal and remove previously-undetected malware components.\n�\tEnsure that the machine has the latest security updates. In particular, ensure that you have installed the latest software, web browser, and Operating System versions.\n�\tIf credential theft is suspected, reset all relevant users passwords.\n�\tBlock communication with relevant URLs or IPs at the organization�s perimeter.","category":"DefenseEvasion","assignedTo":null,"alertWebUrl":"https://security.microsoft.com/alerts/da637551227677560813_-961444813?tid=b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","incidentWebUrl":"https://security.microsoft.com/incidents/28282?tid=b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","actorDisplayName":null,"threatDisplayName":null,"threatFamilyName":null,"mitreTechniques":["T1564.001"],"createdDateTime":"2021-04-27T12:19:27.7211305Z","lastUpdateDateTime":"2021-05-02T14:19:01.3266667Z","resolvedDateTime":null,"firstActivityDateTime":"2021-04-26T07:45:50.116Z","lastActivityDateTime":"2021-05-02T07:56:58.222Z","comments":[],"evidence":[{"@odata.type":"#microsoft.graph.security.deviceEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"firstSeenDateTime":"2020-09-12T07:28:32.4321753Z","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","azureAdDeviceId":null,"deviceDnsName":"tempDns","osPlatform":"Windows10","osBuild":22424,"version":"Other","healthStatus":"active","riskScore":"medium","rbacGroupId":75,"rbacGroupName":"UnassignedGroup","onboardingStatus":"onboarded","defenderAvStatus":"unknown","loggedOnUsers":[],"roles":["compromised"],"tags":["Test Machine"],"vmMetadata":{"vmId":"ca1b0d41-5a3b-4d95-b48b-f220aed11d78","cloudProvider":"azure","resourceId":"/subscriptions/8700d3a3-3bb7-4fbe-a090-488a1ad04161/resourceGroups/WdatpApi-EUS-STG/providers/Microsoft.Compute/virtualMachines/NirLaviTests","subscriptionId":"8700d3a3-3bb7-4fbe-a090-488a1ad04161"}},{"@odata.type":"#microsoft.graph.security.fileEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"detectionStatus":"detected","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","roles":[],"tags":[],"fileDetails":{"sha1":"5f1e8acedc065031aad553b710838eb366cfee9a","sha256":"8963a19fb992ad9a76576c5638fd68292cffb9aaac29eb8285f9abf6196a7dec","fileName":"MsSense.exe","filePath":"C:\\Program Files\\temp","fileSize":6136392,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null}},{"@odata.type":"#microsoft.graph.security.processEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"processId":4780,"parentProcessId":668,"processCommandLine":"\"MsSense.exe\"","processCreationDateTime":"2021-08-12T12:43:19.0772577Z","parentProcessCreationDateTime":"2021-08-12T07:39:09.0909239Z","detectionStatus":"detected","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","roles":[],"tags":[],"imageFile":{"sha1":"5f1e8acedc065031aad553b710838eb366cfee9a","sha256":"8963a19fb992ad9a76576c5638fd68292cffb9aaac29eb8285f9abf6196a7dec","fileName":"MsSense.exe","filePath":"C:\\Program Files\\temp","fileSize":6136392,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"parentProcessImageFile":{"sha1":null,"sha256":null,"fileName":"services.exe","filePath":"C:\\Windows\\System32","fileSize":731744,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"userAccount":{"accountName":"SYSTEM","domainName":"NT AUTHORITY","userSid":"S-1-5-18","azureAdUserId":null,"userPrincipalName":null}},{"@odata.type":"#microsoft.graph.security.registryKeyEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"registryKey":"SYSTEM\\CONTROLSET001\\CONTROL\\WMI\\AUTOLOGGER\\SENSEAUDITLOGGER","registryHive":"HKEY_LOCAL_MACHINE","roles":[],"tags":[]}]}]}]}
   - path: /api/incidents
     methods: [GET]
     query_params:

packages/m365_defender/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.4.0"
+  changes:
+    - description: Add New Incident Data Stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/4435
 - version: "1.3.0"
   changes:
     - description: Update package to ECS 8.5.0.

packages/m365_defender/data_stream/incident/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,4 @@
+fields:
+  tags:
+    - preserve_original_event
+    - preserve_duplicate_custom_fields

packages/m365_defender/data_stream/incident/_dev/test/pipeline/test-incident.log

@@ -0,0 +1 @@
+{"@odata.type":"#microsoft.graph.security.incident","id":"2972395","incidentWebUrl":"https://security.microsoft.com/incidents/2972395?tid=12f988bf-16f1-11af-11ab-1d7cd011db47","redirectIncidentId":null,"tenantId":"b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","displayName":"Multi-stage incident involving Initial access & Command and control on multiple endpoints reported by multiple sources","createdDateTime":"2021-08-13T08:43:35.5533333Z","lastUpdateDateTime":"2021-09-30T09:35:45.1133333Z","assignedTo":"[email protected]","classification":"truePositive","determination":"multiStagedAttack","status":"active","severity":"medium","tags":["Demo"],"comments":[{"comment":"Demo incident","createdBy":"[email protected]","createdTime":"2021-09-30T12:07:37.2756993Z"}],"alerts":{"@odata.type":"#microsoft.graph.security.alert","id":"da637551227677560813_-961444813","providerAlertId":"da637551227677560813_-961444813","incidentId":"28282","status":"new","severity":"low","classification":"unknown","determination":"unknown","serviceSource":"microsoftDefenderForEndpoint","detectionSource":"antivirus","detectorId":"e0da400f-affd-43ef-b1d5-afc2eb6f2756","tenantId":"b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","title":"Suspicious execution of hidden file","description":"A hidden file has been launched. This activity could indicate a compromised host. Attackers often hide files associated with malicious tools to evade file system inspection and defenses.","recommendedActions":"Collect artifacts and determine scope\n�\tReview the machine timeline for suspicious activities that may have occurred before and after the time of the alert, and record additional related artifacts (files, IPs/URLs) \n�\tLook for the presence of relevant artifacts on other systems. Identify commonalities and differences between potentially compromised systems.\n�\tSubmit relevant files for deep analysis and review resulting detailed behavioral information.\n�\tSubmit undetected files to the MMPC malware portal\n\nInitiate containment & mitigation \n�\tContact the user to verify intent and initiate local remediation actions as needed.\n�\tUpdate AV signatures and run a full scan. The scan might reveal and remove previously-undetected malware components.\n�\tEnsure that the machine has the latest security updates. In particular, ensure that you have installed the latest software, web browser, and Operating System versions.\n�\tIf credential theft is suspected, reset all relevant users passwords.\n�\tBlock communication with relevant URLs or IPs at the organization�s perimeter.","category":"DefenseEvasion","assignedTo":null,"alertWebUrl":"https://security.microsoft.com/alerts/da637551227677560813_-961444813?tid=b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","incidentWebUrl":"https://security.microsoft.com/incidents/28282?tid=b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","actorDisplayName":null,"threatDisplayName":null,"threatFamilyName":null,"mitreTechniques":["T1564.001"],"createdDateTime":"2021-04-27T12:19:27.7211305Z","lastUpdateDateTime":"2021-05-02T14:19:01.3266667Z","resolvedDateTime":null,"firstActivityDateTime":"2021-04-26T07:45:50.116Z","lastActivityDateTime":"2021-05-02T07:56:58.222Z","comments":[],"evidence":[{"@odata.type":"#microsoft.graph.security.deviceEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"firstSeenDateTime":"2020-09-12T07:28:32.4321753Z","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","azureAdDeviceId":null,"deviceDnsName":"tempDns","osPlatform":"Windows10","osBuild":22424,"version":"Other","healthStatus":"active","riskScore":"medium","rbacGroupId":75,"rbacGroupName":"UnassignedGroup","onboardingStatus":"onboarded","defenderAvStatus":"unknown","loggedOnUsers":[],"roles":["compromised"],"tags":["Test Machine"],"vmMetadata":{"vmId":"ca1b0d41-5a3b-4d95-b48b-f220aed11d78","cloudProvider":"azure","resourceId":"/subscriptions/8700d3a3-3bb7-4fbe-a090-488a1ad04161/resourceGroups/WdatpApi-EUS-STG/providers/Microsoft.Compute/virtualMachines/NirLaviTests","subscriptionId":"8700d3a3-3bb7-4fbe-a090-488a1ad04161"}},{"@odata.type":"#microsoft.graph.security.fileEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"detectionStatus":"detected","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","roles":[],"tags":[],"fileDetails":{"sha1":"5f1e8acedc065031aad553b710838eb366cfee9a","sha256":"8963a19fb992ad9a76576c5638fd68292cffb9aaac29eb8285f9abf6196a7dec","fileName":"MsSense.exe","filePath":"C:\\Program Files\\temp","fileSize":6136392,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null}},{"@odata.type":"#microsoft.graph.security.processEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"processId":4780,"parentProcessId":668,"processCommandLine":"\"MsSense.exe\"","processCreationDateTime":"2021-08-12T12:43:19.0772577Z","parentProcessCreationDateTime":"2021-08-12T07:39:09.0909239Z","detectionStatus":"detected","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","roles":[],"tags":[],"imageFile":{"sha1":"5f1e8acedc065031aad553b710838eb366cfee9a","sha256":"8963a19fb992ad9a76576c5638fd68292cffb9aaac29eb8285f9abf6196a7dec","fileName":"MsSense.exe","filePath":"C:\\Program Files\\temp","fileSize":6136392,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"parentProcessImageFile":{"sha1":null,"sha256":null,"fileName":"services.exe","filePath":"C:\\Windows\\System32","fileSize":731744,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"userAccount":{"accountName":"SYSTEM","domainName":"NT AUTHORITY","userSid":"S-1-5-18","azureAdUserId":null,"userPrincipalName":null}},{"@odata.type":"#microsoft.graph.security.registryKeyEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"registryKey":"SYSTEM\\CONTROLSET001\\CONTROL\\WMI\\AUTOLOGGER\\SENSEAUDITLOGGER","registryHive":"HKEY_LOCAL_MACHINE","roles":[],"tags":[]}]}}