[FEATURE] [META] Security Analytics OCSF integration

[sbcd90]

Security Analytics workflows based on log types

Logs based on raw log types

In order to run Detectors on raw logs, the fields of these raw log documents are first mapped to ECS fields using field aliases.

Logs based on ECS schema

Users can also ingest logs in ECS schema format. In this case, Detectors will not require any explicit mappings to match rules with logs.

Logs based on OCSF schema

In future, Security Analytics plan to allow users to ingest data in OCSF schema format. In this case, the fields of these log documents in OCSF format are mapped back to ECS fields again using field aliases.

ECS remains the base format for all log types for which detectors run & matches rules with logs.

Proposed Data Structure to store mappings

A flat table is used to store field mappings from raw log fields to ECS fields to OCSF fields. The Raw Log Types field serve as the primary key for this flat table.

This will help in switching the base log type from ECS to OCSF in future easily without any backward compatibility issues.

Support for custom log types

Security Analytics today only support 13 pre-defined log types. But in future, we extend Security Analytics to support custom log types.

Users can define their own custom mappings, custom Sigma Rules & use them to run Detectors on logs from a custom datasource.

Design

The Log Type object will be stored in an OpenSearch index named .opensearch-sap-log-types-config.
The mappings for the index will be defined as follows:

[{
  "name": "Cloudtrail",
  "description": "",
  "mappings": [
    {
      "raw_field": "userIdentity.arn",
      "ecs": "aws-cloudtrail-user_identity-arn",
      "ocsf": "actor.user.uuid"
    },
    {
      "raw_field": "",
      "ecs": "",
      "ocsf": ""
    }
  ]
}]

[getsaurabh02]

Proposing to add below Static Mapping Dictionary to start with (as resource file), which can be loaded by the plugin on startup for usage across the system during rule configuration and execution.

Sigma Rule Field              || Log Type  || Raw Log Field                               || ECS Field          || OCSF Field        || Agent 1 Schema           || Agent 2 Schema
AuthenticationPackageName     || Windows   || winlog.event_data.AuthenticationPackageName || <<ECS (optional)>> || OCSF <<optional>> || Agent-1-Field <optional> || Agent-1-Field <optional>

In addition to the mapping, the Custom Log Type interface will allow users to create additional mappings which will then persisted by the system. The combination of static (pre-defined) and user-defined mappings can be then used all together for rule execution.

[getsaurabh02]

Hashing out the OCSF Integration from the parent issue here #459