Add documentation for OCSF field mapping and correlation engine API #4549

cwillum · 2023-07-12T23:45:30Z

Description

Adds documentation for OCSF field mapping types, and adds APIs for correlation engine.

Issues Resolved

Fixes #4500

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: cwillum <[email protected]>

sbcd90 · 2023-07-13T20:15:54Z

the ocsf logs integration piece looks good but correlation engine api documentation is missing.

Signed-off-by: cwillum <[email protected]>

getsaurabh02 · 2023-07-13T20:54:58Z

_security-analytics/api-tools/correlation-eng.md

+| `query` | String | TBD. |
+| `category` | String | TBD. |


Can we provide description for TBDs here?

getsaurabh02 · 2023-07-13T20:56:56Z

_security-analytics/api-tools/correlation-eng.md

+| :--- | :--- |:--- |
+| `finding` | String | The finding ID. |
+| `detector_type` | String | The log type for the detector. |
+| `nearby_findings` | Number | TBD. |


Same as above

Signed-off-by: cwillum <[email protected]>

cwillum · 2023-07-13T21:16:17Z

@sbcd90 @getsaurabh02 Yes, I could use some help with a few field/parameter descriptions. Those will be conspicuous with the "TBD" placeholder. The last thing I need is a screenshot that shows the log type selectors with VPC_Flow (or however that is labeled). Thanks for looking.

Signed-off-by: cwillum <[email protected]>

Naarcha-AWS

One small style question. Otherwise, LGTM

_security-analytics/api-tools/rule-api.md

Signed-off-by: Subhobrata Dey <[email protected]>

Signed-off-by: cwillum <[email protected]>

This reverts commit c6db296. merge conflict with same edits on another branch#

Signed-off-by: cwillum <[email protected]>

cwillum · 2023-07-14T19:31:02Z

Merge conflicts from same edits on different branches.

This reverts commit 7a70dc3. merge conflict with second branch :wq

This reverts commit ef13a74.

Signed-off-by: cwillum <[email protected]>

natebower

@cwillum Please see my comments and changes and let me know if you have any questions. Thanks!

natebower · 2023-07-16T13:59:13Z

_security-analytics/api-tools/correlation-eng.md

+
+# Correlation engine APIs
+
+Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain span of time, and perform other tasks.


Suggested change

Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain span of time, and perform other tasks.

Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain time window, and perform other tasks.

_security-analytics/api-tools/correlation-eng.md

natebower · 2023-07-16T14:05:41Z

_security-analytics/api-tools/correlation-eng.md

+
+## List all findings and their correlations within a time window
+
+Provides a list of all findings and their correlations within a specified window of time.


Suggested change

Provides a list of all findings and their correlations within a specified window of time.

Provides a list of all findings and their correlations within a specified time window:

natebower · 2023-07-16T14:10:50Z

_security-analytics/api-tools/mappings-api.md

@@ -11,6 +11,17 @@ The following APIs can be used for a number of tasks related to mappings, from c

 ## Get Mappings View

+Returns a view of the fields contained in an index used as a log source.


Please rephrase as a complete sentence naming the noun (what is used to do this?).

natebower · 2023-07-16T14:11:46Z

_security-analytics/api-tools/rule-api.md

@@ -11,7 +11,7 @@ The following APIs can be used for a number of tasks related to rules, from sear

 ## Create Custom Rule

-The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information on how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
+The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).


Suggested change

The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).

The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see the information provided in [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).

natebower · 2023-07-16T14:13:14Z

_security-analytics/sec-analytics-config/detectors-config.md

@@ -60,6 +60,15 @@ Security Analytics takes advantage of prepackaged Sigma rules for security event

 Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, choose the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder.

+#### Amazon Security Lake logs
+
+[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).


Suggested change

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

natebower · 2023-07-16T14:16:24Z

_security-analytics/sec-analytics-config/detectors-config.md

+
+[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
+
+The log types from Security Lake that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, since logs such as CloudTrail can be captured conceivably in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.


Suggested change

The log types from Security Lake that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, since logs such as CloudTrail can be captured conceivably in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.

The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.

natebower · 2023-07-16T14:18:58Z

_security-analytics/api-tools/rule-api.md

@@ -11,7 +11,7 @@ The following APIs can be used for a number of tasks related to rules, from sear

 ## Create Custom Rule

-The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information on how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
+The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).


Confirm capitalization of the API name.

Signed-off-by: cwillum <[email protected]>

…search-project/documentation-website into fix#4500-sec-analytics-ocsf

Signed-off-by: cwillum <[email protected]>

…pensearch-project#4549) * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * add missing param descriptions (opensearch-project#4555) Signed-off-by: Subhobrata Dey <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit c6db296. merge conflict with same edits on another branch# * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit 7a70dc3. merge conflict with second branch :wq * Revert "fix#4500 ocsf fields and api" This reverts commit ef13a74. * fix#4500 edits post merge conflict Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> --------- Signed-off-by: cwillum <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]>

…4549) * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * add missing param descriptions (#4555) Signed-off-by: Subhobrata Dey <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit c6db296. merge conflict with same edits on another branch# * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit 7a70dc3. merge conflict with second branch :wq * Revert "fix#4500 ocsf fields and api" This reverts commit ef13a74. * fix#4500 edits post merge conflict Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> --------- Signed-off-by: cwillum <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]>

fix#4500 ocsf fields and api

7209299

Signed-off-by: cwillum <[email protected]>

cwillum added 2 - In progress Issue/PR: The issue or PR is in progress. release-notes PR: Include this PR in the automated release notes security-analytics v2.9.0 labels Jul 12, 2023

cwillum self-assigned this Jul 12, 2023

cwillum requested review from hdhalter, kolchfa-aws, Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners July 12, 2023 23:45

cwillum added 2 commits July 12, 2023 16:58

fix#4500 ocsf fields and api

1ca0dba

Signed-off-by: cwillum <[email protected]>

fix#4500 ocsf fields and api

d09d450

Signed-off-by: cwillum <[email protected]>

fix#4500 ocsf fields and api

68c5147

Signed-off-by: cwillum <[email protected]>

sbcd90 approved these changes Jul 13, 2023

View reviewed changes

getsaurabh02 approved these changes Jul 13, 2023

View reviewed changes

getsaurabh02 requested changes Jul 13, 2023

View reviewed changes

getsaurabh02 reviewed Jul 13, 2023

View reviewed changes

fix#4500 ocsf fields and api

ea63aa2

Signed-off-by: cwillum <[email protected]>

fix#4500 ocsf fields and api

ef13a74

Signed-off-by: cwillum <[email protected]>

cwillum added 3 - Tech review PR: Tech review in progress 4 - Doc review PR: Doc review in progress and removed 2 - In progress Issue/PR: The issue or PR is in progress. 3 - Tech review PR: Tech review in progress labels Jul 14, 2023

Naarcha-AWS approved these changes Jul 14, 2023

View reviewed changes

_security-analytics/api-tools/rule-api.md Outdated Show resolved Hide resolved

sbcd90 and others added 4 commits July 14, 2023 12:48

add missing param descriptions (#4555)

9f00119

Signed-off-by: Subhobrata Dey <[email protected]>

fix#4500 ocsf fields and api

c6db296

Signed-off-by: cwillum <[email protected]>

Revert "fix#4500 ocsf fields and api"

c0869cd

This reverts commit c6db296. merge conflict with same edits on another branch#

fix#4500 ocsf fields and api

7a70dc3

Signed-off-by: cwillum <[email protected]>

cwillum added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Jul 14, 2023

cwillum added 3 commits July 14, 2023 12:39

Revert "fix#4500 ocsf fields and api"

332a4a6

This reverts commit 7a70dc3. merge conflict with second branch :wq

Revert "fix#4500 ocsf fields and api"

2fb1bb6

This reverts commit ef13a74.

fix#4500 edits post merge conflict

4335661

Signed-off-by: cwillum <[email protected]>

natebower reviewed Jul 16, 2023

View reviewed changes

cwillum added 5 commits July 17, 2023 11:34

fix#4500 ocsf fields and api

e8e40de

Signed-off-by: cwillum <[email protected]>

Merge branch 'fix#4500-sec-analytics-ocsf' of https://github.com/open…

1b470c3

…search-project/documentation-website into fix#4500-sec-analytics-ocsf

fix#4500 ocsf fields and api

69fca8a

Signed-off-by: cwillum <[email protected]>

fix#4500 ocsf fields and api

18faa1a

Signed-off-by: cwillum <[email protected]>

fix#4500 ocsf fields and api

ff017a1

Signed-off-by: cwillum <[email protected]>

cwillum removed the 5 - Editorial review PR: Editorial review in progress label Jul 17, 2023

cwillum merged commit 49e8391 into main Jul 17, 2023

cwillum mentioned this pull request Jul 25, 2023

[DOC] Update API documentation for Correlation Engine in Security Analytics #4375

Closed

4 tasks

prudhvigodithi added release-notes PR: Include this PR in the automated release notes and removed release-notes PR: Include this PR in the automated release notes labels Oct 3, 2023

hdhalter deleted the fix#4500-sec-analytics-ocsf branch March 28, 2024 21:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation for OCSF field mapping and correlation engine API #4549

Add documentation for OCSF field mapping and correlation engine API #4549

cwillum commented Jul 12, 2023 •

edited

Loading

sbcd90 commented Jul 13, 2023

getsaurabh02 Jul 13, 2023

getsaurabh02 Jul 13, 2023

cwillum commented Jul 13, 2023

Naarcha-AWS left a comment

cwillum commented Jul 14, 2023

natebower left a comment

natebower Jul 16, 2023

natebower Jul 16, 2023

natebower Jul 16, 2023

natebower Jul 16, 2023

natebower Jul 16, 2023

natebower Jul 16, 2023

natebower Jul 16, 2023


		# Correlation engine APIs

		Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain span of time, and perform other tasks.


		## List all findings and their correlations within a time window

		Provides a list of all findings and their correlations within a specified window of time.

		@@ -11,6 +11,17 @@ The following APIs can be used for a number of tasks related to mappings, from c

		## Get Mappings View

		Returns a view of the fields contained in an index used as a log source.

	The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
	The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see the information provided in [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).


		[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

		The log types from Security Lake that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as DNS logs when [defining a detector](#step-1-define-a-detector). Furthermore, since logs such as CloudTrail can be captured conceivably in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.

Add documentation for OCSF field mapping and correlation engine API #4549

Add documentation for OCSF field mapping and correlation engine API #4549

Conversation

cwillum commented Jul 12, 2023 • edited Loading

Description

Issues Resolved

Checklist

sbcd90 commented Jul 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwillum commented Jul 13, 2023

Naarcha-AWS left a comment

Choose a reason for hiding this comment

cwillum commented Jul 14, 2023

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwillum commented Jul 12, 2023 •

edited

Loading