Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add category to custom log types #634

Merged
merged 2 commits into from
Oct 4, 2023
Merged

add category to custom log types #634

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
849ad58
add category to custom log types
sbcd90 Oct 4, 2023
90cd502
make category field optional
sbcd90 Oct 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/main/java/org/opensearch/securityanalytics/action/IndexCustomLogTypeRequest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@
if (!find) {
throw new ActionRequestValidationException();
}
String category = customLogType.getCategory();

Check warning on line 64 in src/main/java/org/opensearch/securityanalytics/action/IndexCustomLogTypeRequest.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/action/IndexCustomLogTypeRequest.java#L64 

Added line #L64 was not covered by tests
if (!CustomLogType.VALID_LOG_CATEGORIES.contains(category)) {
throw new ActionRequestValidationException();

Check warning on line 66 in src/main/java/org/opensearch/securityanalytics/action/IndexCustomLogTypeRequest.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/action/IndexCustomLogTypeRequest.java#L66 

Added line #L66 was not covered by tests
}
return null;
}

Expand Down
31 changes: 30 additions & 1 deletion src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import org.opensearch.core.xcontent.XContentParserUtils;

import java.io.IOException;
import java.util.List;
import java.util.Map;

import static org.opensearch.securityanalytics.action.IndexCustomLogTypeResponse.CUSTOM_LOG_TYPES_FIELD;
Expand All @@ -27,11 +28,23 @@

private static final Logger log = LogManager.getLogger(CustomLogType.class);

public static final List<String> VALID_LOG_CATEGORIES = List.of(

Check warning on line 31 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L31 

Added line #L31 was not covered by tests
"Access Management",
"Applications",
"Cloud Services",
"Network Activity",
"Security",
"System Activity",
"Other"
);

public static final String CUSTOM_LOG_TYPE_ID_FIELD = "custom_logtype_id";

private static final String NAME_FIELD = "name";

private static final String DESCRIPTION_FIELD = "description";

private static final String CATEGORY_FIELD = "category";
private static final String SOURCE_FIELD = "source";

private static final String TAGS_FIELD = "tags";
Expand All @@ -44,6 +57,8 @@

private String description;

private String category;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not an optional parameter, we need this to have a default value for users upgrading the cluster or this will cause issues for them.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a default value for category field.


private String source;

private Map<String, Object> tags;
Expand All @@ -58,12 +73,14 @@
Long version,
String name,
String description,
String category,
String source,
Map<String, Object> tags) {
this.id = id != null ? id : NO_ID;
this.version = version != null ? version : NO_VERSION;
this.name = name;
this.description = description;
this.category = category;

Check warning on line 83 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L83 

Added line #L83 was not covered by tests
this.source = source;
this.tags = tags;
}
Expand All @@ -75,6 +92,7 @@
sin.readString(),
sin.readString(),
sin.readString(),
sin.readString(),

Check warning on line 95 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L95 

Added line #L95 was not covered by tests
sin.readMap()
);
}
Expand All @@ -86,6 +104,7 @@
null,
input.get(NAME_FIELD).toString(),
input.get(DESCRIPTION_FIELD).toString(),
input.get(CATEGORY_FIELD).toString(),

Check warning on line 107 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L107 

Added line #L107 was not covered by tests
input.get(SOURCE_FIELD).toString(),
(Map<String, Object>) input.get(TAGS_FIELD)
);
Expand All @@ -97,6 +116,7 @@
out.writeLong(version);
out.writeString(name);
out.writeString(description);
out.writeString(category);

Check warning on line 119 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L119 

Added line #L119 was not covered by tests
out.writeString(source);
out.writeMap(tags);
}
Expand All @@ -106,6 +126,7 @@
return builder.startObject()
.field(NAME_FIELD, name)
.field(DESCRIPTION_FIELD, description)
.field(CATEGORY_FIELD, category)

Check warning on line 129 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L129 

Added line #L129 was not covered by tests
.field(SOURCE_FIELD, source)
.field(TAGS_FIELD, tags)
.endObject();
Expand All @@ -121,6 +142,7 @@

String name = null;
String description = null;
String category = null;

Check warning on line 145 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L145 

Added line #L145 was not covered by tests
String source = null;
Map<String, Object> tags = null;

Expand All @@ -136,6 +158,9 @@
case DESCRIPTION_FIELD:
description = xcp.text();
break;
case CATEGORY_FIELD:
category = xcp.text();
break;

Check warning on line 163 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L162-L163 

Added lines #L162 - L163 were not covered by tests
case SOURCE_FIELD:
source = xcp.text();
break;
Expand All @@ -146,7 +171,7 @@
xcp.skipChildren();
}
}
return new CustomLogType(id, version, name, description, source, tags);
return new CustomLogType(id, version, name, description, category, source, tags);

Check warning on line 174 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L174 

Added line #L174 was not covered by tests
}

public static CustomLogType readFrom(StreamInput sin) throws IOException {
Expand Down Expand Up @@ -177,6 +202,10 @@
return description;
}

public String getCategory() {
return category;

Check warning on line 206 in src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/model/CustomLogType.java#L206 

Added line #L206 was not covered by tests
}

public String getSource() {
return source;
}
Expand Down
24 changes: 23 additions & 1 deletion src/main/resources/OSMapping/logtypes.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
"others_application": {
"name": "others_application",
"description": "Application logs",
"category": "Other",
"source": "Sigma",
"tags": {
"correlation_id": 0
Expand All @@ -10,6 +11,7 @@
"others_apt": {
"name": "others_apt",
"description": "Apt logs",
"category": "Other",
"source": "Sigma",
"tags": {
"correlation_id": 1
Expand All @@ -18,6 +20,7 @@
"others_cloud": {
"name": "others_cloud",
"description": "Cloud logs",
"category": "Other",
"source": "Sigma",
"tags": {
"correlation_id": 2
Expand All @@ -26,6 +29,7 @@
"others_compliance": {
"name": "others_compliance",
"description": "Compliance logs",
"category": "Other",
"source": "Sigma",
"tags": {
"correlation_id": 4
Expand All @@ -34,6 +38,7 @@
"linux": {
"name": "linux",
"description": "Sys logs",
"category": "System Activity",
"source": "Sigma",
"tags": {
"correlation_id": 5
Expand All @@ -42,6 +47,7 @@
"others_macos": {
"name": "others_macos",
"description": "MacOS logs",
"category": "System Activity",
"source": "Sigma",
"tags": {
"correlation_id": 6
Expand All @@ -50,6 +56,7 @@
"network": {
"name": "network",
"description": "Network logs",
"category": "Network Activity",
"source": "Sigma",
"tags": {
"correlation_id": 7
Expand All @@ -58,6 +65,7 @@
"others_proxy": {
"name": "others_proxy",
"description": "Proxy logs",
"category": "Other",
"source": "Sigma",
"tags": {
"correlation_id": 8
Expand All @@ -66,6 +74,7 @@
"others_web": {
"name": "others_web",
"description": "Web logs",
"category": "Other",
"source": "Sigma",
"tags": {
"correlation_id": 9
Expand All @@ -74,6 +83,7 @@
"windows": {
"name": "windows",
"description": "Windows logs",
"category": "System Activity",
"source": "Sigma",
"tags": {
"correlation_id": 10
Expand All @@ -82,14 +92,16 @@
"ad_ldap": {
"name": "ad_ldap",
"description": "Ad/ldap logs",
"category": "Access Management",
"source": "Sigma",
"tags": {
"correlation_id": 11
}
},
"apache_access": {
"name": "apache_access",
"description": "Apt logs",
"description": "Apache Access logs",
"category": "Access Management",
"source": "Sigma",
"tags": {
"correlation_id": 12
Expand All @@ -98,6 +110,7 @@
"cloudtrail": {
"name": "cloudtrail",
"description": "Cloudtrail Raw or OCSF based logs",
"category": "Cloud Services",
"source": "Sigma",
"tags": {
"correlation_id": 14
Expand All @@ -106,6 +119,7 @@
"dns": {
"name": "dns",
"description": "DNS Raw or Route53 OCSF based logs",
"category": "Network Activity",
"source": "Sigma",
"tags": {
"correlation_id": 15
Expand All @@ -114,6 +128,7 @@
"github": {
"name": "github",
"description": "Github logs",
"category": "Applications",
"source": "Sigma",
"tags": {
"correlation_id": 16
Expand All @@ -122,6 +137,7 @@
"m365": {
"name": "m365",
"description": "M365 logs",
"category": "Applications",
"source": "Sigma",
"tags": {
"correlation_id": 17
Expand All @@ -130,6 +146,7 @@
"gworkspace": {
"name": "gworkspace",
"description": "GWorkspace logs",
"category": "Applications",
"source": "Sigma",
"tags": {
"correlation_id": 18
Expand All @@ -138,6 +155,7 @@
"okta": {
"name": "okta",
"description": "Okta logs",
"category": "Access Management",
"source": "Sigma",
"tags": {
"correlation_id": 19
Expand All @@ -146,6 +164,7 @@
"azure": {
"name": "azure",
"description": "Azure logs",
"category": "Cloud Services",
"source": "Sigma",
"tags": {
"correlation_id": 20
Expand All @@ -154,6 +173,7 @@
"s3": {
"name": "s3",
"description": "S3 logs",
"category": "Cloud Services",
"source": "Sigma",
"tags": {
"correlation_id": 21
Expand All @@ -162,6 +182,7 @@
"test_windows": {
"name": "test_windows",
"description": "Test Windows Log Type for integ tests. Please do not use.",
"category": "Other",
"source": "Sigma",
"tags": {
"correlation_id": 22
Expand All @@ -170,6 +191,7 @@
"vpcflow": {
"name": "vpcflow",
"description": "VPC Flow Raw or OCSF based logs",
"category": "Network Activity",
"source": "Sigma",
"tags": {
"correlation_id": 23
Expand Down
11 changes: 10 additions & 1 deletion src/main/resources/mappings/log_type_config_mapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"_meta" : {
"schema_version": 1
"schema_version": 2
},
"dynamic_templates": [
{
Expand Down Expand Up @@ -50,6 +50,15 @@
}
}
},
"category": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"source": {
"type": "text",
"fields": {
Expand Down
7 changes: 5 additions & 2 deletions src/test/java/org/opensearch/securityanalytics/TestHelpers.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,17 +147,20 @@ public static Detector randomDetector(String name,
return new Detector(null, null, name, enabled, schedule, lastUpdateTime, enabledTime, detectorType, user, inputs, triggers, Collections.singletonList(""), "", "", "", "", "", "", Collections.emptyMap(), Collections.emptyList());
}

public static CustomLogType randomCustomLogType(String name, String description, String source) {
public static CustomLogType randomCustomLogType(String name, String description, String category, String source) {
if (name == null) {
name = "custom-log-type";
}
if (description == null) {
description = "custom-log-type-desc";
}
if (category == null) {
category = "Other";
}
if (source == null) {
source = "Sigma";
}
return new CustomLogType(null, null, name, description, source, null);
return new CustomLogType(null, null, name, description, category, source, null);
}

public static Detector randomDetectorWithNoUser() {
Expand Down
Loading
Loading