Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Log Type JSON format #465

Merged
merged 15 commits into from
Jul 5, 2023

Conversation

petardz
Copy link
Contributor

@petardz petardz commented Jun 20, 2023

Description

  1. Removed converting "." to "_" when parsing Sigma rules
  2. Introduced new Log Type JSON format to replace existing fieldmappings.yml/mappings.json files
  3. Added loading of log types from new Log Type JSON files
  4. Replaced usage of fieldmappings.yml with LogType loaded from new JSON format

Issues Resolved

#459

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Petar Dzepina <[email protected]>
petardz and others added 7 commits June 20, 2023 02:07
Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: Petar Dzepina <[email protected]>
@codecov
Copy link

codecov bot commented Jun 21, 2023

Codecov Report

Merging #465 (2cfcb39) into main (d14e764) will increase coverage by 0.02%.
The diff coverage is 30.13%.

@@             Coverage Diff              @@
##               main     #465      +/-   ##
============================================
+ Coverage     28.21%   28.23%   +0.02%     
- Complexity      897      906       +9     
============================================
  Files           231      235       +4     
  Lines          9326     9443     +117     
  Branches       1074     1081       +7     
============================================
+ Hits           2631     2666      +35     
- Misses         6459     6545      +86     
+ Partials        236      232       -4     
Impacted Files Coverage Δ
...rch/securityanalytics/SecurityAnalyticsPlugin.java 0.00% <0.00%> (ø)
...ecurityanalytics/logtype/BuiltinLogTypeLoader.java 0.00% <0.00%> (ø)
...arch/securityanalytics/logtype/LogTypeService.java 0.00% <0.00%> (ø)
...lytics/transport/TransportIndexDetectorAction.java 0.00% <0.00%> (ø)
...yanalytics/transport/TransportIndexRuleAction.java 0.00% <0.00%> (ø)
...g/opensearch/securityanalytics/util/FileUtils.java 0.00% <0.00%> (ø)
...opensearch/securityanalytics/util/RuleIndices.java 0.00% <0.00%> (ø)
...rg/opensearch/securityanalytics/model/LogType.java 71.42% <71.42%> (ø)
...ecurityanalytics/rules/backend/OSQueryBackend.java 67.24% <100.00%> (+0.86%) ⬆️
.../securityanalytics/rules/backend/QueryBackend.java 64.22% <100.00%> (+0.11%) ⬆️

sbcd90 and others added 2 commits June 21, 2023 22:48
Signed-off-by: Subhobrata Dey <[email protected]>

for(Mapping m : mappings) {
out.writeString(m.getRawField());
out.writeString(m.getEcs());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use out.writeCollection method here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


public class LogType implements Writeable, ToXContentObject {

private static final String ID = "id";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

for(String category: ruleCategories){
queryBackendMap.put(category, new OSQueryBackend(category, true, true));
for(String category: ruleCategories) {
Map<String, String> fieldMappings = logTypeService.getRuleFieldMappings(category);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getRuleFieldMappings internally calls getAllLogTypes from BuiltinLogTypeLoader which internally calls loadBuiltinLogTypes.
Why do we need to load all log types & then filter for particular category in memory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actual loading of logTypes from disk will be done only 1 time. I replaced getAllLogTypes with BuiltinLogTypeLoader.getLogTypeByName(logType) existing method.

petardz added 3 commits June 23, 2023 00:48
Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: Petar Dzepina <[email protected]>
sbcd90
sbcd90 previously approved these changes Jun 23, 2023
Signed-off-by: Petar Dzepina <[email protected]>
@sbcd90 sbcd90 merged commit faccf6c into opensearch-project:main Jul 5, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 5, 2023
* New Log Type JSON format
Signed-off-by: Petar Dzepina <[email protected]>
Co-authored-by: Subhobrata Dey <[email protected]>
(cherry picked from commit faccf6c)
sbcd90 pushed a commit that referenced this pull request Jul 11, 2023
* New Log Type JSON format
Signed-off-by: Petar Dzepina <[email protected]>
Co-authored-by: Subhobrata Dey <[email protected]>
(cherry picked from commit faccf6c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants