[BUG] winlog.event_id alias field for windows log type has duplicate mappings which returns incorrect log index field in mapping view API #786
Assignees
Labels
bug
Something isn't working
Comments
Merged
5 tasks
jowg-amazon
added a commit
that referenced
this issue
Mar 7, 2024
… mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
opensearch-trigger-bot bot
pushed a commit
that referenced
this issue
Mar 7, 2024
… mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 689760e)
opensearch-trigger-bot bot
pushed a commit
that referenced
this issue
Mar 7, 2024
… mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 689760e)
jowg-amazon
added a commit
to jowg-amazon/security-analytics
that referenced
this issue
Mar 8, 2024
… mapping view API (opensearch-project#786) (opensearch-project#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
AWSHurneyt
pushed a commit
that referenced
this issue
Mar 8, 2024
… mapping view API (#786) (#788) (#898) * field mapping changes * add integ test * turn unmappedfieldaliases as set and add integ test * add comments * fix integ tests * moved logic to method for better readability --------- Signed-off-by: Joanne Wang <[email protected]>
sbcd90
pushed a commit
to sbcd90/security-analytics
that referenced
this issue
Mar 10, 2024
Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 4d4f5e3) Co-authored-by: Joanne Wang <[email protected]> Reduce log level for informative message (opensearch-project#203) (opensearch-project#833) Signed-off-by: Enrico Tröger <[email protected]> Co-authored-by: Enrico Tröger <[email protected]> Updated alert creation following common-utils PR 584. (opensearch-project#837) (opensearch-project#839) Signed-off-by: AWSHurneyt <[email protected]> (cherry picked from commit 8adb9c3) Co-authored-by: AWSHurneyt <[email protected]> Release notes for 2.12.0 (opensearch-project#834) (opensearch-project#841) * release notes for 2.12 Signed-off-by: Joanne Wang <[email protected]> * update release notes Signed-off-by: Joanne Wang <[email protected]> * update release notes Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 414484a) Co-authored-by: Joanne Wang <[email protected]> Remove blocking calls and change threat intel feed flow to event driven (opensearch-project#871) (opensearch-project#876) * remove actionGet() and change threat intel feed flow to event driven Signed-off-by: Surya Sashank Nistala <[email protected]> * fix javadocs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert try catch removals Signed-off-by: Surya Sashank Nistala <[email protected]> * use action listener wrap() in detector threat intel code paths Signed-off-by: Surya Sashank Nistala <[email protected]> * add try catch Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> (cherry picked from commit 172d58d) Co-authored-by: Surya Sashank Nistala <[email protected]> Fail the flow the when detectot type is missing in the log types index (opensearch-project#845) (opensearch-project#857) Signed-off-by: Megha Goyal <[email protected]> (cherry picked from commit 8d19912) Co-authored-by: Megha Goyal <[email protected]> [BUG] ArrayIndexOutOfBoundsException for inconsistent detector index behavior (opensearch-project#843) (opensearch-project#858) * Catch ArrayIndexOutOfBoundsException when detector is missing Signed-off-by: Megha Goyal <[email protected]> * Add a check on SearchHits.getHits() length Signed-off-by: Megha Goyal <[email protected]> * Remove index out of bounds exception Signed-off-by: Megha Goyal <[email protected]> --------- Signed-off-by: Megha Goyal <[email protected]> (cherry picked from commit 0ef8543) Co-authored-by: Megha Goyal <[email protected]> Backport opensearch-project#873 and opensearch-project#789 (opensearch-project#895) * support object fields in aggregation based sigma rules (opensearch-project#789) Signed-off-by: Subhobrata Dey <[email protected]> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (opensearch-project#786) (opensearch-project#788) (opensearch-project#898) * field mapping changes * add integ test * turn unmappedfieldaliases as set and add integ test * add comments * fix integ tests * moved logic to method for better readability --------- Signed-off-by: Joanne Wang <[email protected]> Add throw for empty strings in rules with modifier contains, startwith, and endswith (opensearch-project#860) (opensearch-project#896) * add validation for empty strings with contains, startswith and endswith modifiers * throw exception if empty string with contains, startswith, or endswith * change var name * add modifiers to log --------- Signed-off-by: Joanne Wang <[email protected]> Add an "exists" check for "not" condition in sigma rules (opensearch-project#852) (opensearch-project#897) * test design Signed-off-by: Joanne Wang <[email protected]> * working version Signed-off-by: Joanne Wang <[email protected]> * cleaning up Signed-off-by: Joanne Wang <[email protected]> * testing Signed-off-by: Joanne Wang <[email protected]> * working version Signed-off-by: Joanne Wang <[email protected]> * working version Signed-off-by: Joanne Wang <[email protected]> * refactored querybackend Signed-off-by: Joanne Wang <[email protected]> * working on tests Signed-off-by: Joanne Wang <[email protected]> * fixed alerting and finding tests Signed-off-by: Joanne Wang <[email protected]> * fix correlation tests Signed-off-by: Joanne Wang <[email protected]> * working all tests Signed-off-by: Joanne Wang <[email protected]> * moved test and changed alias for adldap Signed-off-by: Joanne Wang <[email protected]> * added more tests Signed-off-by: Joanne Wang <[email protected]> * cleanup code Signed-off-by: Joanne Wang <[email protected]> * remove exists flag Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 656a5fe) Co-authored-by: Joanne Wang <[email protected]> Add goyamegh as a maintainer (opensearch-project#868) (opensearch-project#899) Signed-off-by: Megha Goyal <[email protected]> Refactor invocation of Action listeners in correlations (opensearch-project#880) (opensearch-project#900) * Refactor invocation of Action listeners in correlations * Close hanging tasks in correlations workflow * Logging finding id and monitor id in error logs --------- Signed-off-by: Megha Goyal <[email protected]> Add search request timeouts for correlations workflows (opensearch-project#893) (opensearch-project#901) * Reinstating more leaks plugged-in for correlations workflows Signed-off-by: Megha Goyal <[email protected]> * Add search timeouts to all correlation searches Signed-off-by: Megha Goyal <[email protected]> * Fix logging and exception messages Signed-off-by: Megha Goyal <[email protected]> * Change search timeout to 30 seconds Signed-off-by: Megha Goyal <[email protected]> --------- Signed-off-by: Megha Goyal <[email protected]> (cherry picked from commit 75c4429) Co-authored-by: Megha Goyal <[email protected]>
eirsep
pushed a commit
to eirsep/security-analytics
that referenced
this issue
Mar 13, 2024
… mapping view API (opensearch-project#786) (opensearch-project#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
jowg-amazon
added a commit
to jowg-amazon/security-analytics
that referenced
this issue
Mar 14, 2024
… mapping view API (opensearch-project#786) (opensearch-project#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
jowg-amazon
added a commit
to jowg-amazon/security-analytics
that referenced
this issue
Mar 14, 2024
… mapping view API (opensearch-project#786) (opensearch-project#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
5 tasks
jowg-amazon
added a commit
to jowg-amazon/security-analytics
that referenced
this issue
Mar 15, 2024
… mapping view API (opensearch-project#786) (opensearch-project#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
jowg-amazon
added a commit
that referenced
this issue
Mar 15, 2024
* support object fields in aggregation based sigma rules (#789) Signed-off-by: Subhobrata Dey <[email protected]> * Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> * Fix get mappings view API incorrectly returning ecs path (#867) * add logic and integ tests to not add duplicate to log-types-config index Signed-off-by: Joanne Wang <[email protected]> * change naming for existingFieldMapping and change contains to equals Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> * fix integ test (#918) Signed-off-by: Joanne Wang <[email protected]> * get all findings as part of findings API enhancement (#803) * get all findings as part of findings API enhancement Signed-off-by: Riya Saxena <[email protected]> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <[email protected]> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <[email protected]> * added support for param in Finding API Signed-off-by: Riya Saxena <[email protected]> * added detectionType as param for Findings API enhancements Signed-off-by: Riya Saxena <[email protected]> * added few tests to validate findings by params Signed-off-by: Riya Saxena <[email protected]> * added test for searchString param in FindingsAPI Signed-off-by: Riya Saxena <[email protected]> * adding addiional params findingIds, startTime and endTime as findings API enhancement Signed-off-by: Riya Saxena <[email protected]> * added params in getFindingsByDetectorId func * changed the startTime and endTime req input format * fix merge conflixt * fix integ test failures in findings API * fix integ tests * fix integ tests for findings Signed-off-by: Subhobrata Dey <[email protected]> --------- Signed-off-by: Riya Saxena <[email protected]> Signed-off-by: Riya <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> * Feature findings api enhancements (#914) * get all findings as part of findings API enhancement Signed-off-by: Riya Saxena <[email protected]> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <[email protected]> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <[email protected]> * added support for param in Finding API Signed-off-by: Riya Saxena <[email protected]> * added detectionType as param for Findings API enhancements Signed-off-by: Riya Saxena <[email protected]> * added few tests to validate findings by params Signed-off-by: Riya Saxena <[email protected]> * added test for searchString param in FindingsAPI Signed-off-by: Riya Saxena <[email protected]> * adding addiional params findingIds, startTime and endTime as findings API enhancement Signed-off-by: Riya Saxena <[email protected]> * added params in getFindingsByDetectorId func * changed the startTime and endTime req input format * fix merge conflixt * fix integ test failures in findings API * fix integ tests * refactored the logic Signed-off-by: Riya Saxena <[email protected]> * remove unused imports * address the pr comments Signed-off-by: Riya Saxena <[email protected]> * address pr comments Signed-off-by: Riya Saxena <[email protected]> * SA integ tests fix * SA integ tests fix * fix integ tests for findings Signed-off-by: Subhobrata Dey <[email protected]> * fix conflixt errors Signed-off-by: Riya Saxena <[email protected]> * fix conflixt errors Signed-off-by: Riya Saxena <[email protected]> * fix conflixt errors Signed-off-by: Riya Saxena <[email protected]> * fix conflixt errors Signed-off-by: Riya Saxena <[email protected]> * fix integ tests Signed-off-by: Riya Saxena <[email protected]> * fix integ tests Signed-off-by: Riya Saxena <[email protected]> * fix integ tests Signed-off-by: Riya Saxena <[email protected]> * fix flaky integ tests Signed-off-by: Riya Saxena <[email protected]> * address pr comments Signed-off-by: Riya Saxena <[email protected]> --------- Signed-off-by: Riya Saxena <[email protected]> Signed-off-by: Riya <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> * fix findings api integ tests Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: Joanne Wang <[email protected]> Signed-off-by: Riya Saxena <[email protected]> Signed-off-by: Riya <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: Riya <[email protected]>
praveensameneni
pushed a commit
that referenced
this issue
Mar 16, 2024
* Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> * support object fields in aggregation based sigma rules (#789) Signed-off-by: Subhobrata Dey <[email protected]> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * version bump Signed-off-by: Surya Sashank Nistala <[email protected]> * AggregationBackendTests fix Signed-off-by: Riya Saxena <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: Riya Saxena <[email protected]> Co-authored-by: Joanne Wang <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: Surya Sashank Nistala <[email protected]>
riysaxen-amzn
pushed a commit
that referenced
this issue
Mar 18, 2024
… mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
riysaxen-amzn
pushed a commit
that referenced
this issue
Mar 18, 2024
… mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]>
riysaxen-amzn
added a commit
that referenced
this issue
Mar 18, 2024
* Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> * Fix get mappings view API incorrectly returning ecs path (#867) * add logic and integ tests to not add duplicate to log-types-config index Signed-off-by: Joanne Wang <[email protected]> * change naming for existingFieldMapping and change contains to equals Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> * fix integ test (#918) Signed-off-by: Joanne Wang <[email protected]> * fix detector writeTo() method missing fields (#695) * fix detector writeTo() method missing fields Signed-off-by: Surya Sashank Nistala <[email protected]> * fix test Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * removing threatIntel related code Signed-off-by: Riya Saxena <[email protected]> * removing threatIntel related code Signed-off-by: Riya Saxena <[email protected]> * ignore flaky tests Signed-off-by: Riya Saxena <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: Riya Saxena <[email protected]> Co-authored-by: Joanne Wang <[email protected]> Co-authored-by: Surya Sashank Nistala <[email protected]>
engechas
pushed a commit
that referenced
this issue
Apr 10, 2024
… mapping view API (#786) (#788) (#890) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 689760e) Co-authored-by: Joanne Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the bug?
winlog.event_idecs field has three different raw fields associated with it which can lead to incorrect field being returned by the mappings/view API. This causes an issue when user tries to create a detector with rule(s) that require mapping between the ecs field to a log index field.For e.g. if the log index has field
EventID, then the mapping view API should be able to automatically mapwinlog.event_idtoEventIDand send that mapping as part of the response but it ends up sending mapping betweenwinlog.event_idtoevent_uidor some other field not present in the index.How can one reproduce the bug?
Steps to reproduce the behavior:
EventIDfieldwinlog.event_idis not mapped toEventIDWhat is the expected behavior?
winlog.event_idshould be mapped to the field present in the log index.What is your host/environment?
Security analytics 2.11
The text was updated successfully, but these errors were encountered: