Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Error when using aggregation expression in sigma rules #458

Closed
dnhodgson opened this issue Jun 7, 2023 · 9 comments
Closed

[BUG] Error when using aggregation expression in sigma rules #458

dnhodgson opened this issue Jun 7, 2023 · 9 comments
Assignees
@sbcd90 @eirsep
Labels
bug Something isn't working

Comments

@dnhodgson
Copy link

What is the bug?
Error when using aggregation expression in sigma rules

When creating a detector with an aggregation rule I get the following error even when manually creating a timestamp alias in the security_analytics/mappings

POST /_plugins/_security_analytics/mappings
{
   "index_name": "logs",
   "rule_topic": "linux",
   "alias_mappings": {
        "properties": {
            "timestamp": {
                "type": "alias",
                "path": "@timestamp"
            }
        }
    }
}

[2023-06-07T14:17:46,125][ERROR][o.o.s.t.TransportIndexDetectorAction] [opensearch-node1] Unable to verify presence of timestamp alias for index [null] in detector [user auth 5]. Not setting time range filter for bucket level monitor.
org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: index_name is missing;
        at org.opensearch.action.ValidateActions.addValidationError(ValidateActions.java:44) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.securityanalytics.action.GetIndexMappingsRequest.validate(GetIndexMappingsRequest.java:40) ~[?:?]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:177) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:472) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:459) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.createBucketLevelMonitorRequest(TransportIndexDetectorAction.java:500) [opensearch-security-analytics-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.buildBucketLevelMonitorRequests(TransportIndexDetectorAction.java:462) [opensearch-security-analytics-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.createMonitorFromQueries(TransportIndexDetectorAction.java:239) [opensearch-security-analytics-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction$AsyncIndexDetectorsAction$10.onResponse(TransportIndexDetectorAction.java:1103) [opensearch-security-analytics-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.securityanalytics.transport.TransportIndexDetectorAction$AsyncIndexDetectorsAction$10.onResponse(TransportIndexDetectorAction.java:1080) [opensearch-security-analytics-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:401) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.ActionListener$5.onResponse(ActionListener.java:266) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:658) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:427) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:421) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:299) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:139) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:151) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.7.0.jar:2.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

When the job runs I get the following error

[2023-06-07T14:39:21,980][INFO ][o.o.a.c.s.JobScheduler   ] [opensearch-node3] Scheduling jobId : JURMlogBtN83KwnnPBAl, name: user auth 1
[2023-06-07T14:39:23,235][INFO ][o.o.s.t.SecureTransportAction] [opensearch-node3] User and roles string from thread context: admin|admin|own_index,all_access
[2023-06-07T14:40:18,489][INFO ][o.o.a.a.AlertIndices     ] [opensearch-node3] Index mapping of .opensearch-sap-linux-alerts is updated
[2023-06-07T14:40:18,522][INFO ][o.o.a.a.AlertIndices     ] [opensearch-node3] Index mapping of .opensearch-sap-linux-alerts-history-2023.06.07-1 is updated
[2023-06-07T14:40:18,536][INFO ][o.o.a.a.AlertIndices     ] [opensearch-node3] Index mapping of .opensearch-sap-linux-findings-2023.06.07-1 is updated
[2023-06-07T14:40:19,448][INFO ][o.o.a.InputService       ] [opensearch-node3] Error collecting inputs for monitor: JURMlogBtN83KwnnPBAl
org.opensearch.action.search.SearchPhaseExecutionException: 
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:128) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.7.0.jar:2.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.NullPointerException: value must not be null
        at org.opensearch.commons.alerting.aggregation.bucketselectorext.BucketSelectorExtAggregator.doReduce(BucketSelectorExtAggregator.kt:122) ~[?:?]
        at org.opensearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:228) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:565) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:536) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:153) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:137) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.7.0.jar:2.7.0]
        ... 8 more
[2023-06-07T14:40:19,448][INFO ][o.o.a.TriggerService     ] [opensearch-node3] Error running trigger [sS0elogBwTbYNvhJbrj6] for monitor [JURMlogBtN83KwnnPBAl]
java.lang.IndexOutOfBoundsException: Empty list doesn't contain element at index 0.
        at kotlin.collections.EmptyList.get(Collections.kt:36) ~[kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at kotlin.collections.EmptyList.get(Collections.kt:24) ~[kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at org.opensearch.alerting.TriggerService.runBucketLevelTrigger(TriggerService.kt:96) [opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.BucketLevelMonitorRunner.runMonitor(BucketLevelMonitorRunner.kt:135) [opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at org.opensearch.alerting.BucketLevelMonitorRunner$runMonitor$1.invokeSuspend(BucketLevelMonitorRunner.kt) [opensearch-alerting-2.7.0.0.jar:2.7.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at kotlinx.coroutines.ResumeModeKt.resumeUninterceptedMode(ResumeMode.kt:46) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.internal.ScopeCoroutine.onCompletionInternal$kotlinx_coroutines_core(Scopes.kt:28) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:305) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.JobSupport.tryFinalizeSimpleState(JobSupport.kt:264) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:762) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:117) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]

How can one reproduce the bug?

1) Create an index called "logs"
PUT logs
{
  "mappings": {
    "properties": {
      "user": {
        "type": "keyword"
      },
      "result": {
        "type": "keyword"
      }
    }
  }
}
2) Create a detection rule as follows:
id: 25b9c01c-350d-4b95-bed1-836d04a4f324
logsource:
  product: linux
title: User auth failed count
description: test
tags: []
falsepositives:
  - ''
level: medium
status: experimental
references: []
author: Darcy Hodgson
detection:
  timespan: 5m
  selection:
    result: Failed
  condition:
    - selection | count(result) by user > 2
3) Create a detector for the logs index with this rule enabled.
4) Start creating simple documents like the following example
{
    "user": "joe",
    "result": "Failed",
    "@timestamp": "2023-06-07T14:46:18Z"
}

What is the expected behavior?
detector should count that there are events matching the rule and create some findings

What is your host/environment?

  • OS: Rocky
  • Version 9
  • Opensearch 2.7.0
  • Opensearch-dashboards 2.7.0
@dnhodgson dnhodgson added bug Something isn't working untriaged labels Jun 7, 2023
@petardz
Copy link
Contributor

petardz commented Jun 20, 2023 •
edited
Loading

@dnhodgson Can you please share your Detector payload?

Did you call POST /_plugins/_security_analytics/mappings after ingesting that document or before?

@dnhodgson
Copy link
Author

Here is the detector, basically empty except for the one new rule.

{
  "_id": "qV0J2ogBkrwQRiHzXT_l",
  "_version": 1,
  "detector": {
    "name": "user auth failed count detector",
    "detector_type": "linux",
    "enabled": true,
    "schedule": {
      "period": {
        "interval": 1,
        "unit": "MINUTES"
      }
    },
    "inputs": [
      {
        "detector_input": {
          "description": "",
          "indices": [
            "logs"
          ],
          "custom_rules": [
            {
              "id": "THUI2ogB9tTD_zFnkofZ"
            }
          ],
          "pre_packaged_rules": []
        }
      }
    ],
    "last_update_time": "2023-06-20T18:20:25.570Z",
    "enabled_time": "2023-06-20T18:20:25.570Z"
  }
}

Yes, I missed that in my steps but, create the index, create the security mapping, create the new rules, create the detector, start loading records.

@getsaurabh02
Copy link
Member

@petardz is looking into it

@petardz
Copy link
Contributor

petardz commented Jun 20, 2023 •
edited
Loading

@dnhodgson Your call to POST /_plugins/_security_analytics/mappings is trying to add alias field "timestamp" to an existing field "@timestamp". You don't have "@timestamp" field in your index.

In your case, you don't have to apply any mappings or call mappings APIs if your index already has field of type date called timestamp:

PUT logs
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      },
      "user": {
        "type": "keyword"
      },
      "result": {
        "type": "keyword"
      }
    }
  }
}

Also regarding your rule, count agg with field is not supported currently. You can find supported examples here: link

@dnhodgson
Copy link
Author

ok, I see you can do a count(*) so I changed my query and tested again but still get no results and errors.

updated detection rule

id: dywh3ogBTP4IzJwH0eG3
logsource:
  product: linux
title: User auth failed count
description: test
tags: []
falsepositives:
  - ''
level: medium
status: experimental
references: []
author: Darcy Hodgson
detection:
  selection:
    result: Failed
    user: joe
  condition:
    - selection | count(*) by user > 2

I tried to be more specific by adding in the specific user name but that doesn't seem to make any difference.

@dnhodgson
Copy link
Author

More investigations into the errors, the first error, [2023-06-21T15:28:26,706][INFO ][o.o.a.InputService ] [opensearch-node1] Error collecting inputs for monitor: qH4i3ogBOrf1sDh_kqa-

When I look into that monitor GET _plugins/_alerting/monitors/qH4i3ogBOrf1sDh_kqa-

{
  "_id": "qH4i3ogBOrf1sDh_kqa-",
  "_version": 3,
  "_seq_no": 4,
  "_primary_term": 1,
  "monitor": {
    "type": "monitor",
    "schema_version": 6,
    "name": "user failed auth count detector",
    "monitor_type": "bucket_level_monitor",
    "enabled": true,
    "enabled_time": 1687353986642,
    "schedule": {
      "period": {
        "interval": 1,
        "unit": "MINUTES"
      }
    },
    "inputs": [
      {
        "search": {
          "indices": [
            "logs"
          ],
          "query": {
            "query": {
              "bool": {
                "must": [
                  {
                    "query_string": {
                      "query": """(result: "Failed") AND (user: "joe")""",
                      "fields": [],
                      "type": "best_fields",
                      "default_operator": "or",
                      "max_determinized_states": 10000,
                      "enable_position_increments": true,
                      "fuzziness": "AUTO",
                      "fuzzy_prefix_length": 0,
                      "fuzzy_max_expansions": 50,
                      "phrase_slop": 0,
                      "escape": false,
                      "auto_generate_synonyms_phrase_query": true,
                      "fuzzy_transpositions": true,
                      "boost": 1
                    }
                  },
                  {
                    "range": {
                      "timestamp": {
                        "from": "{{period_end}}||-1h",
                        "to": "{{period_end}}",
                        "include_lower": false,
                        "include_upper": true,
                        "format": "epoch_millis",
                        "boost": 1
                      }
                    }
                  }
                ],
                "adjust_pure_negative": true,
                "boost": 1
              }
            },
            "version": true,
            "seq_no_primary_term": true,
            "aggregations": {
              "result_agg": {
                "terms": {
                  "field": "user",
                  "size": 10,
                  "min_doc_count": 1,
                  "shard_min_doc_count": 0,
                  "show_term_doc_count_error": false,
                  "order": [
                    {
                      "_count": "desc"
                    },
                    {
                      "_key": "asc"
                    }
                  ]
                }
              }
            }
          }
        }
      }
    ],
    "triggers": [
      {
        "bucket_level_trigger": {
          "id": "dywh3ogBTP4IzJwH0eG3",
          "name": "User auth failed count",
          "severity": "medium",
          "condition": {
            "buckets_path": {
              "_cnt": "_cnt"
            },
            "parent_bucket_path": "result_agg",
            "script": {
              "source": "params._cnt > 2.0",
              "lang": "painless"
            },
            "gap_policy": "skip"
          },
          "actions": []
        }
      }
    ],
    "last_update_time": 1687353986642,
    "data_sources": {
      "query_index": ".opensearch-sap-linux-detectors-queries",
      "findings_index": ".opensearch-sap-linux-findings",
      "findings_index_pattern": "<.opensearch-sap-linux-findings-{now/d}-1>",
      "alerts_index": ".opensearch-sap-linux-alerts",
      "alerts_history_index": ".opensearch-sap-linux-alerts-history",
      "alerts_history_index_pattern": "<.opensearch-sap-linux-alerts-history-{now/d}-1>",
      "query_index_mappings_by_type": {
        "text": {
          "analyzer": "rule_analyzer"
        }
      },
      "findings_enabled": true
    },
    "owner": "security_analytics"
  }
}

I can run the query manually and get results however it looks like the trigger condition is incorrect. If I create the same monitor and trigger with the API it also fails with the same errors. It seems that one problem with the trigger condition buckets_path should be "_cnt":"_count" however this isn't the whole problem.

If I use the editor to create a monitor with the same functionality I can get it to work, the only different seems to be the aggregation, terms vs composite, the latter being the one created with the editor and works to detect the count of the users > 2 and trigger the alerts. the outputs from the trigger being slightly different.

From the detector created monitor, trigger input and output:
Note, with _cnt fixed to make it kind of work.

TRIGGER CONDITION:
{
    "buckets_path": {
        "_cnt": "_count"
    },
    "parent_bucket_path": "result_agg",
    "script": {
        "source": "params._cnt > 2.0",
        "lang": "painless"
    },
    "gap_policy": "skip"
}

TRIGGER CONDITION RESPONSE:
[
    {
        "doc_count": 83,
        "key": "joe"
    }
]

from the manual created monitor that works creating alerts.

TRIGGER CONDITION:
{
    "buckets_path": {
        "_count": "_count"
    },
    "parent_bucket_path": "composite_agg",
    "script": {
        "source": "params._count > 2",
        "lang": "painless"
    },
    "gap_policy": "skip"
}

TRIGGER CONDITION RESPONSE:
[
    {
        "doc_count": 83,
        "key": {
            "user": "joe"
        }
    }
]

@dnhodgson
Copy link
Author

Also note, I can't manually update the detector monitor with the API to test because it changes the monitor sources back to the .opensearch-alerting-* indexes.

@petardz
Copy link
Contributor

petardz commented Jun 21, 2023 •
edited
Loading

Thank you for this analysis!

As you concluded, issue was that bucket_path. Terms agg returns field called _count and not _cnt. I fixed that and I am able to generate Findings/Alerts with your modified rule(replaced field inside count with *).

I'll push a quick PR soon for this.

@petardz petardz mentioned this issue Jun 22, 2023
Closed
5 tasks
@dnhodgson
Copy link
Author

Is this patch going to make it into the 2.9.0 release?

@sbcd90 sbcd90 mentioned this issue Sep 30, 2023
Merged
5 tasks
@eirsep eirsep closed this as completed Oct 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sbcd90 sbcd90

@eirsep eirsep

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@sbcd90 @dnhodgson @eirsep @petardz @getsaurabh02 @AWSHurneyt