crowdstrike: Return empty events array when no resources in alert, host. #10831

kcreddy · 2024-08-21T06:41:04Z

Proposed commit message

Return empty events array when no resources in alert and host data-streams.

When there are no resources in first API call, current CEL code returns state.
But this state doesn't have events inside it. As per CEL docs, the events field
is necessary. Without this, the following errors appear and lead to restarting of input.

{"log.level":"error","@timestamp":"2024-08-21T06:26:37.252Z","message":"unexpected missing events array from evaluation","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"input_source":"http://svc-crowdstrike-host:8090","input_url":"http://svc-crowdstrike-host:8090","ecs.version":"1.6.0","log.logger":"input.cel","log.origin":{"file.line":350,"file.name":"cel/input.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1"},"service.name":"filebeat","id":"cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-21T06:26:37.252Z","message":"unregistering","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"input_type":"cel","key":"cel-crowdstrike_host-20697e27-0353-4b36-8c17-18f5ea89ae90::http://svc-crowdstrike-host:8090","ecs.version":"1.6.0","log.logger":"metric_registry","log.origin":{"file.line":70,"file.name":"inputmon/input.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry.func1"},"service.name":"filebeat","id":"cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90::http://svc-crowdstrike-host:8090","uuid":"4ee4a37e-1dde-445d-9f17-2429c435d848","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-21T06:26:37.252Z","message":"Input 'cel' failed with: input.go:130: input cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90 failed (id=cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90)\n\tunexpected type returned for evaluation events: <nil>","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"log.logger":"input.cel","log.origin":{"file.line":132,"file.name":"compat/compat.go","function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1"},"service.name":"filebeat","id":"cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

By modifying "resources": [] in the config-alert.yml and config-host.yml:

Before: (example with host)
eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v --data-streams=host

{"log.level":"error","@timestamp":"2024-08-21T06:26:37.252Z","message":"unexpected missing events array from evaluation","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"input_source":"http://svc-crowdstrike-host:8090","input_url":"http://svc-crowdstrike-host:8090","ecs.version":"1.6.0","log.logger":"input.cel","log.origin":{"file.line":350,"file.name":"cel/input.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1"},"service.name":"filebeat","id":"cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-08-21T06:26:37.252Z","message":"unregistering","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"input_type":"cel","key":"cel-crowdstrike_host-20697e27-0353-4b36-8c17-18f5ea89ae90::http://svc-crowdstrike-host:8090","ecs.version":"1.6.0","log.logger":"metric_registry","log.origin":{"file.line":70,"file.name":"inputmon/input.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry.func1"},"service.name":"filebeat","id":"cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90::http://svc-crowdstrike-host:8090","uuid":"4ee4a37e-1dde-445d-9f17-2429c435d848","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-21T06:26:37.252Z","message":"Input 'cel' failed with: input.go:130: input cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90 failed (id=cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90)\n\tunexpected type returned for evaluation events: <nil>","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"log.logger":"input.cel","log.origin":{"file.line":132,"file.name":"compat/compat.go","function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1"},"service.name":"filebeat","id":"cel-crowdstrike.host-20697e27-0353-4b36-8c17-18f5ea89ae90","ecs.version":"1.6.0","ecs.version":"1.6.0"}

After:
No error messages in agent logs.

eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v --data-streams=host,alert

--- Test results for package: crowdstrike - START ---
╭─────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE     │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ crowdstrike │ alert       │ system    │ common    │ PASS   │ 37.831208584s │
│ crowdstrike │ host        │ system    │ common    │ PASS   │ 36.294177167s │
╰─────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: crowdstrike - END   ---
Done

elasticmachine · 2024-08-21T06:47:56Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elasticmachine · 2024-08-21T07:08:12Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

chrisberkhout · 2024-08-21T10:10:53Z

packages/crowdstrike/data_stream/alert/agent/stream/cel.yml.hbs

@@ -75,7 +75,7 @@ program: |
      }
    )
  ).as(state, state.with(
-    !has(state.resources) ? state : // Exit early due to GET failure or no resources to collect.
+    !has(state.resources) ? {"events": []} : // Exit early due to GET failure or no resources to collect.


That changes a couple of other things:

it doesn't return the calculated want_more and offset (maybe not important)

it doesn't return the error event in the case of a GET failure (important)

I think the change should instead be to insert "events": [], below the ?"resources" line.

Same for both data streams.

Nice catch. I updated the CEL code with your suggestion and re-ran the system tests. No error messages in the logs, so its working.

andrewkroh

When there are no resources in first API call, current CEL code returns state

Can we mock this in our system test scenario? Like have it return no resources in a response before it returns resources.

kcreddy · 2024-08-21T13:07:26Z

Can we mock this in our system test scenario? Like have it return no resources in a response before it returns resources.

I was unable to mock the no resources scenario at the beginning, but at the end. i.e., first it returns resources and subsequent calls returns no resources. Commit for this change: eb72037.

The issue I had with making it at the beginning was that I am unable to increment offset because resources.size() is 0. Whatever offset that I start with, I end up with same value, thus unable to move to next page. https://github.com/elastic/integrations/blob/main/packages/crowdstrike/data_stream/alert/agent/stream/cel.yml.hbs#L55-L58

elasticmachine · 2024-08-21T13:30:33Z

💚 Build Succeeded

Buildkite Build
Commit: eb72037

History

💚 Build #14852 succeeded 9d1ee20
💚 Build #14820 succeeded 04b5269

elastic-sonarqube · 2024-08-21T13:30:37Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-08-21T15:04:08Z

Package crowdstrike - 1.39.1 containing this change is available at https://epr.elastic.co/search?package=crowdstrike

…st. (elastic#10831) Return empty events array when no resources in alert and host data-streams. When there are no resources in first API call, current CEL code returns state. But this state doesn't have events inside it. As per CEL input docs, the events field is necessary. Without this, the errors occur and lead to restarting of input.

kcreddy added 2 commits August 21, 2024 12:05

Return empty events array when no resources in alert, host.

d0deba1

update pr

04b5269

kcreddy added Integration:crowdstrike CrowdStrike bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Aug 21, 2024

kcreddy marked this pull request as ready for review August 21, 2024 06:47

kcreddy requested a review from a team as a code owner August 21, 2024 06:47

chrisberkhout requested changes Aug 21, 2024

View reviewed changes

Address PR comments

9d1ee20

kcreddy requested a review from chrisberkhout August 21, 2024 10:39

andrewkroh reviewed Aug 21, 2024

View reviewed changes

address pr comments

eb72037

kcreddy requested a review from andrewkroh August 21, 2024 13:30

andrewkroh approved these changes Aug 21, 2024

View reviewed changes

chrisberkhout approved these changes Aug 21, 2024

View reviewed changes

kcreddy merged commit 07bf7ed into elastic:main Aug 21, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crowdstrike: Return empty events array when no resources in alert, host. #10831

crowdstrike: Return empty events array when no resources in alert, host. #10831

kcreddy commented Aug 21, 2024 •

edited

Loading

elasticmachine commented Aug 21, 2024

elasticmachine commented Aug 21, 2024 •

edited

Loading

chrisberkhout Aug 21, 2024

kcreddy Aug 21, 2024

andrewkroh left a comment •

edited

Loading

kcreddy commented Aug 21, 2024

elasticmachine commented Aug 21, 2024

elastic-sonarqube bot commented Aug 21, 2024

elasticmachine commented Aug 21, 2024

crowdstrike: Return empty events array when no resources in alert, host. #10831

crowdstrike: Return empty events array when no resources in alert, host. #10831

Conversation

kcreddy commented Aug 21, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

elasticmachine commented Aug 21, 2024

elasticmachine commented Aug 21, 2024 • edited Loading

🚀 Benchmarks report

chrisberkhout Aug 21, 2024

Choose a reason for hiding this comment

kcreddy Aug 21, 2024

Choose a reason for hiding this comment

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

kcreddy commented Aug 21, 2024

elasticmachine commented Aug 21, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Aug 21, 2024

Quality Gate passed

elasticmachine commented Aug 21, 2024

kcreddy commented Aug 21, 2024 •

edited

Loading

elasticmachine commented Aug 21, 2024 •

edited

Loading

andrewkroh left a comment •

edited

Loading