[ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2 #131166

bfilar · 2022-04-28T16:38:38Z

Summary

Follow up to a closed PR in 8.1 [ML] Adding V3 modules for Security_Linux and Security_Windows #123274
Follow up to another closed PR in 8.2 [ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2 #128099
This PR seeks to consolidate the current set of Security ML Modules: https://github.com/elastic/security-team/issues/2960
Added files for new + refactored v3 jobs for both security_linux and security_windows - for use within the Security app.
Deprecated v1 and v2 versions of those jobs
Deprecated Auditbeat Host Process module due to a duplicative job in the Linux Security module
Consolidated auth modules for Linux and Windows will be consolidated into their respective OS modules

Files/Job Artifacts:

2 updated manifest .json files - for both linux and windows
Updated/new ML Job configurations for 26 jobs - each with associated datafeed configuration files:
- security_linux: 14 jobs
  - v3_linux_anomalous_network_activity
  - v3_linux_anomalous_network_port_activity_ecs
  - v3_linux_anomalous_process_all_hosts_ecs
  - v3_linux_anomalous_user_name_ecs
  - v3_linux_network_configuration_discovery
  - v3_linux_network_connection_discovery
  - v3_linux_rare_metadata_process
  - v3_linux_rare_metadata_user
  - v3_linux_rare_sudo_user
  - v3_linux_rare_user_compiler
  - v3_linux_system_information_discovery
  - v3_linux_system_process_discovery
  - v3_linux_system_user_discovery
  - v3_rare_process_by_host_linux_ecs
- security_windows: 12 jobs
  - v3_rare_process_by_host_windows_ecs
  - v3_windows_anomalous_network_activity_ecs
  - v3_windows_anomalous_path_activity_ecs
  - v3_windows_anomalous_process_all_hosts_ecs
  - v3_windows_anomalous_process_creation
  - v3_windows_anomalous_script
  - v3_windows_anomalous_service
  - v3_windows_anomalous_user_name_ecs
  - v3_windows_rare_metadata_process
  - v3_windows_rare_metadata_user
  - v3_windows_rare_user_runas_event
  - v3_windows_rare_user_type10_remote_login

Tests:

Individual job test tracking stats available here: https://docs.google.com/spreadsheets/d/1JOUIVsitaMdEdhM3WT2Eag4ELI-rI2Jec7bXildJsdQ/edit#gid=0

@randomuserid to also post more updates as needed to this issue + regarding tests, thanks

removed references to deprecated v1 modules which no longer exist

remove references to deprecated module and modify module names to match the latest v3 modules being committed.

think this is what the linter wants

…into bfilar-ml-refactor

bfilar · 2022-04-28T16:40:02Z

@peteharverson @pheyos, the SecML team will coordinate w/ @spong to identify the engineering stakeholders for failing tests. We will seek to resolve these in the coming weeks.

elasticmachine · 2022-04-28T16:55:11Z

Pinging @elastic/ml-ui (:ml)

spong · 2022-04-28T17:26:48Z

@peteharverson @pheyos, the SecML team will coordinate w/ @spong to identify the engineering stakeholders for failing tests. We will seek to resolve these in the coming weeks.

Sounds good! We have this issue for tracking the rework of these tests in support of the v3 modules: #128318

Ideally we'd like to coordinate that rework as part of this PR to ensure coverage, but if they need to happen as a follow-up as to not block you folks that's fine as well too. So feel free to skip any Security Solution tests (maybe add a TODO: PR 131166 comment above them) that are giving you trouble for the time being.

randomuserid · 2022-05-09T13:54:31Z

@elasticmachine merge upstream

remove a dup json element

into bfilar.ml-refactor-2

add the Security: Windows prefix which was missing

missing bracket

...l/server/models/data_recognizer/modules/security_windows/ml/v3_windows_anomalous_script.json

the prefix was in the wrong place

peteharverson

Tested job creation works inside the ML job wizard. LGTM.

kibana-ci · 2022-05-17T18:15:22Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	5.0MB	5.0MB	-74.0B

History

💚 Build #45403 succeeded 4974858
💔 Build #45381 failed 927c270
💔 Build #45271 failed 9621f03
💔 Build #45133 failed f579bc7
💔 Build #45125 failed 019bb00
💛 Build #45038 was flaky d918c12

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @randomuserid @bfilar

lcawl

Text strings LGTM

spong

Checked out, tested locally, and verified functionality of the new v3 modules within the Security Solution app. Was able to install and start a job from the ML Job Settings UI, and then created a detection rule referencing the job without issue.

Note I: I did see one warning in the Kibana console after initiating the install, which hits the /api/ml/modules/setup/security_windows_v3 API with this payload:

Payload

Kibana Console Warning

[2022-05-17T14:50:44.805-06:00][WARN ][plugins.ml] Data recognizer could not estimate model memory limit {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[Overall] cardinality estimate required for [by_field_name] [winlog.event_data.ServiceName] but not supplied"}],"type":"illegal_argument_exception","reason":"[Overall] cardinality estimate required for [by_field_name] [winlog.event_data.ServiceName] but not supplied"},"status":400}

@peteharverson, is there something we need to update on our end with this request, or is this expected?

Note II: With regards to the skipped tests, we have this issue for tracking #128318 and @banderror will be following up with a PR to unskip these tests and move them to the new v3 modules.

All that said, LGTM from the Security Solution side of the house! 😀 Excited for the consolidation here -- thanks @bfilar & @randomuserid!

yctercero

Just looking at the exceptions cypress test (code owners) - if we can track that skip that'd be great.

dhurley14

LGTM! Looks @elastic/security-solution-platform only needed to review a skipped test.

peteharverson · 2022-05-18T09:57:40Z

is there something we need to update on our end with this request, or is this expected?

[2022-05-17T14:50:44.805-06:00][WARN ][plugins.ml] Data recognizer could not estimate model memory limit {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[Overall] cardinality estimate required for [by_field_name] [winlog.event_data.ServiceName] but not supplied"}],"type":"illegal_argument_exception","reason":"[Overall] cardinality estimate required for [by_field_name] [winlog.event_data.ServiceName] but not supplied"},"status":400}

@spong in the data set you are using for your tests, are there any docs with winlog.event_data.ServiceName used as the by_field_name for the v3_windows_anomalous_service job? The checks used to estimate the model memory limit for the job will not succeed if they cannot obtain an estimate for the cardinality of some of the fields used in the job. I get the same error using my test data set, where none of the docs contain winlog.event_data.ServiceName.

spong · 2022-05-18T16:55:52Z

@spong in the data set you are using for your tests, are there any docs with winlog.event_data.ServiceName used as the by_field_name for the v3_windows_anomalous_service job? The checks used to estimate the model memory limit for the job will not succeed if they cannot obtain an estimate for the cardinality of some of the fields used in the job. I get the same error using my test data set, where none of the docs contain winlog.event_data.ServiceName.

Ahh, that's indeed the case, my sample data does not include those fields either. Thanks for confirming Pete! 🙂

…ecurity modules (#153222) The cold and frozen query filters which were added [here](#116696) were accidentally removed in this [PR](#131166)

bfilar and others added 17 commits March 9, 2022 14:53

consolidate Security ML Modules

a8d2ebb

removal of auditbeat host processes ecs module

d3d25c6

removing siem_winlogbeat_auth after consolidating into windows_security

95b7c8e

renamed to avoid job collisions

dc6aac6

Update recognize_module.ts

2952d2a

removed references to deprecated v1 modules which no longer exist

test fixes

014896b

remove references to deprecated module and modify module names to match the latest v3 modules being committed.

Merge branch 'main' into bfilar-ml-refactor

6017c46

Update recognize_module.ts

c34ae06

think this is what the linter wants

Merge branch 'bfilar-ml-refactor' of https://github.com/elastic/kibana …

39ee594

…into bfilar-ml-refactor

Merge branch 'main' into bfilar-ml-refactor

5c316a2

deprecating winlogbeat and auditbeat modules

755b26e

fixes test post-deprecation of modules

3a16dd1

fixes typo in test

b37e0df

revert linting changes

ff9aee5

revert linting changes pt2

d4e3cba

fixing test in setup_module.ts

b59d8b9

ml module refactor

9a6e965

bfilar requested review from a team as code owners April 28, 2022 16:38

bfilar assigned randomuserid and bfilar Apr 28, 2022

peteharverson added :ml Feature:Anomaly Detection ML anomaly detection labels Apr 28, 2022

Merge branch 'main' into bfilar.ml-refactor-2

5c97101

randomuserid added release_note:enhancement release_note:deprecation labels May 9, 2022

randomuserid added 5 commits May 17, 2022 10:42

linting

c2832fe

remove a dup json element

Merge branch 'bfilar.ml-refactor-2' of https://github.com/elastic/kibana

62726d6

into bfilar.ml-refactor-2

Update v3_windows_anomalous_script.json

927c270

add the Security: Windows prefix which was missing

Update v3_linux_anomalous_network_activity.json

702e642

missing bracket

Merge branch 'main' into bfilar.ml-refactor-2

4974858

peteharverson reviewed May 17, 2022

View reviewed changes

...l/server/models/data_recognizer/modules/security_windows/ml/v3_windows_anomalous_script.json Outdated Show resolved Hide resolved

peteharverson changed the title ~~[DRAFT][ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2~~ [ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2 May 17, 2022

Update v3_windows_anomalous_script.json

719d6f1

the prefix was in the wrong place

peteharverson approved these changes May 17, 2022

View reviewed changes

lcawl approved these changes May 17, 2022

View reviewed changes

spong approved these changes May 17, 2022

View reviewed changes

yctercero approved these changes May 17, 2022

View reviewed changes

dhurley14 approved these changes May 17, 2022

View reviewed changes

bfilar merged commit f85c39e into main May 18, 2022

bfilar deleted the bfilar.ml-refactor-2 branch May 18, 2022 14:33

kibanamachine added the backport:skip This commit does not require backporting label May 18, 2022

This was referenced May 19, 2022

[DOCS] Updates list of security and auditbeat anomaly detection configurations elastic/stack-docs#2140

Merged

[DOCS] Remove broken links to discontinued anomaly detection jobs elastic/security-docs#1998

Merged

nastasha-solomon mentioned this pull request May 20, 2022

[DOCS] 8.3.0 Release Notes elastic/security-docs#1964

Closed

lcawl mentioned this pull request May 20, 2022

[ML] Minor edits in prebuilt job descriptions #132633

Merged

spong mentioned this pull request May 23, 2022

[SecuritySolution][Detections] Adds UI for new Rule Fields: Related Integrations, Required Fields, and Setup #131475

Merged

5 tasks

banderror mentioned this pull request May 30, 2022

[Security Solution][Detections] Re-enable and fix broken ML tests as result of new V3 Modules #128318

Closed

xcrzx mentioned this pull request Jun 7, 2022

[Security Solution] Re-enable broken ML tests #133510

Merged

jgowdyelastic mentioned this pull request Mar 15, 2023

[ML] Reinstating cold and frozen tier filters for linux and windows security modules #153222

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2 #131166

[ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2 #131166

bfilar commented Apr 28, 2022

bfilar commented Apr 28, 2022

elasticmachine commented Apr 28, 2022

spong commented Apr 28, 2022

randomuserid commented May 9, 2022

peteharverson left a comment

kibana-ci commented May 17, 2022

lcawl left a comment

spong left a comment

yctercero left a comment

dhurley14 left a comment

peteharverson commented May 18, 2022

spong commented May 18, 2022

[ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2 #131166

[ML] Adding v3 modules for Security_Linux and Security_Windows and Deprecating v1 + v2 #131166

Conversation

bfilar commented Apr 28, 2022

Summary

Files/Job Artifacts:

Tests:

bfilar commented Apr 28, 2022

elasticmachine commented Apr 28, 2022

spong commented Apr 28, 2022

randomuserid commented May 9, 2022

peteharverson left a comment

Choose a reason for hiding this comment

kibana-ci commented May 17, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

History

lcawl left a comment

Choose a reason for hiding this comment

spong left a comment

Choose a reason for hiding this comment

Payload

Kibana Console Warning

yctercero left a comment

Choose a reason for hiding this comment

dhurley14 left a comment

Choose a reason for hiding this comment

peteharverson commented May 18, 2022

spong commented May 18, 2022