[ML] Adding V3 modules for Security_Linux and Security_Windows

[dishadasgupta]

Summary

• Follow up to [ML] Update Security_Linux and Security_Windows Modules to V3 #123000 since the previous PR overwrote the V2 module/files, was simpler to kick off a new PR
• Adds files for new + refactored v3 jobs for both security_linux and security_windows - for use within the Security app.

Files/Job Artifacts:

---

• 2 updated manifest .json files - for both linux and windows

• Updated/new ML Job configurations for 26 jobs - each with associated datafeed configuration files:

  • security_linux: 14 jobs

    • v3_linux_anomalous_network_activity
    • v3_linux_anomalous_network_port_activity_ecs
    • v3_linux_anomalous_process_all_hosts_ecs
    • v3_linux_anomalous_user_name_ecs
    • v3_linux_network_configuration_discovery
    • v3_linux_network_connection_discovery
    • v3_linux_rare_metadata_process
    • v3_linux_rare_metadata_user
    • v3_linux_rare_sudo_user
    • v3_linux_rare_user_compiler
    • v3_linux_system_information_discovery
    • v3_linux_system_process_discovery
    • v3_linux_system_user_discovery
    • v3_rare_process_by_host_linux_ecs

  • security_windows: 12 jobs

    • v3_rare_process_by_host_windows_ecs
    • v3_windows_anomalous_network_activity_ecs
    • v3_windows_anomalous_path_activity_ecs
    • v3_windows_anomalous_process_all_hosts_ecs
    • v3_windows_anomalous_process_creation
    • v3_windows_anomalous_script
    • v3_windows_anomalous_service
    • v3_windows_anomalous_user_name_ecs
    • v3_windows_rare_metadata_process
    • v3_windows_rare_metadata_user
    • v3_windows_rare_user_runas_event
    • v3_windows_rare_user_type10_remote_login

Tests:

---

Individual job test tracking stats available here: https://docs.google.com/spreadsheets/d/1JOUIVsitaMdEdhM3WT2Eag4ELI-rI2Jec7bXildJsdQ/edit#gid=0

@randomuserid to also post more updates as needed to this issue + regarding tests, thanks

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[dishadasgupta]

@elastic/ml-ui - Referring to #85065, I see there are some other edits I need to make to ensure these show up in the UI, doing so

[dishadasgupta]

Also updated some descriptions/docs based on @szabosteve comments left from #123000

[dishadasgupta]

A few questions in order to progress on the errors I'm seeing above:

I'm failing this test from this code part here. My understanding is the reference module ids need to be updated to also include security_windows_v3. I'm a little unfamiliar with how to do this - is there a better point person I can work with to get this fixed? I'm anticipating a similar fix needed to include security_linux_v3 in other relevant areas too.

Happy to pair and provide more context, please let me know what I can do from my end to help, thank you!

[pheyos]

@dishadasgupta I've looked into your two question and for the linter one find my suggested code change below.

For the test failures: I haven't looked too closely into the module definitions yet, but if the v3 modules are supposed to basically match the same data, then those failures are expected and the fix is to add security_linux_v3 to the expected recognized module everywhere we already recognize security_linux (and the same for the windows module). Note, that the order matters. Here's what worked for me locally: https://gist.github.com/pheyos/bb396d6b954c0af3ec26c202d2392efb
If the v3 module is not supposed to match the same data, we should take a closer look what's going on.

[bfilar]

This is a new refactored job which works on ECS compatible events across multiple indices. This sentence appears in the description field for 14 of the jobs.

We should remove that language due to:

1. It significantly increases the size of the overall description
2. It is disjointed to read, compared to the rest of the description.

I believe we can capture the “refactor” sentiment in release notes.

[sophiec20]

Thanks for getting this ready for review well before FF. Some minor points

1. Module description

This appears in the create job wizard tile. There is limited space in the tile - suggest we keep it succinct and descriptive.

Current: "Security: Linux v3 (2022). This module is a replacement for the v2 Linux module. This module contains all shipping ML jobs for Linux host based threat hunting and detection.."

V2 module description was
"Detect suspicious activity using ECS Linux events. Tested with Auditbeat and the Elastic agent."

Suggest something like "Detect suspicious activity using ECS Linux events. Replaces Security Linux V2." - (there is no need to repeat the module title).

2. Standard configs

These comments apply to multiple job configs:

The following fields can be removed. Recommend standardising across all modules.
detector_index
categorization_examples_limit
time_format

The following fields are applicable. Recommend applying and standardising for all modules:
"allow_lazy_open": true
"created_by": "ml-module-security-linux" or "created_by": "ml-module-security-windows"

Note that the created_by field is used in ML telemetry. This was not versioned for V2 and for ML telemetry purposes, it is not needed to be versioned for V3.

[randomuserid]

"created_by": "ml-module-security-linux" or "created_by": "ml-module-security-windows"

Note that the created_by field is used in ML telemetry. This was not versioned for V2 and for ML telemetry purposes, it is not needed to be versioned for V3.

@SourinPaul @donaherc where these are used for telemetry now, in ML job telemetry, what values should we be placing in the created_by field?

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 29377d1
• Storybooks Preview

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	4.6MB	4.6MB	+42.0B

History

• 💛 Build #18748 was flaky 81c4fb4
• 💚 Build #18517 succeeded 2f27547
• 💔 Build #18462 failed 70bafa2
• 💔 Build #18460 failed 8f35086
• 💔 Build #18416 failed 3bdb2f8
• 💔 Build #18188 failed 0fc7ba8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @randomuserid @dishadasgupta

[randomuserid]

We're going to pause this and hold it for 8.2. See https://github.com/elastic/security-team/issues/1490