Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Detection Engine] Adds 8.6 rules #146402

Merged
merged 5 commits into from
Nov 29, 2022
Merged

Conversation

terrancedejesus
Copy link
Contributor

Summary

Pull updates to detection rules from https://github.com/elastic/detection-rules/tree/0600b575650f3bec81720da72b1cea7481978576.

Checklist

Delete any items that are not applicable to this PR.

@terrancedejesus terrancedejesus requested a review from a team as a code owner November 28, 2022 14:37
@terrancedejesus terrancedejesus self-assigned this Nov 28, 2022
@terrancedejesus terrancedejesus added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting auto-backport Deprecated - use backport:version if exact versions are needed v8.6.0 trade-artifacts Issues related to TRADE artifact building and releasing labels Nov 28, 2022
@terrancedejesus
Copy link
Contributor Author

Although we added rule updates in ##145256 and would expect a decrease in rule updates, it seems there was some bulk updates with tags.

@terrancedejesus
Copy link
Contributor Author

terrancedejesus commented Nov 28, 2022

@xcrzx or @banderror - Any chance you have some time to review these errors? The seem related to the Kibana CI job and some TS files.

Buildkite Job - https://buildkite.com/elastic/kibana-pull-request/builds/91038#0184bead-77f7-4df7-aefe-62a1c7542dca

The build API docs and check types seem to be failing with a similar error message

Build API Docs and Check Types Error
proc [tsc] x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/notice.ts:1:1 - error TS1208: 'notice.ts' cannot be compiled under '--isolatedModules' because it is considered a global script file. Add an import, export, or an empty 'export {}' statement to make it a module.
--
  | proc [tsc]
  | proc [tsc] 1 /* eslint-disable @kbn/eslint/require-license-header */
  | proc [tsc]
  | proc [tsc]
  | proc [tsc]
  | proc [tsc] Found 1 error.
  | proc [tsc]
  | ERROR Unable to build TS project refs
  | 🚨 Error: The command exited with status 1
  | Running local post-command hook | 0s
  | Agent Debug Info | 5s
  | Agent Metrics:
  | https://kibana-ops-buildkite-monitoring.kb.us-central1.gcp.cloud.es.io:9243/app/metrics/link-to/host-detail/kb-n2-4-spot-483f3b754c18dc73?to=1669653726615&from=1669646526615
  | Agent Logs:
  | https://kibana-ops-buildkite-monitoring.kb.us-central1.gcp.cloud.es.io:9243/app/logs/link-to/host-logs/kb-n2-4-spot-483f3b754c18dc73?time=1669646526615
  | 2022-11-28 14:59:35 WARN   No meta-data value exists with key `0184bead-77f7-4df7-aefe-62a1c7542dca_is_test_execution_step`, returning the supplied default ""
  | user command error: exit status 1



From your review does it seem like this is a problem with our rules that TRaDE can adjust on our end?

@xcrzx
Copy link
Contributor

xcrzx commented Nov 29, 2022

@terrancedejesus With regards to the types error, an export statement was removed from the notice.ts file:

Screenshot 2022-11-29 at 10 38 57

It looks like notice.ts is an autogenerated file, but recently, it was modified manually (see this commit).

So I believe this CLI command should be updated to paste an empty export {} to the generated file to make it a module, so Typescript wouldn't complain anymore.

Meanwhile, I've manually added an empty export statement in this PR as a workaround.

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Security Solution Tests #2 / Alerts detection rules table auto-refresh should disable auto refresh when any rule selected and enable it after rules unselected
  • [job] [logs] Security Solution Tests #1 / Cases Creates a new case with timeline and opens the timeline

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id before after diff
osquery 1 2 +1

ESLint disabled line counts

id before after diff
enterpriseSearch 19 21 +2
fleet 59 65 +6
osquery 109 115 +6
securitySolution 442 448 +6
total +20

Total ESLint disabled count

id before after diff
enterpriseSearch 20 22 +2
fleet 68 74 +6
osquery 110 117 +7
securitySolution 519 525 +6
total +21

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @terrancedejesus

@terrancedejesus
Copy link
Contributor Author

@terrancedejesus With regards to the types error, an export statement was removed from the notice.ts file:

Screenshot 2022-11-29 at 10 38 57

It looks like notice.ts is an autogenerated file, but recently, it was modified manually (see this commit).

So I believe this CLI command should be updated to paste an empty export {} to the generated file to make it a module, so Typescript wouldn't complain anymore.

Meanwhile, I've manually added an empty export statement in this PR as a workaround.

Thank you for pointing this out and taking the time to dig into it! I will get this updated on our end temporarily as we look to adjust the kibana PR CLI command on our end to align with the removal of FS rules.

@terrancedejesus terrancedejesus merged commit d4aff16 into main Nov 29, 2022
@terrancedejesus terrancedejesus deleted the detection-rules/8.6-0600b575 branch November 29, 2022 14:33
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.6 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 146402

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

@xcrzx
Copy link
Contributor

xcrzx commented Nov 29, 2022

💚 All backports created successfully

Status Branch Result
8.6

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

xcrzx pushed a commit to xcrzx/kibana that referenced this pull request Nov 29, 2022
## Summary

Pull updates to detection rules from
https://github.com/elastic/detection-rules/tree/0600b575650f3bec81720da72b1cea7481978576.

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)

Co-authored-by: Dmitrii <[email protected]>
(cherry picked from commit d4aff16)

# Conflicts:
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/collection_email_powershell_exchange_mailbox.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/collection_winrar_encryption.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/command_and_control_common_webservices.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/command_and_control_dns_tunneling_nslookup.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/command_and_control_port_forwarding_added_registry.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/command_and_control_rdp_tunnel_plink.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/command_and_control_remote_file_copy_desktopimgdownldr.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/command_and_control_remote_file_copy_mpcmdrun.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/command_and_control_teamviewer_remote_file_copy.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/credential_access_cmdline_dump_tool.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/credential_access_dump_registry_hives.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/credential_access_mimikatz_memssp_default_logs.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/credential_access_mod_wdigest_security_provider.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/credential_access_symbolic_link_to_shadow_copy_created.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_amsienable_key_mod.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_clearing_windows_console_history.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_clearing_windows_event_logs.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_create_mod_root_certificate.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_defender_disabled_via_registry.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_defender_exclusion_via_powershell.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_disable_posh_scriptblocklogging.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_disable_windows_firewall_rules_with_netsh.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_disabling_windows_defender_powershell.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_disabling_windows_logs.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_enable_inbound_rdp_with_netsh.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_enable_network_discovery_with_netsh.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_execution_msbuild_started_by_office_app.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_ms_office_suspicious_regmod.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_powershell_windows_firewall_disabled.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_suspicious_process_creation_calltrace.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_system_critical_proc_abnormal_file_activity.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_unusual_ads_file_creation.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/defense_evasion_workfolders_control_execution.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_adfind_command_activity.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_admin_recon.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_net_view.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_peripheral_device.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_posh_invoke_sharefinder.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_remote_system_discovery_commands_windows.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_security_software_grep.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_security_software_wmic.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_suspicious_self_subject_review.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/discovery_whoami_command_activity.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_abnormal_process_id_file_created.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_command_shell_started_by_svchost.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_from_unusual_path_cmdline.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_linux_netcat_network_connection.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_ms_office_written_file.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_pdf_written_file.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_reverse_shell_via_named_pipe.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_revershell_via_shell_cmd.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_suspicious_jar_child_process.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_suspicious_pdf_reader.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_suspicious_powershell_imgload.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_user_exec_to_pod.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_via_hidden_shell_conhost.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/execution_via_xp_cmdshell_mssql_stored_procedure.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/impact_backup_file_deletion.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/impact_hosts_file_modified.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/index.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/lateral_movement_execution_via_file_shares_sequence.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/lateral_movement_remote_services.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/notice.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/persistence_exposed_service_created_with_type_nodeport.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/persistence_run_key_and_startup_broad.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/persistence_shell_activity_by_web_server.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/persistence_via_update_orchestrator_service_hijack.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/persistence_webshell_detection.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_disable_uac_registry.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_persistence_phantom_dll.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_pod_created_with_hostipc.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_pod_created_with_hostnetwork.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_pod_created_with_hostpid.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_pod_created_with_sensitive_hospath_volume.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_printspooler_suspicious_spl_file.json
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules/privilege_escalation_privileged_pod_created.json
xcrzx added a commit that referenced this pull request Nov 29, 2022
# Backport

This will backport the following commits from `main` to `8.6`:
- [[Detection Engine] Adds 8.6 rules
(#146402)](#146402)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Terrance
DeJesus","email":"[email protected]"},"sourceCommit":{"committedDate":"2022-11-29T14:33:20Z","message":"[Detection
Engine] Adds 8.6 rules (#146402)\n\n## Summary\r\n\r\nPull updates to
detection rules
from\r\nhttps://github.com/elastic/detection-rules/tree/0600b575650f3bec81720da72b1cea7481978576.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),\r\nuses
sentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n\r\nCo-authored-by:
Dmitrii
<[email protected]>","sha":"d4aff1626f0bb98739224a7652e17258e5594f78","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","auto-backport","trade-artifacts","v8.6.0"],"number":146402,"url":"https://github.com/elastic/kibana/pull/146402","mergeCommit":{"message":"[Detection
Engine] Adds 8.6 rules (#146402)\n\n## Summary\r\n\r\nPull updates to
detection rules
from\r\nhttps://github.com/elastic/detection-rules/tree/0600b575650f3bec81720da72b1cea7481978576.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),\r\nuses
sentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n\r\nCo-authored-by:
Dmitrii
<[email protected]>","sha":"d4aff1626f0bb98739224a7652e17258e5594f78"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"8.6","label":"v8.6.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Terrance DeJesus <[email protected]>
terrancedejesus added a commit that referenced this pull request Nov 30, 2022
…r 8.6 Release (#146730)

## Summary

This PR copies all pre-packaged rules from main and mirrors them in 8.6.
From the prior rule update to FS rules, backporting failed and had to be
manually triggered to backport from main to 8.6. However, after further
inspection, the rules were not mirrored correctly.

* Rule updates for 8.6 to main:
#146402
* Manual backport to 8.6: #146588

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes trade-artifacts Issues related to TRADE artifact building and releasing v8.6.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants