Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[8.x] [SecuritySolution][SIEM Migrations] Rule migrations storage (#1…
…97032) (#197603) # Backport This will backport the following commits from `main` to `8.x`: - [[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)](#197032) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sergi Massaneda","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-24T09:56:08Z","message":"[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)\n\n## Summary\r\n\r\nissue: elastic/security-team#10654 the persistence layer for the rule migrations from other\r\nvendors, as part of the SIEM Rule migrations effort.\r\n\r\n### Changes\r\n\r\n- Schemas created for `SiemRuleMigration` document entity, along with\r\n`ElasticRule` and `OriginalRule`.\r\n\r\n- New API `/internal/siem_migrations/rules` was created:\r\n- `POST` -> Receives an array of (original) rules and stores them with\r\n`status: pending` to be processed. Responds with the `migration_id` that\r\nwill be used to start the migration background task (implementation\r\ndetails here: https://github.com/elastic/security-team/issues/10850).\r\n - `GET` -> (to be implemented later)\r\n\r\n- New `SiemMigrationsService` added to the `securitySolution` route\r\ncontext, to encapsulate all operations related to SIEM migrations (We\r\nstart with _rule_ migrations, but there are more \"kinds\" of SIEM\r\nmigrations in the pipeline: _dashboards_, _saved queries_...). It\r\ncontains:\r\n\r\n- `SiemRuleMigrationsService` to encapsulate all operations related to\r\nSIEM rule migrations.\r\n- `RuleMigrationsDataStream` class to manage the\r\n`.kibana.siem-rule-migrations-<spaceId>` data stream operations using\r\n`DataStreamSpacesAdapter`.\r\n- It exposes a client with abstracted operations that are exposed to the\r\nAPI routes:\r\n- `create`: indexes an array of _SiemRuleMigration_ documents to the\r\ndata stream\r\n- `search`: searches _SiemRuleMigration_ documents by specific terms.\r\n\r\n> [!NOTE] \r\n> Without `siemMigrationsEnabled` experimental flag the new API route\r\nwon't be registered, and the `SiemRuleMigrationsService` _setup_ won't\r\nbe called, so no index/component template will be installed to ES.\r\n\r\n### Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\n<details>\r\n <summary>Example curl request</summary>\r\n\r\n```\r\ncurl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\nThe newly created documents can be retrieved using Kibana DevTools\r\nconsole:\r\n```\r\nGET .kibana.siem-rule-migrations-default/_search\r\n```\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"d7109d67810cedfce1ad2bf9e8fd826b20aee06b","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat Hunting","Team: SecuritySolution","backport:prev-minor","8.18 candidate","v8.17.0"],"number":197032,"url":"https://github.com/elastic/kibana/pull/197032","mergeCommit":{"message":"[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)\n\n## Summary\r\n\r\nissue: elastic/security-team#10654 the persistence layer for the rule migrations from other\r\nvendors, as part of the SIEM Rule migrations effort.\r\n\r\n### Changes\r\n\r\n- Schemas created for `SiemRuleMigration` document entity, along with\r\n`ElasticRule` and `OriginalRule`.\r\n\r\n- New API `/internal/siem_migrations/rules` was created:\r\n- `POST` -> Receives an array of (original) rules and stores them with\r\n`status: pending` to be processed. Responds with the `migration_id` that\r\nwill be used to start the migration background task (implementation\r\ndetails here: https://github.com/elastic/security-team/issues/10850).\r\n - `GET` -> (to be implemented later)\r\n\r\n- New `SiemMigrationsService` added to the `securitySolution` route\r\ncontext, to encapsulate all operations related to SIEM migrations (We\r\nstart with _rule_ migrations, but there are more \"kinds\" of SIEM\r\nmigrations in the pipeline: _dashboards_, _saved queries_...). It\r\ncontains:\r\n\r\n- `SiemRuleMigrationsService` to encapsulate all operations related to\r\nSIEM rule migrations.\r\n- `RuleMigrationsDataStream` class to manage the\r\n`.kibana.siem-rule-migrations-<spaceId>` data stream operations using\r\n`DataStreamSpacesAdapter`.\r\n- It exposes a client with abstracted operations that are exposed to the\r\nAPI routes:\r\n- `create`: indexes an array of _SiemRuleMigration_ documents to the\r\ndata stream\r\n- `search`: searches _SiemRuleMigration_ documents by specific terms.\r\n\r\n> [!NOTE] \r\n> Without `siemMigrationsEnabled` experimental flag the new API route\r\nwon't be registered, and the `SiemRuleMigrationsService` _setup_ won't\r\nbe called, so no index/component template will be installed to ES.\r\n\r\n### Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\n<details>\r\n <summary>Example curl request</summary>\r\n\r\n```\r\ncurl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\nThe newly created documents can be retrieved using Kibana DevTools\r\nconsole:\r\n```\r\nGET .kibana.siem-rule-migrations-default/_search\r\n```\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"d7109d67810cedfce1ad2bf9e8fd826b20aee06b"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197032","number":197032,"mergeCommit":{"message":"[SecuritySolution][SIEM Migrations] Rule migrations storage (#197032)\n\n## Summary\r\n\r\nissue: elastic/security-team#10654 the persistence layer for the rule migrations from other\r\nvendors, as part of the SIEM Rule migrations effort.\r\n\r\n### Changes\r\n\r\n- Schemas created for `SiemRuleMigration` document entity, along with\r\n`ElasticRule` and `OriginalRule`.\r\n\r\n- New API `/internal/siem_migrations/rules` was created:\r\n- `POST` -> Receives an array of (original) rules and stores them with\r\n`status: pending` to be processed. Responds with the `migration_id` that\r\nwill be used to start the migration background task (implementation\r\ndetails here: https://github.com/elastic/security-team/issues/10850).\r\n - `GET` -> (to be implemented later)\r\n\r\n- New `SiemMigrationsService` added to the `securitySolution` route\r\ncontext, to encapsulate all operations related to SIEM migrations (We\r\nstart with _rule_ migrations, but there are more \"kinds\" of SIEM\r\nmigrations in the pipeline: _dashboards_, _saved queries_...). It\r\ncontains:\r\n\r\n- `SiemRuleMigrationsService` to encapsulate all operations related to\r\nSIEM rule migrations.\r\n- `RuleMigrationsDataStream` class to manage the\r\n`.kibana.siem-rule-migrations-<spaceId>` data stream operations using\r\n`DataStreamSpacesAdapter`.\r\n- It exposes a client with abstracted operations that are exposed to the\r\nAPI routes:\r\n- `create`: indexes an array of _SiemRuleMigration_ documents to the\r\ndata stream\r\n- `search`: searches _SiemRuleMigration_ documents by specific terms.\r\n\r\n> [!NOTE] \r\n> Without `siemMigrationsEnabled` experimental flag the new API route\r\nwon't be registered, and the `SiemRuleMigrationsService` _setup_ won't\r\nbe called, so no index/component template will be installed to ES.\r\n\r\n### Testing locally\r\n\r\nEnable the flag\r\n```\r\nxpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']\r\n```\r\n\r\n<details>\r\n <summary>Example curl request</summary>\r\n\r\n```\r\ncurl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \\\r\n--header 'kbn-xsrf;' \\\r\n--header 'x-elastic-internal-origin: security-solution' \\\r\n--header 'elastic-api-version: 1' \\\r\n--header 'Content-Type: application/json' \\\r\n--data '[\r\n {\r\n \"id\": \"f8c325ea-506e-4105-8ccf-da1492e90115\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Add User Account Type\",\r\n \"description\": \"The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.\",\r\n \"query\": \"sourcetype=\\\"linux:audit\\\" type=ADD_USER \\n| rename hostname as dest \\n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \\n| `security_content_ctime(firstTime)` \\n| `security_content_ctime(lastTime)`\\n| search *\",\r\n \"query_language\":\"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1136\"\r\n ]\r\n },\r\n {\r\n \"id\": \"7b87c556-0ca4-47e0-b84c-6cd62a0a3e90\",\r\n \"vendor\": \"splunk\",\r\n \"title\": \"Linux Auditd Change File Owner To Root\",\r\n \"description\": \"The following analytic detects the use of the '\\''chown'\\'' command to change a file owner to '\\''root'\\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.\",\r\n \"query\": \"`linux_auditd` `linux_auditd_normalized_proctitle_process`\\r\\n| rename host as dest \\r\\n| where LIKE (process_exec, \\\"%chown %root%\\\") \\r\\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \\r\\n| `security_content_ctime(firstTime)` \\r\\n| `security_content_ctime(lastTime)`\\r\\n| `linux_auditd_change_file_owner_to_root_filter`\",\r\n \"query_language\": \"spl\",\r\n \"mitre_attack_ids\": [\r\n \"T1222\"\r\n ]\r\n }\r\n]'\r\n```\r\n</details>\r\n\r\nThe newly created documents can be retrieved using Kibana DevTools\r\nconsole:\r\n```\r\nGET .kibana.siem-rule-migrations-default/_search\r\n```\r\n\r\n### Screenshots\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"d7109d67810cedfce1ad2bf9e8fd826b20aee06b"}},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
- Loading branch information
