[SecuritySolution][SIEM Migrations] Rule migrations storage

[semd]

Summary

issue: https://github.com/elastic/security-team/issues/10654?reload=1

Implements the persistence layer for the rule migrations from other vendors, as part of the SIEM Rule migrations effort.

Changes

• Schemas created for SiemRuleMigration document entity, along with ElasticRule and OriginalRule.

• New API /internal/siem_migrations/rules was created:

  • POST -> Receives an array of (original) rules and stores them with status: pending to be processed. Responds with the migration_id that will be used to start the migration background task (implementation details here: https://github.com/elastic/security-team/issues/10850).
  • GET -> (to be implemented later)

• New SiemMigrationsService added to the securitySolution route context, to encapsulate all operations related to SIEM migrations (We start with rule migrations, but there are more "kinds" of SIEM migrations in the pipeline: dashboards, saved queries...). It contains:

  • SiemRuleMigrationsService to encapsulate all operations related to SIEM rule migrations.
    • RuleMigrationsDataStream class to manage the .kibana.siem-rule-migrations-<spaceId> data stream operations using DataStreamSpacesAdapter.
    • It exposes a client with abstracted operations that are exposed to the API routes:
      • create: indexes an array of SiemRuleMigration documents to the data stream
      • search: searches SiemRuleMigration documents by specific terms.

Note

Without siemMigrationsEnabled experimental flag the new API route won't be registered, and the SiemRuleMigrationsService setup won't be called, so no index/component template will be installed to ES.

Testing locally

Enable the flag

xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']

Example curl request

curl --location 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
    {
        "id": "f8c325ea-506e-4105-8ccf-da1492e90115",
        "vendor": "splunk",
        "title": "Linux Auditd Add User Account Type",
        "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
        "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
        "query_language":"spl",
        "mitre_attack_ids": [
            "T1136"
        ]
    },
    {
        "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
        "vendor": "splunk",
        "title": "Linux Auditd Change File Owner To Root",
        "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
        "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
        "query_language": "spl",
        "mitre_attack_ids": [
            "T1222"
        ]
    }
]'

The newly created documents can be retrieved using Kibana DevTools console:

GET .kibana.siem-rule-migrations-default/_search

Screenshots

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[kapral18]

2 nits and LGTM :) Good job.

[banderror]

Nit: can we register conditionally inside the registerSiemMigrationsRoutes function itself? That way next time we need to remove the feature flag check, it won't trigger additional CODEOWNERS reviews.

[semd]

@elasticmachine merge upstream

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11496951526

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 810cb2c

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
securitySolution	117	118	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
securitySolution	32	33	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	87.6KB	87.6KB	+25.0B

Unknown metric groups

API count

id	before	after	diff
securitySolution	185	186	+1

History

• 💔 Build #245452 failed c31885d
• 💔 Build #245181 failed 46804a0
• 💔 Build #244380 failed c61964e

cc @semd

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 197032

Questions ?

Please refer to the Backport tool documentation

[semd]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation